Some time ago, in a company not too far away, I woke up with unstructured data on my mind, thinking about ways to correlate real-time clickstream information with my existing customer base. The internal IT folks said that this was not possible, at least in a timeframe that would help my decision-making. This merely increased my fascination with people interacting with all these layers of technology in unpredictable ways, defying the cry of the old-school database administrator: “that which exists must fit into my predefined schema!” The marketer in me just wanted to make sense of it all, but I was at a loss at how to get around that schema requirement and let my data crayon wander outside the lines.
The second week of September of 2008 found me in Beaumont Texas, helping my parents weather Hurricane Ike. Before the winds picked up, you could hear a deep sound off to the east, like a freight train that never quite appears over the horizon. I woke up this morning hearing that same sound in my head, but this time it was the sound of Big Data in the realm of public discussion. Consider the following chart, which any reader can recreate for themselves in a few minutes:
Every once in a while, you wake up and realize that the world is a different place than it was in the not-too-distant past. New ideas gain momentum in the sphere of public discussion and can either sweep you off your feet with excitement or knock you down with surprise. There is a “big data” hurricane coming over the horizon right now, and each of us have a choice: be swept off of our feet with what it can offer, or get knocked down by competitors who have gotten there first.
Next week I’ll share some thoughts about what folks can do to prepare, and ways Splunk can help.