Dark Data Has Huge Potential, But Not If We Keep Ignoring It

Splunk released the State of Dark Data Report today about dark data and AI and—spoiler alert—IT and business leaders around the world think those things are important. But the report also points out a developing situation that had worried me a bit, and should probably concern you if you’re in the technology business: the gap between the expectation and reality of dark data and AI. But first, let me go back and define a term I used up there in the first line.

“Dark data” may sound like the title of a cyberpunk novel, but it’s really just a dramatic term for the unknown and untapped data across your organization, generated by systems, devices and interactions. Gartner defines dark data as:

“The information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Similar to dark matter in physics, dark data often comprises most organizations’ universe of information assets. Thus, organizations often retain dark data for compliance purposes only. Storing and securing data typically incurs more expense (and sometimes greater risk) than value.”*

Dark data is valuable, in part, because there’s so much of it, and it could be used to unlock business value. The more data you have, the more you can learn about every aspect of your operations. The larger the dataset, the more accurate the analysis. If you ignore the challenge of dark data you’re leaving money on the table. In fact, 76 percent of survey respondents agree “the organization that has the most data is going to win.”

Let’s not forget there is a real monetary cost to having dark data and not using it, in addition to the opportunity cost. Data storage may be a heck of a lot cheaper than it used to be, but it still isn’t free. If you’re paying to store data you don’t use, you’re wasting money and opportunities to leverage that data.

So if dark data is so valuable and leaders know it, why aren’t companies using it? Because they don’t have the right tools. Think of the “zombie” systems that haunt the server rooms of so many companies: legacy monitoring solutions kludged together with duct tape and chicken wire and kept running by the sweat, prayers and swears of IT managers. The true darkness inherent in both comes from their wasted, hidden, potential value. Companies that refuse to move into the future with new IT solutions will never be able to take advantage of dark data. They will lose the data race before they get to the starting line.

Now back to that worrying situation I mentioned at the start. You could easily read this report as a reaffirmation of the obvious: data is valuable and AI can help us use it more effectively. But there’s a deeper interpretation of the report’s key findings:

1. Senior IT and business leaders agree dark data has tremendous value, but
2. they don’t know how to use it (in fact, 56 percent say that “data-driven” is just a slogan in their organization),
3. they’re counting on AI to help them extract the value from it, yet don’t have the technology, expertise or staff necessary to use AI, and
4. a majority of respondents (69 percent) admit they aren’t even trying to learn new data skills, with roughly half saying they’re too old (which in my opinion is a cop-out—they’re really saying, “Data is too hard”).

The only country in this global survey where a high percentage of respondents said they’re working hard to meet the challenges of both dark data and AI is China.

No matter how you interpret the findings of this report, it’s time to figure out your dark data strategy. Is your company keeping pace to stay competitive? Are you hiring the right people and encouraging a data-driven mindset in your organization and offering education programs to make it real? And are you investing in technology that will help you bring your dark data into the light?

That last one is something Splunk can help you with.

I encourage you to read the full State of Dark Data Report, which goes into far more detail on respondents’ attitudes and plans for dark data, AI and the future of work. You can check it out here. 

^{*Gartner IT Glossary, "Dark Data," 2019.}

----------------------------------------------------
Thanks!
Tim Tully