Data Challenges Pave Way for Creative Strategies

Like water flowing through a hydroelectric dam, data is the life force that powers our modern world. It fuels our systems, products, services — and every decision we make.

But having enough data isn’t the problem. Most organizations are drowning in oceans of it. That sea level has grown exponentially over the years, thanks to cloud services, IoT systems, AI platforms and countless other products — all stored (some might even say trapped) in various locations.

These massive data volumes come with a slew of consequences, from outpacing budgets to  overwhelming teams. According to the latest Splunk report The New Rules of Data Management, 67% of respondents cited large data volumes and growth as a challenge when implementing their data strategy, while 73% labeled data volume as a primary cause for rising data management costs.

But in addition to costs, the noise and complexity often create blind spots that can stymie operations and increase the risk of missing critical incidents and alerts that could compromise or disrupt systems. And as with any problem, the longer organizations ignore these issues, the more difficult it is for them to swim their way out.

The good news is that these challenges are not insurmountable. Addressing them just requires a clear strategy. A data strategy — which should include a clear vision on how to manage and gain value from telemetry data — is an essential part of your business that will help it thrive in a data-centric world. In this article, we discuss recent data management trends, tech leaders’ emerging approaches and solutions to a rising tide of data, and best practices that will help strengthen your teams and make your business better. 

AI ups the data management game

It’s not lost on anyone that AI has the potential to be a force multiplier for business systems and processes. That holds especially true for data management. According to Splunk’s data management report, a staggering 98% of survey respondents agreed that AI made their data management strategy easier, with 33% saying it was significantly easier.

For Dmitry Gromov, senior director, SRE at data analytics firm Dun and Bradstreet, AI has the potential to uplevel his services. Because he and his teams cover all aspects of monitoring, Gromov has increasingly incorporated more business services that encompass “how to monitor AI and how to use AI in observability,” he said. “Like any other technology, it needs to be monitored.”

From a data management perspective, Gromov said that AI has the potential to help organizations analyze data in their environment to provide insights, as well as look for anomalies that could lead to possible disruptions. “AI can show where the gaps are in my coverage, and if I have everything set up right. But going a little bit further, AI can help you understand what data you have and get context. And it gives you a better idea of what you might be missing,” he said.

He’s not wrong. Where AI offers the most value for organizations is by performing routine administrative functions — 73% of respondents said that AI enhanced data quality by automating repetitive tasks such as documentation, or analysis, which can lead to faster decision making. That in turn has the potential to free up teams to focus on critical investigations and other functions that move the business forward such as research and development, product enhancement, or other innovations.

But AI can provide higher-value data management tasks as well. Data management survey respondents also hailed the technology for data discovery (59%), including scanning large datasets and identifying trends and anomalies, as well as supporting initiatives like data virtualization (45%). Additionally, AI can be highly effective in maintaining high data quality by detecting unwanted changes across various data quality dimensions.

Gromov added that AI’s processing and analytical capabilities can also help cut down on the amount of duplicative data, in turn, cutting down on complexity and making teams more efficient. “With AI in the observability space, we have the ability to get this data in one place where we can analyze it for redundancies,” he said. Creating copies of data for specific analyses, instead of reusing the same data for different purposes, can harm data quality and greatly increase the organization's costs and effort to manage the data.

Sharing data across the aisle

Effective data management often requires more than just good tooling. Like any sound and resilient strategy, data management necessitates communication, collaboration and shared knowledge between teams — also leading to good data governance.

These practices also include data sharing and reuse. According to the Splunk data management survey, data reuse within individual teams is gaining traction, with 42% saying they leverage this practice within their individual teams and 60% saying they use it across security, IT, and engineering.

But what’s more, teams who employ data reuse practices see strong and tangible results, with improvements critical metrics such as overall impact of incidents (52% vs. 35% others), total number of data breach incidents (44% vs. 33% others), and MTTR (69% vs. 60% others).

With this in mind, when teams are analyzing incidents that they can’t quite figure out, it’s helpful to circle SecOPs and IT to review dashboards and work as a team to get to the root cause collectively. More teams with diverse skill sets processing and discussing the data ultimately reduces the amount of time spent on investigations. And when they don’t have to wait long for an accurate analysis, team leaders can simply make decisions faster.

A unified platform gives a big picture view

Having a window into the entirety of your data environment is critical for effective data management — that holds especially true when data is stored in numerous locations and formats throughout your organization. Unless you’re leveraging a data federation practice, a distributed data environment can put a big damper on quick data access, data quality, and cost of data management.

For most organizations, it’s an all too common challenge. In the data management report, 53% said they had to log into different platforms to access different data sources. Meanwhile, only a small number of respondents say their data management strategy included components such as unified visibility (13%) and unified accessibility (11%).

A unified analytics platform, however, can help fix that — it offers benefits like faster time to insights, reduced tool costs, and improved governance.

For example, let’s say that your platform is primarily a security tool that gives your team the ability to execute on basic security functions and use cases. But because the platform provides a holistic view of your environment and unfettered access to all of your data, your team can also obtain answers to other types of queries that may not be security-related.

With a platform that allows you to find relevant telemetry based upon alerts, look for bottlenecks, and hunt for various artifacts, you can more easily scale and evolve with changing priorities.

Gromov echoed that consolidating all monitoring tools into one platform has “made a huge impact” when they share data across teams.

“We used to have different tools across the organization. Now the team knows one tool,” he said. “Besides the contract and licensing, we have the ability to share the knowledge of our specific tools across all teams — you can share expertise, share examples, share dashboards. And when the teams respond to incidents, it’s one team that helps.”

Ultimately, there is no-one-size-fits-all for data management. SecOps, IT, and engineering teams will often have to tailor and curate their own data and data management strategies based on their business needs, goals, and priorities. But regardless of organizational size or mission, data practices that include collaboration, sharing and reuse, a platform approach, and AI to automate routine tasks will likely breathe new life into your data strategy that will drive your business forward, faster. 

To learn more about how to implement strong practices and create a winning data management strategy that sets you up for success, download The New Rules of Data Management report.