We all have a nine-to-five job. But I consider Splunk part of my five-to-nine. That’s when I choose to be at the office working on something because I enjoy it. It’s at the center of everything we do.
Mississippi Department of Revenue Saves Millions Preventing Downtime with Splunk Observability
Key Challenges:
The Mississippi Department of Revenue (MDOR) needed to manage rapid 2000% growth in its IT footprint. To transition away from reactive IT practices, and translate complex metrics into clear business outcomes to leadership, the team needed to replace a fragmented monitoring system with a centralized observability platform as they scaled.
Key Results:
With Splunk, MDOR transformed its observability practice for a new state-controlled, $130-million liquor distribution facility, significantly reducing downtime risks, streamlining federal audits through enhanced transparency, and empowering their team to shift from a reactive, break-fix cycle to a proactive, data driven, securityconscious operation.
Government IT goes far beyond just keeping the lights on.
For the Mississippi Department of Revenue, the stakes are measured in taxpayer dollars and the number of citizens impacted. Over the last decade, the state’s IT operations team steered the MDOR through a period of rapid expansion, as the agency’s IT footprint grew by 2,000% with investments in new technologies and structural changes.
But as the agency grew, MDOR’s ITOps team struggled to stay afloat, responding reactively to issues and incidents as they occurred. To support this massive growth, the team needed to transition away from a reactive, “cowboy” style of IT and replace a fragmented, ad-hoc monitoring system with a centralized, scalable observability platform that would allow them to see the entirety of their environment and keep pace with the rising number of incidents.
That’s where Splunk came in. The agency chose Splunk as its strategic partner to monitor and manage its massive new ecosystem. Splunk collapsed silos across their environment, unified visibility, improved data access, and monitored the entire system’s health — crucial for maintaining valuable uptime and meeting business objectives.
“Whenever we even think about mapping critical dependencies in an outage scenario, our Splunk instance is the first thing we need to have up and running. It’s the nerve center of application health and performance.” said Mike Dehaan, Chief Technology Officer at Mississippi Department of Revenue. “When we respond to issues, we always go to Splunk first.”
Outcomes:
- $1M+ saved
- 2000% IT growth
Building observability from the ground up
A major turning point in MDOR’s transformation was the construction of a new, high-tech liquor distribution warehouse. As the sole reseller of wine and spirits in Mississippi, the agency manages a $130-million annual business, but their previous facility lacked the network infrastructure to support modern operations.
Moving to a new, highly automated facility built on a containerization platform provided a “once in a career” opportunity to modernize. To ensure success, Dehaan prioritized observability from the outset. “If you don’t think about observability and monitoring from day one, trying to shoehorn it in on the back end is always going to be challenging,” he explained.
To manage the project’s complexity, the team made a strategic decision to prioritize their monitoring infrastructure. “We implemented the Splunk tooling first,” Dehaan noted. “Because we implemented new tooling first, it is actively helping us throughout the deployment process. As we chip away at this monster of a project, the first tool we take out and the last one we put away is Splunk.”
When pitching Splunk Observability to agency leadership, Dehaan illustrated the cost of failure in stark terms: a few days of downtime would cost well over $1 million. By prioritizing observability, the team ensured they could maintain business continuity even while navigating the unprecedented project velocity common in the industry today.
“Downtime is such a critical issue that we wanted to incorporate observability from the get-go because we knew it would directly translate to our business outcomes,” he said. “We ran the numbers. If we missed just three days of shipping because we didn’t have the tooling to identify and resolve issues, that alone would cost more than the entire investment in the Splunk platform. It made the ROI clear to everyone.”
When new people join our operations teams, they are in the first Splunk 4 Rookies class available. From cloud services to on-prem networks and everything in between, our engineers are more effective in their role when they learn the basics of Splunk.
Strong SecOps leads to better compliance
As the MDOR’s reliance on Splunk Observability deepened, the agency began to see the platform as more than just an operational tool — it was also an anchor for their security posture. Dehaan integrated security awareness into every operator’s workflow, dismantling the traditional siloed approach.
“I made a big bet,” he said. “We needed to embed security consciousness into every single one of our operators. One mistake from a team member can be missed by compliance staff focused on spreadsheet audits rather than operational reality.”
With Splunk, the team was able to embed security operations “as a survival mechanism,” to keep pace with rapid growth “Our security risk massively increased with all the data we brought on,” Dehaan said. “The only way we could accommodate that growth is by being security conscious.”
This security strategy was put to the test during a regulatory audit, which required the team to prepare for three to four months, then undergo two weeks of on-site auditing. But with the visibility and documentation Splunk provided, the MDOR team was ready. The auditors were so impressed that they wrote a letter to the Commissioner praising their performance and closing out the audit early. “I think that effort hits a little bit harder than just passing the audit,” Dehaan said.
ITSI highlights value to leaders
As Dehaan moved into a more executive role, he needed a way to communicate the health of the agency’s infrastructure to commissioners and non-technical stakeholders who weren’t interested in switch vulnerabilities or log errors. This is where Splunk ITSI (IT Service Intelligence) became a game changer.
“Our Commissioner wants to see how healthy our applications are,” Dehaan noted. “He doesn’t want to get into the technical weeds. He wants clean, rolled-up statistics. ITSI allows us to take these really vast data sets and collapse them down into specific consumable insights.”
By mapping technical performance to business outcomes, the team regularly demonstrates exactly how IT investments are driving the agency’s mission. It has transformed the IT department from a cost center into a partner that provides real-time visibility into the health of the state’s most critical services.
"We're directly focusing on outcomes for our executives to service the agency. ITSI is really helping us there,” he said.
Whenever we even think about mapping critical dependencies in an outage scenario, our Splunk instance is the first thing we need to have up and running. It’s the nerve center of application health and performance.
From break-fix to branching out operations
Before Splunk, the team was trapped in a reactive break-fix cycle, spending hours chasing outages.
Not so anymore. With Splunk, analysts fundamentally shifted their perception of data, from reactive to holistic and strategic. It’s a shift that translates to any job they take on when they join the operations team. "When new people join our operations teams, they are in the first Splunk 4 Rookies class available,” added Justin Dennis, Lead Infrastructure Engineer at the MDOR. “From cloud services to on-prem networks and everything in between, our engineers are more effective in their role when they learn the basics of Splunk."
Team members are not just proficient with Splunk for operations, they're actively expanding its utility. The platform, which started as a technical tool for infrastructure monitoring, now informs development teams, QA teams, and business units across the enterprise. Over the next few years, the goal is to continue embedding business logic into their Splunk instance.
“Start small and start curious,” Dehaan said. “Everything is like Legos — one brick at a time. It might seem like a monster at first. But once you take that first step, it clicks a lot quicker than you’d imagine.”
Splunking curiosity
In a landscape where government agencies are under constant pressure to modernize with smaller budgets and fewer resources, the Mississippi Department of Revenue proved that resilience is in the power of data and the courage to build a culture that values curiosity as much as code.
“We all have a nine-to-five job. But I consider Splunk part of my five-to-nine. That’s when I choose to be at the office working on something because I enjoy it. It’s at the center of everything we do,” Dehaan said. “Stroke the curiosity. There’s so much value in your data that you’re not taking advantage of. As you start digging into that, it’s so fun to watch it come alive.”
