Real ROI using Splunk–from the Great White North to the Deep South

We wrapped up our SplunkLive events for 2011 by migrating to warmer climates–namely, deep in the heart of Texas.

We even had a few customers from the Great White North join us on our migration path. Josh Diakun from Interac/ Acxsys and Derek Mock of Ceryx flew from Toronto to wow the Texans with all the different ways they’re using Splunk. And for some local flavor, Bob Jones from the City of Corpus Christi rounded out the agenda with tips and tricks to help fledgling Splunkers get started.

<public service announcement>

First I must say, this. This is a trio of superstars in the Splunk community. They use Answers, attended .conf, post apps to Splunkbase, hold lunch and learns at their companies, host users’ groups. I know folks like to contribute in different ways, so please, contribute to the Splunk community in whichever mechanism works for you, or let me know how you’d like to contribute and we’ll fold that into upcoming community efforts. OR if you need help getting started, check out this blog with loads of resources, or let me know.

</public service announcement>

First up, was Josh Diakun (@iam_Joshd on Twitter), Senior Systems Admin for Interac / Acxsys Corporation. Acxsys Corporation is the leading Canadian provider of debit card services. Acxsys brought Splunk in for continuous monitoring and to get a better handle on their security posture. They needed visibility across network, application and security systems and found Splunk was the best answer.

Josh worked with his security team to build an Enterprise Security app on Splunk. They started with 25 critical searches and the app has evolved to deliver 75 reports across 22 individually customized views. High level dashboards give red/ yellow/ green status across several key areas, including: data loss prevention, application thresholds, change monitoring, VPN summary and utilization times and brute force attacks.

Once they built out the Security app, they had the expertise to craft all the other apps they needed, so Josh went to work creating:

• Enterprise Storage Analytics App for Hitachi USP Series
• In-house Application Monitoring App
• In-house Operational Monitoring App
• In-house Systems Management App
• RSA SecurID Appliance Reporting App (get it on Splunkbase!)
• Barracuda Web Filter Reporting App (get it on Splunkbase!)

And Josh isn’t opposed to grabbing apps already in existence on Splunkbase, like these:

• Splunk for *NIX
• Splunk Deployment Monitor
• Splunk for OSSEC
• Splunk App for Web Intelligence
• Cisco Security Suite
• Sideview Utils
• Google Maps
• Splunk Mobile

Since bringing in Splunk they’ve been able to realize benefits across the organization. They’ve simplified business processes, and improved performance across marketing campaigns based on new visibility into web data. They are speeding investigations and becoming more proactive around applications and infrastructure. In fact, they’ve calculated an annual ROI of $500,000 based on the productivity increases Splunk provides to the security department.

Plus, there’s unexpected benefits–like a $100,000 return on investment from using Splunk as an analytics engine for their enterprise storage system (Hitachi USP series).

Keep up with Josh’s latest exploits by following him on Twitter.

Next up was Bob Jones, Director of Information Security for the City of Corpus Christi. For the past 3 years, Corpus Christi has been named among the Top 10 Digital Cities–so we’re dealing with pros here. For Bob running a city often feels like it’s 26 companies in one–utilities, libraries, parks, each with unique compliance mandates, management systems and siloed departments.

“Splunk’s ability to index everything was a main selling point for me,” said Bob. “We couldn’t wait for vendors to develop connectors or collectors for the loads of systems and log types coursing across the city’s infrastructure. I really just needed a view into what’s happening now. And then how does that compare to an hour ago, or yesterday, or last month.”

Splunk gave them the centralized view into the data they needed. Now they’re identifying security incidents 70% faster.

“With Splunk we have been able to identify key security-related articles of interest in over 500 million entries in a matter of seconds—compared to the much, much slower methods provided by vendors of our source systems.”

While spreading adoption of new systems can sometimes be a challenge, Bob said that once he started to visualize their data in Splunk, folks were much more interested in adopting.

“The eye can’t easily make sense of log files–but when you have a visual understanding of what’s normal, and what’s not–it becomes easy to pinpoint issues and prioritize investigations.”

Bob also found that configuring Windows lightweight forwarders allowed him to deploy once and leave folks alone–thus driving adoption.

Rounding out the day was Derek Mock, Director of Software Development from Ceryx, a leading provider of hosted enterprise applications. As is typical in many IT organizations, they were suffering from “death by terminal” and wanted to bring all of their data into a single view.

They went to work making it a possibility in Splunk. Security app, operational monitoring views, customer experience app–all built on Splunk–each providing monitoring, alerting, reporting and trending–plus direct drilldown into the data. The customer experience app is helping them to speed customer service response by tracking and exposing key metrics for various customers.

On the operational monitoring front, they’re using Splunk to track:

• Email violations / compromised account monitoring
• System performance monitoring
• Capacity management
• Error/Warning message detection
• System integrity monitoring

As cloud is a huge part of their service offering they built a custom app for their cloud services, monitoring for performance, usage, capacity, dashboards for auditing purposes and even web analytics. They’re driving visibility into key metrics and driving adoption across the staff using large monitors to display Splunk dashboards.

Derek’s also been working to deliver intelligence to the business. He’s partnered with key stakeholders and asked them what types of trends or data might be interesting to them. While he won’t answer all requests, he is able to build out some of the dashboards pretty quickly–underscoring the value and ROI Splunk is delivering across the organization.

Other tips from Derek:

• Do a live demo with management to show non-technical people the power of Splunk
• Ask business owners what questions they’d like to ask/ answer, but don’t know who/how to ask
• Get the Google maps app (ar amMap)–people love to visualize where customers are coming from and what they’re doing
• Storage sizing/performance
• Get involved in the Splunk community

They all set Mark Seward, Splunk’s Director of Security and Compliance Marketing, up perfectly to discuss and demo Security Intelligence, and Rahul Deshmukh, Director of Web Intelligence Marketing, was over the moon from Josh’s commentary about the Splunk App for Web Intelligence being the “coolest Splunk app ever.”

Thanks again to all of our fantastic customer presenters–we appreciate your ideas and support.

----------------------------------------------------
Thanks!
Erin Sweeney