By Patrick Ogdin June 28, 2011
I had the opportunity to attend our SplunkLive event in Columbus the other day. Before I even mention the event, let me go on record as saying that the Blue Ribbon Pot Roast at the Tip Top Kitchen is probably the best darn pot roast I’ve ever had. Hopefully my mother doesn’t read this. Let’s talk SplunkLive. For those of you who have never been to a SplunkLive, it’s an event we hold regularly in cities all over the world for Splunk customers, partners, and those interested in learning a bit more about what’s going on at Splunk to interact. Having been in the tech industry for awhile, I’ve been to many vendor events where you get to hear one side of the story; the vendor’s. We’re very fortunate at Splunk to have a product that people really like and use so we ask customers to come and tell their story at SplunkLive about how Splunk has helped them solve problems or made their company/institution/etc more productive. I feel that these customer led conversations are way more relevant than anything coming out of vendor’s mouth since they are talking about real world applications without any abstraction. At other companies, it was hard enough to get public customer references much less have a customer come to speak about your product and hold an open dialogue with the audience (or contribute applications or help to answer product questions). Incredible! In Columbus we heard from Scott Reasinger, Director of E-commerce Technology at Finish Line, the athletic retailer and Christopher Robinson, VP UNIX Server Management for KeyBank.
Splunk in Finish Line’s eCommerce Environment
Finish Line has a large web and mobile presence and also runs over 660 storefronts. Scott talked about the challenges involved in keeping the online portion of Finish Line’s business running through peak periods like Black Friday which is a theme we consistently hear from our retail customers like Macys.com. He said that with a multi-tiered service delivery environment (web/app/database), it’s difficult to find and fix problems given the complexity of the system as a whole. After a small proof-of-concept with Splunk, they started finding failed purchase transactions almost immediately in their application server logs and other problems with locking on the database. Their traditional tools were giving them alerts but by the time they got the call from their monitoring team, it was too late. They started using Splunk as the “canary in the mine” to look for the conditions that would lead up to problems with the database and then remediate the issue before it could cause further harm. The agility that they were able to achieve with Splunk was compelling so they started throwing everything but the kitchen sink (only because it didn’t generate any logs) at Splunk. Finish Line uses Tealeaf for user experience monitoring and by tracing the Tealeaf session ID’s through every layer of the system they were able to use Splunk to stitch together what was happening on the user end, through web services calls, and finally to the database yielding a clear end-to-end picture. To quote Scott, “Extraordinarily powerful.” Now when a customer would call with an error condition, they could immediately dive in and diagnose the problem, complete the customer order showing an immediate return on investment with Splunk while increasing customer satisfaction. Troubleshooting coupon usage, finding and alerting on XML errors, determining when transactions were out of an acceptable duration, the list goes on. Splunk helps them save huge amounts of time. All in all a great talk from Scott.
Wrangling Everything UNIX with Splunk at KeyBank
Key Bank is a longtime Splunk customer and Christopher Robinson uses Splunk in his UNIX environment. He supports one of the largest banks in the US with19000 employees and nearly $100 Billion in assets. He’s got about 1000 boxes, 4 operating systems, and 14 kernels to manage and said that Splunk is a “godsend for troubleshooting”. The mantra at Key Bank is to do things better, get it done faster, and maybe save a little money along the way. We’d like to think Splunk helps them do all three. Christopher’s Splunk deployment spans two common Splunk use cases. They’re using Splunk on the IT Operations side for troubleshooting an enterprise directory infrastructure, watching for hardware errors on servers, and handful of other useful stuff. But they’re Splunk environment is also helping meet compliance and regulatory mandates common in the banking world like PCI and Sarbanes-Oxley. Using Splunk role based access controls and a few Splunk dashboards they quickly whipped up, they are able to show auditors when systems were accessed and that proper procedures were followed. This makes responding to audit requests less painful and less time consuming. Christopher envisions Splunk becoming the cross platform log aggregator for the bank over time and since Splunk tends to spread virally, he has to temper the enthusiasm for Splunk on other teams who are chomping at the bit to send him data until he can grow out his Splunk environment. Christopher and his team, who were in attendance, waxed philosophically about Splunk; “Splunk is a journey, not a destination”. I subscribe to this. Sure, we’ll give you some cool stuff out-of-the-box but in my opinion, the reason people find Splunk so powerful is that real process improvement and maintaing stellar service levels in the IT department takes continuous interaction with your systems and data. Splunk enables that. Also a great presentation from Christopher with some tips and advice from a team of seasoned Splunk veterans on deployment best practices.
Thanks again to Scott, Christopher, and everyone who attended the sessions. Hope to see you again at another SplunkLive Columbus!