Live from Deep in the Heart of Texas: It’s SplunkLive Dallas

SplunkLive Dallas was the perfect mix of up-and-coming Splunk users and veterans showing off some deep technical know-how.

A few highlights from the day:

Rachel Neal, Client Manager for Connectivity Services from DerbySoft kicked off the customer stories for the day. DerbySoft provides electronic distribution, reservation, financial, and marketing services for hotels and travel distributors. She supports 4 globally distributed data centers, with developers and customers all over the globe–needless to say it’s a challenging environment to manage. She was candid, funny and told a great story. (I happened to be at JCPenney to check out their holiday readiness war room the following day and they told me she was their favorite!) A few stories from Rachel:

• Developers used to pick and choose from the error queue. When we got Splunk in, we saw there weren’t 50-some errors, there were more like 10K errors! But one was the same error repeating 732 times a day. Once we resolved that, we were able to improve our baseline and become more intelligent and proactive about addressing issues in our infrastructure
• “Splunk gives us a full-time 24-hour global support employee, alerting on, and in some cases, resolving errors.”
• “We have integration with our customers, so some of our errors are actually customer errors. Splunk triggers alerts that go directly to customers with all of the relevant bug/troubleshooting info.”
• Before Splunk problem solving: 17 steps/ 30 minutes; with Splunk: 1 step/ 3 minutes
• “It’s easy to chart things, just try it. Issues pop out–it’s easy to visualize what’s different.”

• Splunk is part of motivation plans for our employees. Our developers like to track who has the cleanest code/ fewest bugs—they view it as a competition and want to be the best. Splunk visualizes this to spur the competition. And the code is much, much cleaner
• “We’re driving revenue off of our Splunk alerts.”

Moving forward, Rachel and co. are looking to expand the number of data types they’re feeding into Splunk to work towards greater visibility and new opportunities to deliver added value services for their customers.

Splunk power user Gregg Woodcock is a Senior Member of the Technical Staff at MetroPCS. MetroPCS provides unlimited wireless communications service for a flat-rate with no annual contract to more than 9 million subscribers. MetroPCS is using Splunk (and achieving serious ROI) in 4 primary categories:

• Launching New Products and Services: Speeding application de-bugging means bringing new products to market faster, with fewer bugs, making for a better user experience
• Tariff Justification/ Optimization: Splunk statistical analysis uses RDBMS lookup to calculate cost per call
• Call Detail Record Visibility: Splunk’s ability to ingest any format without parsers or adapters speeds deployment and time to value
• Detecting Abuse: Reports and dashboards highlight possible abusers—key indicator of Terms of Service Abusers

And they’ve created dashboard views to detail revenue optimization and carrier savings.

MetroPCS Daily Revenue Optimization Dashboard

MetroPCS Carrier and Destination Savings Dashboard

He even shared an analogy that his team knew of earthquakes in Trinidad and Tabago before news broke as dashboards and alerts indicated their Answer/ Seize ratios (ASR) dropped in the region.

He also plugged Splunk as a Big data solution for them. They collect more than a terabyte a day in call detail record (CDR) data alone—and need to correlate it with various data sources. Splunk gives them  the flexibility and visibility they need to drive new business opportunities.

He shared s few scripts and macros he’d written and detailed how a send SNMP script they found on Splunkbase enabled them to launch a partner’s beta software that didn’t have SNMP alerting. Now the scheduled searches automatically raise alarms in our NOC.

Beyond that, Gregg had a few great quotes and a few handy tips I’d like to pass along:

• Always have way more disk that you think you’ll need
• Always have more indexers than you think you’ll need
• PUT THE DEPLOYMENT SERVER IN FIRST (it’s a pain to wedge in later and you WILL have to do this eventually)
• Convert discoveries into scheduled searches (don’t have the same “surprise” twice)
• “Even though we haven’t hired anybody new, with Splunk it’s like we’ve hired 20 people—people get little slivers of their life back–each new Splunk user spawns a mini person!” (Possibly not the most eloquent statement, but it’s certainly a funny visual!)
• “Anyone who does anything useful in Splunk has already paid back the implementation time for Splunk.”
• “Big data problems aren’t fun with traditional tools, but they’re fun to solve and dig in more with Splunk.”
• He also likened Splunk to legos, “You can always build something new, interesting, useful and it’s fun to play with.”

Finally we had Michael Gout, Lead Technical Architect for a leading provider of financial services. A polished former Marine, he started out with, “If you don’t have Splunk, you don’t know what you’re missing!” The organization has ranked among The Best Places to Work in IT list published by ComputerWorld for the last 2 years.

Michael primarily highlighted the way they’ve implemented High availability with Splunk, but he still offered a few endorsements:

• Running entirely in VMware on ESX, can always spin up another instance relatively easily
• “I anticipate having thousands of Splunk indexers in the future.”
• “We don’t have to compare notes with other teams to solve isues anymore, we all have the same view of the data with Splunk.”
• “Splunk is the swiss army knife of operational intelligence at our organization.”
• They have 2500 employees in IT, they love Splunk and run lunch and learns for new groups internally
• New folks always want their data in, so they’ve created a submission form and review committee. They’re asking who would look at the data and what new opportunities are they trying to derive from the data
• “We’re consolidating a bunch of systems moving to rely on dashboards in Splunk and abandoning other tools.”
• “This is the first tool people stop me in the hallway and thank me for bringing it in.”

A few screenshots which could help you in planning your HA implementation of Splunk:

Redundant Indexing Design

Redundant Indexing Forwarder Config

Redundant Heavy Forwarders

Redundant Search Heads

Thanks again to our fabulous customer presenters. Remember to stay in touch with the Dallas Users’ group:

----------------------------------------------------
Thanks!
Erin Sweeney