Last week the Splunklive program rolled into Charlotte, NC. Given the commercial importance of this city and the level of interest we saw today, this was probably overdue. And besides, it was a really good day—great customer presentations, serious audience interest, excellent southern food, and possibly the best weather in the U.S.
nTelos: Management Dashboards in Hours
John Lewis, Manager of IT Assurance for this provider of wireless and wireline services, had some choice quotes about Splunk. “Splunk was brought in for security and compliance, and is now a ‘knowledge sharing system’. Splunk is an eye-opener.” And “Splunk is a data warehouse for IT and business data.”
The IT Assurance Group is an internal oversight group responsible for Information Security, Business Continuity and Compliance (SOX, PCI, CPNI). Their environment includes Mixed OS (Windows, Linux, iSeries, SCO, Solaris) and a multi-tiered web architecture: IIS and Apache web servers, database servers (MSSQL, Oracle), App servers and custom web apps, IPTV and a data warehouse.
A major theme of the presentation was Splunk flexibility: indexing any data, even from apps not designed for external logging; so many data input mechanisms (from syslog to scripting); types of dashboards and reports that can be created; and use cases ranging from proactive customer support to fraud detection.
John showed actual dashboards he created for others at nTelos. He recently downloaded the Splunk add-on for google maps and incorporated geolocation into a dashboard—“management loves it.” When asked by someone in the audience how long it took to build those dashboards, he replied “hours, not days or months.” The IPTV dashboard easily identifies spikes in issues and enables customer support to proactively address issues.
What’s next? Focusing on fraud detection. Having IT and business people work together to understand the new capabilities, and leverage that to find fraud.
UNC-Chapel Hill: So Many Unanticipated Uses for Splunk
James Ervin, from UNC’s IT Services Enterprise Systems department, gave a detailed presentation on their environment, Splunk deployment, and use cases. “The unanticipated uses of Splunk turned out to be just as important as the anticipated ones.” Primary use cases for Splunk include security (“SIM/SIEM” type functionality), Identity Management, App management, Infrastructure management and troubleshooting, and Resource management. An important aspect of their university environment: new log sources arriving almost daily as new applications, servers installed.
UNC is indexing 100 GB/day with Splunk, and the user base has grown to 150. James described many of the classic issues that organizations face and why they chose Splunk—including how well Splunk manages unstructured data. Once Splunk was deployed, they realized that many of their systems were not logging locally (too much data, performance issues, etc), which is unacceptable for compliance, security or systems management. Splunk became the de facto standard for collecting, centralizing, and managing the log information.
He told the audience “Anything is 10X faster with Splunk.” When asked during the Q&A how many people are managing their Splunk deployment, he replied “You’re looking at him!”
Cisco: Identifying New Security Threats
Dave Schwartzburg of Cisco’s CSIRT team (Computer Security Incident Response Team) gave the third customer presentation of the day. Not surprisingly, Cisco has a massive infrastructure: 130,000+ Windows desktops, 90,000 Solaris, Linux, Mac desktops, 15,000+ Data Center servers , 40,000 routers…
Cisco is using Splunk in three broad areas: Investigate, Mitigate, and Prevent. Splunk has enabled Cisco to do data collection and correlations previously impossible. As a result, they’ve identified malware and other security issues and attacks not previously detected—10% of their cases now are from these newly detected issues.
Cisco has created a variety of alerts and reports with Splunk, including malware detection, detecting when users turn off CSA (Cisco Security Agent), find suspicious files, watching for specific MAC addresses, and special “Red Carpet Reports” that monitor executive systems to make sure they aren’t infected or compromised.
Lunchtime Customer Panel: Open Q&A
This included additional users such as Family Dollar Stores, Teletek, and a major financial services institution that uses Splunk for its web operations. When someone in the audience asked: “did Splunk replace anything in your environment?” one customer replied “Yes, not knowing what the hell is going on!” He added that now they get automatically notified if a network goes down, if there’s a config change and many other events they want to know about. Others added that Splunk has replaced homegrown scripts, MQ central logging and some web analytics. When asked about any surprises with Splunk the answers ranged from the unanticipated use cases, to the broad set of users, to Splunk Support–“One of the best vendor support teams we have. We have yet to have a problem that hasn’t been fixed quickly by the Support team.”