From Switchboards to Dashboards: Communications Leader Finds New Solutions
Since 1930, CenturyLink has grown from a rural telephone carrier to a consumer, business and enterprise broadband, security, IT services and cloud provider across the U.S. and around the world. Along the way, it has faced challenges. Since deploying Splunk Enterprise seven years ago and Splunk IT Service Intelligence (ITSI) recently, the company has seen benefits including:
- Successes in mission-critical monitoring, providing executive-level visibility and improved resolution time
- Fast time to value and return on investment (ROI)
- Steep decreases in mean-time-to-identify (MTTI) and mean-time-to-resolution (MTTR)
SPLUNK USE CASES
- Assembling information from a variety of silos and legacy apps
- Need for executive-level visibility into IT and business operations
- Call centers and other divisions turning to IT for help instead of assessing their own problems independently
- Much faster time to identify and resolve issues
- Epic return on investment
- Family of solutions and add-ons for monitoring and analysis
Due to acquisitions, CenturyLink had a need for quick insight into in-house and third-party applications. Previously, something as simple as a new code release for the billing system could cause IT to spend a week running Perl scripts to retrieve log data. “The guy who was trying to do that,” recalls Jeff Burchfield, senior lead software developer, “used to sit right next to me, and you could tell he was frustrated when he had to do it.” After installing Splunk Enterprise, he says, “I had one of the reports done in under an hour. Then we went to proof of concept. Our goal was to produce 15 reports in three days, and I had everything done on day one.”
Today, the Splunk platform is used for business and Operational Intelligence, DevOps, project management analysis drawing on CA Agile Central, pulling information from the company data warehouse—anywhere, Burchfield explains, “for rapid triaging the IT environment to provide critical executive-level visibility. If you name it, we’re probably doing it.”
“With Splunk ITSI, much of this becomes turnkey. Some of the built-in modules within Splunk ITSI allow us to realize an immediate benefit.”
Senior Lead Software Developer, CenturyLink
Keeping tabs on everything—including the bottom line
It’s hard to find a place at the company where the drive for mission-critical monitoring isn’t changing operational workflows. One component of the billing system has multiple servers with a vast number of service layers deployed by each application. “Just watching performance and errors across that one piece of the billing system can be daunting for DevOps personnel, but Splunk centralizes all that activity in one place,” Burchfield says.
Meanwhile, rather than reinvent the wheel, CenturyLink is leaving some legacy monitoring systems in place but bringing them under the Splunk platform umbrella to centralize the siloed systems by extracting activity from them. Burchfield says strategic applications within the IT organization, “keep growing because every service talks to multiple other applications, with growing, intricate dependencies.”
Has it made a difference? Comparing one month in 2017 to the same month a year earlier, the improved monitoring has yielded a decrease in IT outages, MTTI and MTTR.
Empowering call centers and customers
CenturyLink’s call centers were challenged after the acquisitions, not because its systems weren’t up to speed but because its call mix and the time to complete or modify an order had changed.
With Splunk Enterprise in place, instead of calling IT, the centers can use real-time dashboards to see how they are performing compared to other centers. According to Tim Kerrigan, a manager in the Systems Engineering Group, “They can ask themselves whether anything has changed procedurally before we get 30 to 50 people on a triage call.”
Now, Splunk software is providing CenturyLink with business insights, starting with orders taken in Salesforce.com and continuing through systems that don’t produce their own logs. “It’s now generating information that customers are able to see via API calls to other systems,” Kerrigan adds.
“We saw the business value in the data that was within the IT systems. We started communicating that out to the business partners and that’s how it started growing from there, using [Splunk Enterprise] both as an IT operational tool and also as a way to get to more business-specific data that was too costly to be obtained another way. It was there, and now they can get to it easily.”
Manager, Systems Engineering Group, CenturyLink
Getting granular and looking ahead
With Splunk Enterprise deployed across a booming population of users in almost every CenturyLink department, the company recently added Splunk ITSI to its arsenal to aggregate an array of key performance indicators and granular-level detail for executive review. Apps to receive the Splunk ITSI treatment so far include XenApp and a suite of other Citrix desktop and server products, as well as a critical API that allows multiple external applications to interface with the billing system.
Both Splunk solutions bring what Burchfield calls operational visibility—being able to put a massive amount of data into something concise that an executive can review quickly. “With Splunk ITSI, much of this becomes turnkey. Some of the builtin modules within Splunk ITSI allow us to realize an immediate benefit.”
Moving forward, CenturyLink is deploying a connectivity roadmap to Splunk Analytics for Hadoop and other big data repositories, anticipating reports and dashboards for more in-house apps and access to big data.
“CenturyLink puts a heavy focus on customer experience, and Splunk helps ensure our end users are always satisfied with their service team interactions. Splunk Enterprise and Splunk IT Service Intelligence give us real-time, executive-level visibility into the health of our IT applications. This ensures that our representatives can get the information that matters most to each customer at all times. Splunk has helped us raise the bar in customer service by improving time to resolution, resource optimization and executive-level communication.”
Manager, Systems Engineering Group, CenturyLink