By Splunk January 04, 2016
Recently, after working with a Business Team and the Technical Team that supports them at one of our customers, we had an opportunity to witness, first-hand, the struggles each of us in IT Operations has felt at some point. In the words of a wise man, they were experiencing “Mutual Mystification.” The Business Team was concerned with a Product Line and multiple business processes. The Technical Team was trying to translate the technical details of the underlying micro services and how they were related to the Product Line. Within the first 15 minutes, both teams were becoming frustrated and didn’t feel like they were communicating well. This led to a consensus that the meeting was not going to achieve the results both groups were expecting.
The groups understood that they were responsible for each other’s success, but were not sure how to come to an agreement. As another wise man said, “a picture is worth a thousand words.” We decided to show them an example, a picture of how the two teams might work together. Drum roll for the good stuff to come!
We brought the two teams back together, and introduced them to Splunk’s IT Service Intelligence (ITSI). They happened to be a financial institution, so we started with an example of how the business could see the details in “real time” for the indicators they were concerned about; Health of the Services, Revenue by Generation Point, and Interactions with the System (User Count). The Business Team said their VPs needed to be able to see these types of KPIs at all times.
Here is the example we used, the first of two Glass Table visualizations in ITSI:
The Business Team was stoked! However, the Technical Team was full of questions, “How did Splunk get these numbers, what happens when they were not at a certain threshold, etc.?”
These questions (and many others not asked) were completely valid, however we answered their question by asking Technical Team what was really important to them? The team responded that their most important troubleshooting resource is network diagrams. We asked if they would be comfortable seeing the details in something like this?
This launched another set of completely valid questions.
How was this service built?
What does (that component of the service) do?
Where do those components live?
Why does the service do that?
Before we could start to answer, individuals from both teams started to help answer each others questions using the respective visual examples as aides. ITSI had created a communication medium that allowed each team to acknowledge and understand what was most important to each other. More importantly, the two teams started co-developing the Key Performance Indicators they mutually thought were beneficial, as well as ones that only one group may use. Once they completed their list of top 10 KPI’s the teams asked, “How will this happen?”
As we explained to these teams, the flexibility of IT Service Intelligence is one of the primary benefits – the ability to take in the list of Entities (Hosts, Network Devices, just about any unique, well.. Entity) and use it to create those KPIs. We shared with each team how to create a KPI from their unique perspectives (spoiler – anything you can return in a Splunk search can be a KPI). The Technology Team wanted to see the individual Entity performance where as the Business Team only wanted to see the aggregates. Yup, we confirmed: this too can be done with Splunk’s IT Service Intelligence.
The last piece of the instructive session included the introduction of Adaptive Thresholding, which was really received well. We upped the ante just a bit by saying, “But wait! If you like that about IT Service Intelligence, you might also like that it includes Anomaly Detection!”
Why is this SO important?
- The ability to understand Business Services and their interaction with Technical Services came from the availability to literally “see” the mapping through a Glass Table.
- These teams could determine, on the fly, what KPI’s were most important to them, without having to go through a cumbersome process to relate CI’s to Services.
- The ability to harness Adaptive Thresholding (Machine Learning) and Anomaly allows the teams to take some of the guesswork out of what they should be worried about.
- Most importantly, the Business and Technology teams could now quickly isolate Revenue Impacting Events, and provide insight into “why” in near real-time with “feedforward” (how to improve the future) not “feedback” (dig a hole investigating hoping that the nuggets found will fill the hole you have dug).
As a Technologist, I leave you with a challenge to work with the teams that you are supporting. Take the time to see how similar the KPIs and Service Level Agreements between your Business and Technical groups really are. By all means spin up an IT Service Intelligence Test Drive to use as an example and leverage ITSI’s capabilities to showcase the four values above and see how the combined teams respond.
Happy Splunking!!!!
----------------------------------------------------
Thanks!
Nate Smalley