Search with Splunk

Hi everybody. Just want to take a minute to show you how Splunk lets you search for anything in your IT data. You get results right away. And the results are interactive so you can navigate through your data by clicking away!

A user is complaining about a problem with a web application. We get their IP address.

Say you work at a help desk. A customer calls up who is having problems with your website. You get their IP address and type it into Splunk.

Typeahead helps you complete the search.

Typeahead previews matches in your data

You get every event across every IT data source that includes this customer's IP address _instantaneously._

Instant results show you every event containing "10.2.1.44"

Now scroll through the results and check out the interactive timeline to see where you are.

If you decide to focus on a particular event, like the http GET events from the access logs here, you can just click on 'GET' right in your results to add it to your search.

You can quickly eliminate the successful web requests for this customer's IP by clicking on http success codes like '200' and '304 while holding down the alt key. Splunk updates your search with 'NOTs' for each code you click.

Freeform search is just part of the picture. The next search feature that's really great is fields. Splunk automatically extracts and names fields in your IT data.

Dynamically extracted fields move beyond freeform search

Let's see how easy it is. Click on the status field menu so you can break down the different http errors for this customer. The timeline bars highlight as you mouse over each status code and you can quickly see there was a cluster of 503 errors in just a few minutes.

Click on the 503 status code to filter results down since the customer's description of the problem sounded like a 500-series server error.

The timeline lets you zoom in and out by time. Bingo. You see this 503 problem has been happening intermittently about an hour apart for the past few hours.

Callout: Interact with the timeline

Narrow your search to the time of one of the 503 errors by clicking on the event's timestamp.

Correlate by time for individual events by clicking on the timestamp

Here's where it gets interesting. Change your search to look for all events that happened at the same second as one of the 503 errors.

Find everything that happened at the same moment

Now you quickly find errors where the web server couldn't connect to the appserver. To look at just those events, you hold down the ctrl key while clicking on the term 'connection_refused'.

Ctrl-click to replace your entire search (Cmd-click if you're on a Mac)

With a simple starting search and a few clicks you were able to verify a customer's problem report, establish the exact time of the web server errors and get to the root cause on the appserver. You didn't need access to the production servers or need to write any homegrown scripts to parse the data. Life is better with Splunk.