By Splunk November 16, 2008
That’s right! It’s “on fire” folks! Hotter than the sun! Burning its way into the thoughts and minds and data centers across the world.
Unfortunately, what I wanted to talk about today is not related to how hot Splunk is, but rather a very special and sometimes misunderstood character called “the pipe”. For most of us tech geek types, the pipe is our friend. We use it all the time at the command-line to make efficient use of our tools and our time. For non-techie folks, it may be more mysterious or intimidating concept, so I felt it might be a good topic to discuss and demonstrate just what it is and how to use it in the Splunk search box.
Also known as the vertical bar character, the pipe (|) allows you to create simple yet powerful ad-hoc Splunk searches. You might think of it as if it were an actual pipe where things flow into one end and then flow back out the opposite end. Within the context of Splunk searches, the “things” that flow in and out of the pipe are your IT events.
For example, let’s suppose that you searched for all events within your infrastructure that matched the word “error”, regardless of hosts or sourcetypes or timerange. After typing the word “error” within the search box, like this…
…and searching across all time, you would get back some events containing the word error.
But now let’s say you wanted to create a report on how many events containing the word “error” were occurring on each host or server oer all time. To do that, you could use a pipe character within your search to instruct Splunk to take the resulting events from your initial search and treat it as if it were inputs into a second command. Then you could type the second command right after the pipe. In this case, the second command would be a “chart” command and you could specify parameters to count raw events and split them out by each host, like this:
and your results page would switch over to Report mode and display as a graphical report, which would look like this (i.e. try this example on Splunk’s live email demo (http://email.demo.splunk.com)
Notice that the raw event counts for each host are displayed in descending order by default. Let’s say you wanted to display it in ascending order instead. To do this, let’s add one more pipe to this search and use the “sort” command to really demonstrate the piping concept.
Upon executing this search, the result is the same graphical report, but this time notice how the report table below it has changed to display in ascending order now, which will look similar to this in the Splunk email demo:
So this example uses two pipes. However, you could use more pipes if you needed to. Notice how the first part of this search is basically your initial search criteria, and then you pipe from left-to-right using search commands (and parameters) that essentially manipulate your event data as needed in a simple real time ad-hoc fashion. You can think of it in the following way:
<search_criteria> | <command1> <params> | <command2> <params> | <command3> <params>
The pipe character is simply the bridge between your initial search results and each step you take to further manipulate those results.
As another quick example, consider this search:
sourcetype=syslog | rex field=_raw "(?P<ip>\d+\.\d+\.\d+\.\d+)" | transaction fields="ip" | search duration > 100
Here I am using several pipe characters, again going from left to right, to process the initial search on all syslog events into only those events that contain an ip address matching my regular expression and then grouping those matching events where the duration of the entire grouped transaction was greater than 100 seconds.
In Splunk email demo, it will look similar to this:
So now you are probably thinking to yourself, “This piping concept is great, but what kind of commands are there and how can I use them in my searches more effectively?”
Well, one thing you could do is check out the search commands reference in our online docs, which explains the various search commands you can pipe into for creating the exact search or report you need. You could also just type the pipe (vertical bar) character at the end of your search string in the Splunk search box and the type-ahead feature will display all the search commands that are available.
Hint: after you type the pipe, type the a space, and then a letter to see each set of commands that begin with each specific letter, like this:
Once you find a command you want to learn more about, type out the command and you will see a nice in-line help page drop down below the search box containing instructions for using that command. For example, here is what you see after typing the pipe followed by the “chart” command:
It contains information about the parameters you can use with the command and the acceptable formats, etc. Very useful, if you ask me. Try it out and see for yourself. And leave a comment and let me know what you think.
Anyway, I hope this explanation of the pipe character makes sense.
I know that that most of your tech heads understand how to use the pipe character at the command line, but I figured it might be worth my time to discuss it here to allow all you other, more non-technical folks, a chance to realize how to use the pipe in your searches and leverage it to create some very flexible and powerful Splunk searches.
Thanks,
-maverick
----------------------------------------------------
Thanks!
Eric Gardner