Splunk for Xitive Xactions

Happy New Year and thanks to everyone who has been subscribing to my blog recently. I greatly appreciate it!

Every week people ask me to show them how to use Splunk to stitch together multiple events that might exist in different locations within different sources because, from an IT perspective, they are considered to be part of larger transaction groups. They tell me they want to know how to do this because the ability to trend against transitively-related events becomes very powerful in helping them understand the reality of IT operations and how efficiencies can be increased and costs can be more quickly and significantly reduced.

I thought I would share a quick example of how to do this using the transaction command.

Let’s start with a couple sample user activity log files containing some events that are related by multiple keys. Take a moment to study the two following sample activity log files and notice how the user and session key values are related between the files.

XU*** user event: user=maverick credentials cleared
XU*** user event: user=maverick authentication processing complete
XU*** user event: user=johndoe credentials NOT authenticated properly
XU*** user event: user=johndoe illegal login attempt
XS*** transaction event: user=johndoe session=2220 msg: failed login
XS*** transaction event: user=maverick session=1110 msg: successful login
XS*** some other event: session=1110 msg: maverick did something while logged in
XS*** still something else here: session=1110 msg: this user logged out now

Now, if you splunk these two files and specify the first one as sourcetype=xuser and the second as sourcetype=xsession, then executing the following search within the Splunk web user interface

     (sourcetype=xuser OR sourcetype=xsession)

then all of the results from both files are returned, which should look something like this:

Now, to make things a bit easier, let’s save our current search as a custom eventtype and call it “XACTION_EVENT” and then click the “Show Fields” option and search on this new eventtype, which will look like this:


and the search results will look like this:

Next, let’s say you want to correlate all of the events from either file that have matching session keys into one multi-line event (or transaction) grouping. To do this, you might submit the following search:

     eventtype="XACTION_EVENT"  | transaction fields="session" maxspan=1d maxpause=1d

which will take the xuser and xsession sourcetype events and group the ones containing matching session key values. The result set looks like this:

Now, let’s change the search to this:

     eventtype="XACTION_EVENT"  | transaction fields="user" maxspan=1d maxpause=1d

so that now it correlates the events by the user key values, instead of the session key values.

Now, notice the XU and XS characters appearing at the beginning of each event indicating that you are finding matching key values that appear within BOTH sourcetypes. Pretty cool, huh?

But wait, it gets better!

Before I show you just how much better, I want you to scroll back up and look that the resulting events for each of the separate xuser and xsession key searches one more time and this time notice that some of the events have a xsession key AND an xuser key appearing within the same event. Don’t you think it would be more powerful to use those specific events as a kind of ‘bridge’ to correlate all of the key values together into one big transaction? I mean, after all, that’s probably what you really want to know is how everything relates all at once….within one big final truly transitive transactional story, right?

Well, fortunately the Splunk transaction command can do this in a very simple and clean way. And if you paid close attention to how we’ve been using the transaction command, you will see that there is a fields parameter. Notice that the fields parameter is plural. The reason it’s plural is because you can specify more than one key value to match on. And let me remind you, there is no other technology on earth right now that offers a correlation capability as powerful and as easy to use as Splunk.

Okay, so let’s do this transitive transactional search by changing our last search string to include both user and session key values within the fields parameter and separating them with a comma and adding the connected param set to “f”, like this:

     eventtype="XACTION_EVENT"  | transaction fields="session,user" connected=f maxspan=1d maxpause=1d

and now the results below are way more informative and much better at painting the complete transitive transactional picture:

Now, I don’t know if you find this type of transitive transaction analysis useful or not, but my experience with helping my Splunk customers use this command effectively leads me to believe that you do.

Before I end this blog post, I want to make you aware of two additional and powerful artifacts of using the transaction command, which are the duration and linecount fields. Since you are using a command that groups separate events together, the time of the total transaction to take place (i.e. the duration) as well as the number of total events appearing within the transaction (i.e. the lines) are automatically calculated for you.

Therefore, if you take our example transactions shown above and determined (for whatever reason) that any transaction that has a duration greater than three seconds or any transaction containing less than five lines is a bad transaction, you could enhance your search to consider those conditions, like this:

eventtype="XACTION_EVENT"  | transaction fields="session,user"  connected=f  maxspan=1d maxpause=1d | search duration > 3

or this:

eventtype="XACTION_EVENT"  | transaction fields="session,user"  connected=f  maxspan=1d maxpause=1d | search linecount < 5

...and, of course, the resulting effect would be a filtered list of transactions matching your conditions.

Make sense?

BTW, for a few more advanced examples of how to use the transaction command more effectively, see David Carasso's blog post.

Anyway, I hope you are getting a good feel now on how to use the transaction command and I wonder if you are now getting some ideas how you might be able to leverage this very easy yet powerful search command to correlate your events now.

If you do, please leave a comment below and let us know about it. We are always looking for better ways to
   use Splunk for everything!

Eric Gardner

Posted by