By Splunk October 28, 2013
Continuing our delve into the new Microsoft features that were introduced in Splunk Enterprise 6, let’s take a look at performance metrics. We had a couple of really cool features here, dealing with zero values, object recognition, and multi-kv.
The first has to do with how we record zero values. By default, the perfmon data input does not record zero values. This has implications if you want to do things like 95th percentile as I discussed in this blog post. In that blog post, I suggested you might want to (shudder) alter the splunk-perfmon.path file to provide a command line argument. The developers obviously read my blog and decided to make it easier. Now you can just add a parameter to your inputs.conf stanza, like this:
[perfmon://MemoryStats] object = Memory counters = * instances = * interval = 60 showZeroValue = 1
The important bit is that last parameter. If it is set to 1, then the zeros are recorded for each counter/instance combination. The default is the old behavior (i.e. showZeroValue = 0).
Our second feature is the ability to include regular expressions in the object. Let’s take an example of Microsoft SQL Server. If you create a named instance, the SQL Server will maintain counters that look like this:
MSSQL$INSTANCE:Transactions
The INSTANCE is the instance name. This presents a problem when you are trying to provide a generic add-on that reads performance data for Microsoft SQL Server. Our developers came to the rescur again – we can now specify a regular expression. If the object name matches exactly then the regular expression is never consulted. However, if there is no exact match, the object turns into a regular expression match and all objects matching the regular expression are added to the monitoring list. You can do things like this:
[perfmon://MSSQL:Transactions] object = MSSQL[^:]*:Transactions counters = * instances = * interval = 60 showZeroValue = 1
Our final update to perfmon collection deals with an output change. Normally, you will get events that look like this:
10/15/2013 13:39:04.044 -0700 collection=Processor object=Processor counter="% Processor Time" instance=_Total Value=2.2639598290484009
This is good, but what if you have a lot of perfmon to gather – thousands of events? This is not exactly the most efficient method of storing the data. Let’s take a look at an example for an alternate mechanism:
[perfmon://Memory] object = Memory counters = * interval = 60 mode = multikv showZeroValue = 1
That new parameter – mode – switches between multikv and simple outputs. A multikv entry looks like this:
0 33.85326809566461 7734845440 7975034880 14696304640 0 0 3.982737423019366 29.870530672645245 1.991368711509683 1.991368711509683 1.991368711509683 0 299855872 94121984 0 312774 180313 33556298 66895872 89628672 290975744 0 9474048 10956800 8876032 66895872 54.26557951373551 7553560 7376 0 1615249408 1789952 31371264 6088224768 0 14400
Ok – it isn’t the most readable event in the world. But try this:
sourcetype=PerfmonMk:Memory| table Available_MBytes,Committed_Bytes,"Demand_Zero_Faults/sec"
All of a sudden, the information in this table is decoded for you. The format is much more compressed and that makes it take up less license room. Note that the source type has changed (it’s now prefixed with PerfmonMk instead of Perfmon), but other than that – it’s good. What’s more, if you do this to something with instances (for example, LogicalDisk), you still get one event per instance and the first element is the instance name:
C: 82.221877626972017 100744 0 0.893482164153651 0.0089348216415365105 0 0 0.893482164153651 0.0089348216415365105 0.00092000519619113603 0 0.00092000519619113603 9.8839878234777032 0 9.8839878234777032 64775.70259994347 0 64775.70259994347 6553.6000000000004 0 6553.6000000000004 97.408979418055637 1.9767975646955405
All of this makes it very possible to get more performance data to correlate against your other event data and store it in the most efficient way possible. Of course, you do need to upgrade your Universal Forwarder to 6.0, so be aware of that small wrinkle.