Windows Host Monitoring in Splunk 6

Splunk 6 is out! While the most flashy and awesome of features rightly got their day in the sun at the recent .conf 2013, there is lots to love in there for Windows admins as well. I’m going to spend the next few weeks explaining in detail what some of those things are and how you can make use of them. First up – Windows Host Monitoring is one of a bevy of new data inputs available in the Splunk 6 Universal Forwarder.

When I design a new Microsoft related app, inevitably I need some indication of a what sort of host is being monitored. Such information includes the hardware (memory and processor) and operating system. I also need an indication of services running so I can do service monitoring, disk space utilization and so on. All of this information used to be gathered via either PowerShell or WMI, depending on my mood at the time I wrote the app. It was inconsistent at best and didn’t allow for generic monitoring of Windows hosts.

In Splunk 6, we fixed that. There is a new modular input that is distributed with the Splunk 6 Universal Forwarder called WinHostMon. You can configure it like this within inputs.conf:

[WinHostMon://computer]
interval = -1
type = computer
index = windows

The interval of -1 indicates that the host monitor should gather the information once (per reboot) and then never again, so you have to take account of this in your searches and lookups. There are a whole series of things you can retrieve, but you can only retrieve one thing per stanza. When it sends an event, the event looks like this:

Type=Computer
Name="DC"
Domain="splk.com"
Manufacturer="Microsoft Corporation"
Model="Virtual Machine"

Repeat the stanza for type=operatingSystem (with a new name) to get the OS information.

I don’t want to be searching over all-time every time I want to join an event stream to get windows host information – that’s bad. The normal thing to do would be to turn this into a lookup – even at 100K hosts, a lookup won’t be terrible. If you do have 100K endpoints then you already have methods of dealing with large lookups! My search to convert the events into a lookup is as follows:

index=windows (Type=Computer OR Type=OperatingSystem)|stats latest(_time) as _time,latest(OS) as OS,latest(Architecture) as Architecture,latest(Version) as Version,latest(BuildNumber) as BuildNumber,latest(Name) as Name,latest(Domain) as Domain,latest(Manufacturer) as Manufacturer,latest(Model) as Model by host|inputlookup append=T WinHosts.csv|sort _time|stats latest(_time) as _time,latest(OS) as OS,latest(Architecture) as Architecture,latest(Version) as Version,latest(BuildNumber) as BuildNumber,latest(Name) as Name,latest(Domain) as Domain,latest(Manufacturer) as Manufacturer,latest(Model) as Model by host|outputlookup WinHosts.csv

The host is our primary key here. if we were gathering the informationevery X hours, then we would not need the second stats call – we could just call the first stats command and then pipe that to the outputlookup. We have old events to consider, so we create a table of all the new events that have come in within the last X hours, then append the old results, and finally re-do the stats command so we can get the very latest information before pushing it back out to the CSV file that backs our lookup. Set this up as a saved search. You can then add the lookup file as a lookup via a transforms.conf file. If you want this available in other apps, don’t forget to export both the lookup CSV file and the transform entry.

There are other WinHostMon methods for getting service status, processes, installed drivers and installed applications on a single host. This modular input basically unifies all of our differing methods for getting the data under one roof without the need for dealing with multiple WMI calls or installing additional apps like the SA-ModularInput-PowerShell app. As such, it should become part of your toolbox.

Not all apps will work with a Splunk 6 universal forwarder as yet (most notably, all the Microsoft apps are still relying on Splunk 5) so they won’t take advantage of the new universal forwarder features. However, you can bet on us utilizing this functionality in all future development.