If you are a long time Splunker, you might have your environment on an older Splunk version and haven’t taken the plunge to Splunk 6 yet. One of the common questions we get during upgrades is “how do I upgrade all my add-ons?” In Splunk 6, we made some fairly major changes to the Windows inputs, converting perfmon gathering and Windows event log gathering to modular inputs. For example, this means that perfmon is configured in inputs.conf instead of perfmon.conf, and the Windows event logs get an additional couple of slashes in the configuration inside of inputs.conf. How do you slowly upgrade all your universal forwarders from Splunk 5 to Splunk 6 without getting duplication of data and only having one copy of the TA around.
Splunk distributes a free Technology Add-on for Windows and most people use this as the basis for Windows data collection. Let’s use this as the example.
Step 1: Configure the Splunk Technology Add-on for Windows for Splunk 5.x Compatibility
Since we are starting with our Universal Forwarders at Splunk 5.x, we need to support Splunk 5.x in the Splunk Technology Add-on for Windows. This involves just two steps:
- Comment out all the Splunk 6.x stanzas in default/inputs.conf
- Create the Splunk 5.x stanzas in local/inputs.conf
We comment out the Splunk 6.x stanzas because Splunk 5.x doesn’t understand them. To do this, you need to edit the Splunk_TA_windows/default/inputs.conf file. I know this is not normally done and you will have to be careful when upgrading the TA during the process, but it’s really for the best. There will be several stanzas that start with perfmon:// and three that start with WinEventLog:// – you need to comment out the entire stanza. For example:
# [WinEventLog://Security] # disabled = 1 # start_from = oldest # current_only = 0 # checkpointInterval = 5
Now we can move our attention to creating the Splunk 5.x stanzas. The latest Splunk Technology Add-on for Windows does not have any Splunk 5.x stanzas in its default inputs.conf, but there is an inputs.conf.example that has the entries in there. Here is what your local inputs.conf should look like:
[WinEventLog:Application] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 [WinEventLog:Security] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 [WinEventLog:System] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 [script://$SPLUNK_HOME\bin\scripts\splunk-perfmon.path] interval = 3600 disabled = 0 source = PerformanceMonitor queue = winparsing
Note that I have not specified any indices here. You really want to create some indices for these and specify them here. Personally, I do most windows event logs into an index “winevents”, all my perfmon data into “perfmon” and finally, my security logs go into a special “security” index. Also, don’t forget that you are configuring perfmon counter gathering in perfmon.conf – not inputs.conf. The changes come later.
Now that you have the Splunk Technology Add-on for Windows configured for Splunk 5.x, you can push it out with the deployment server.
Step 2: Upgrade your Universal Forwarders to Splunk 6.0.3 or later
Now that you have your Splunk instances working nicely on Splunk 5.x, you can start upgrading your installations of the Universal Forwarder to the latest and greatest Universal Forwarder. I prefer a PowerShell version of the install, but other people have done System Center Configuration Manager (SCCM) or even GPO installations. However you do it, you can upgrade at your leisure. The Splunk 6 Universal Forwarder will upgrade all your Splunk 5.x Windows Inputs configurations to Splunk 6.x on the fly when the Universal Forwarder restarts, so you don’t have to worry about them.
Step 3: Transition to a Splunk 6.x Technology Add-on
Once the transition to the Splunk 6 Universal Forwarder is complete, you can transition to a configuration that supports just Splunk 6.x. This allows you to move to a more normal version of the Splunk_TA_windows that can be downloaded and directly implemented. Here are the steps:
- Back out the changes to default/inputs.conf – un-comment the Splunk 6 compatible stanzas
- Enable the Windows Event Logs in local/inputs.conf
- Convert your perfmon.conf to local/inputs.conf
The first step is easy – especially easy if you kept around a copy of the original default/inputs.conf. You want to uncomment everything that you previously commented out. The second step is equally easy – just replace the local/inputs.conf with the following:
[WinEventLog://Application] disabled = 0 [WinEventLog://Security] disabled = 0 [WinEventLog://System] disabled = 0
Finally, it’s likely that you did some changes to perfmon.conf, so you have a local/perfmon.conf file that needs to be changed. To do this, include the perfmon.conf in the inputs.conf file, then edit each perfmon stanza to include a double-slash and lower-case perfmon. The original perfmon.conf stanza would look like this:
[PERFMON:CPUTime] counters = % Processor Time;% User Time disabled = 1 instances = _Total interval = 10 object = Processor
The new inputs.conf stanza looks like this:
[perfmon://CPUTime] counters = % Processor Time;% User Time disabled = 1 instances = _Total interval = 10 object = Processor
Those bolded sections are the only thing that changed!
Once you have made your changes, you can push out the updated Splunk_TA_windows via your deployment server.
Now you have no reason to go ahead and upgrade your Windows Infrastructure to utilize the Splunk 6 Universal Forwarder.