We get quite a few requests on how to run two Splunk Universal Forwarders on the same Windows host. Why would you do this? The primary reason is that you have a lab environment and want to compare one version of Splunk to another during an evaluation of a new version. You may also have two sets of files you need to ingest into Splunk and the files have differing access permissions such that Splunk needs to run as different users. It’s really an edge case and definitely not something you want to generally do in production.
In Linux, this is a fairly simple process – just install to a different directory and change the ports and you are done. So what about Windows? The Service Manager kicks off the Splunk processes, so it’s not quite as simple. There are a few extra steps needed to tell the Service Manager about the new locations.
WARNING: WE ARE DISCUSSING AN UNSUPPORTED CONFIGURATION.
Don’t expect a sympathetic ear from our support guys when you are using this configuration. They may be sympathetic, but they won’t be able to assist. Most importantly, DO NOT RUN THIS IN PRODUCTION!
Also, there are limitations. You can only run one copy of the driver-related modular inputs (regmon, netmon and perhaps the most serious – MonitorNoHandle). This means these inputs can only appear in one of the universal forwarders. You will get weird and completely random errors and crashes if you break this rule.
So, now we have that out of the way, how do you do it?
Step 1 – Install the Splunk Universal Forwarder as normal.
Install your first Splunk Universal Forwarder just as you would normally. Go through the GUI or use your silent installer to install the first one. We will be adjusting it as needed. Since we are going to be moving it, you may want to specify an alternate directory for installation.
Step 2 – Stop your Splunk Universal Forwarder.
All these changes need to be made without the Splunk Universal Forwarder running.
Step 3 – Move the installation directory (if necessary).
If you are altering an existing environment for the additional forwarder, you are going to have to move it to a new location. This is simply a Move-Item in PowerShell. If you specified an alternate directory during the installation, this is done already.
Step 4 – Change the Splunk Launch Configuration.
The Splunk launch configuration is stored in $SPLUNK_HOME\etc in a file called splunk-launch.conf. There are two lines you need to alter:
SPLUNK_HOME=C:\Program Files\SplunkUniversalForwarder2 SPLUNK_SERVER_NAME=SplunkForwarder2
The SPLUNK_HOME points to your new directory. The SPLUNK_SERVER_NAME is the name of the Service within the Services control panel.
Step 5 – Create the new Service
Open up an elevated cmd prompt (using Run As Administrator) and type the following:
sc create SplunkForwarder2 binpath= "\"C:\Program Files\SplunkUniversalForwarder2\bin\splunkd.exe\" service”
Note that the path is the path to our new directory and the name of the service is the name we set in the splunk-launch.conf file.
Keep this elevated cmd prompt around – we are going to continue using it.
Step 6 – Delete the old service
In that same cmd prompt, type the following:
sc delete SplunkForwarder
This removes the old service so you can install the second instance.
Step 7 – Change the Admin Port
As with any TCP-based service, you can’t have two services listening on the same port. You need to change one. In this case, we need to change one setting. We can do this in that same elevated cmd window:
cd “C:\Program Files\SplunkUniversalForwarder2\bin” .\splunk.exe set splunkd-port 8090
Step 8 – Start the Service
Since we have the elevated cmd window open:
Just as normal, this will start the service for you.
Step 9 – Install your second Splunk Universal Forwarder as normal.
Now that we have created the alternate Splunk Universal Forwarder configuration, we can now install the regular Universal Forwarder.
I have just a few final notes. Firstly, you will remember that the maximum throughput of a universal forwarder is set in the limits.conf file and is limited to 256Kbps. This is a per-service limit, so 2 universal forwarders = 512Kbps. Also, you will see twice the memory footprint and twice the CPU consumption.
Finally – THIS IS UNSUPPORTED.
NO, REALLY, I AM NOT KIDDING – UNSUPPORTED.
Did I mention it’s unsupported?