Documentation:
3.0.2
This page last updated: 01/30/08 01:01pm
Enable forwarding and receiving
Receiving
via SplunkWeb
- Navigate to SplunkWeb on the server that will be receiving data for indexing.
- Click the Admin button in the upper right hand corner of the SplunkWeb interface.
- Select the Distributed tab.
- Click Receive Data.
To begin receiving data:
- Set the radio button to Yes.
- Specify the port that you want Splunk to listen on. This will also be the port that Splunk instances will use to forward data to this server.
- Click the Save button to commit the configuration. You will need to restart the server for your changes to take effect.
You can enable receiving from the Splunk CLI. To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.
First, log in:
./splunk login Splunk username: admin Password:
To enable receiving:
# ./splunk enable listen 42099 -auth admin:changeme Listening for Splunk data on TCP port 42099.
To disable receiving:
# ./splunk disable listen -auth admin:changeme No longer listening for Splunk TCP data. You need to restart the Splunk Server for your changes to take effect.
- Navigate to the SplunkWeb interface on the server that will be forwarding data for indexing.
- Click the Admin button in the upper right hand corner of the SplunkWeb interface,
- Select the Distributed tab,
- Click Forward Data,
To begin forwarding data:
- Set the Forward data to other Splunk Servers? radio button to Yes.
- Specify whether you want to keep a copy of the data local in addition to forwarding or just forward. If you keep a copy locally, this will allow you to search from the local server, but it will take up space as well.
- Specify the Splunk server(s) and port number to which you will forward data. The port number should be the same one that you specified when you configured receiving.
- Click the Save button to commit the configuration. You will need to restart the server for your changes to take effect.
You can enable forwarding from the Splunk CLI. Navigate to your $SPLUNK_HOME/bin directory on the server that will be forwarding data for indexing and log in to the CLI. You can also add Splunk to your path and use the splunk command.
./splunk login Splunk username: admin Password:
To enable forwarding:
# ./splunk add forward-server 10.2.2.2:9999 -auth admin:changeme Added Splunk-2-Splunk forwarding to: 10.2.2.2:9999
To disable forwarding:
# ./splunk remove forward-server 10.2.2.2:9999 -auth admin:changeme Stopped Splunk-2-Splunk forwarding to: 10.2.2.2:9999 You need to restart the Splunk Server for your changes to take effect.
If you have installed Splunk on the server that is generating event data, you will probably elect to forward the events to another Splunk server for indexing. This decreases the workload on the forwarding server. To further reduce the work performed on the forwarding side, you may want to set up a lite-weight forwarder. With a lite weight forwarder, all optional processing is moved to the indexing server. Specifically when a server is made lite-weight, it is modified to:
- turn off Splunk internal logging (via $SPLUNK_HOME/etc/bundles/local/inputs.conf).
- eliminate batch, exec, fifo, tcp, and udp input modules from splunkd (which decreases memory utilization).
- replace splunkd.xml with splunkd.xml.forwarder.
With lite-weight forwarding, timestamp and host processing still happen on the forwarding side so that this data is accurate.
You can use Splunk's CLI to turn lite-weight forwarding on and off. To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.
To enable lite-weight forwarding, use this CLI command on the forwarding server:
./splunk set server-type forwarder
To disable lite-weight forwarding, use this CLI command on the forwarding server:
./splunk set server-type default
If you would like to be able to use a scripted input on your lightweight forwarder, you need to re-enable the exec processor. To do this, go into $SPLUNK_HOME/etc/modules/input/exec and copy the existing config.xml.default to config.xml. This enables the module and on restart it will be inserted into the pipeline.
Transplant Parsing from ForwarderThe default forwarding set-up, even with lite-weight forwarders, is to do some parsing (i.e. character encoding, timestamp extraction, line-merging) on the forwarder and then send the parsed data to be indexed. Although parsing is not nearly as resource intensive as indexing, you may still want to avoid doing it on the forwarder.
To disable parsing on the forwarder, inputs.conf must specify (for each input, or as a global setting):
queue=indexQueue
On the receiving side, the default for splunktcp input is to skip parsing and send data directly to be indexed. To change this, inputs.conf must specify:
[splunktcp://<remote server>:<port>] queue=parsingQueue
Comments
I hope the new section "Transplanting Parsing from Forwarder" is clear and will help forwarder deployments in very resource-sensitive environments
Posted by goldburtd on Jan 10 2008, 10:35am
Yup it is mentioned right above in the second bullet:
"eliminate batch, exec, fifo, tcp, and udp input modules from splunkd (which decreases memory utilization)."
Posted by m@ on Jan 09 2008, 9:03am
FYI, using lite-weight forwarding will disable named pipe (FIFO queue) as an input option.
To enable FIFO queue, take splunk out of server-type forwarder with the following command:
/opt/splunk/bin/splunk set server-type default
and restart Splunk.
Other modules and functionalities are most certainly affected and disabled by lite-forwarding, but I don't have any other data to report.
Posted by bebo on Jan 08 2008, 6:34pm
Added set server-type forwarder command and listed its effect.
Posted by kathleen on Sep 04 2007, 5:58pm
Is this all thats necessary for a lightweight forwarding setup? Could use a more complete description of what actually is going on when you set this up.
Posted by goldburtd on Aug 22 2007, 4:02am