This document last updated: 07/22/08 01:07pm

Print Admin Manual

How Splunk Works

Overview of Splunk

Splunk is search software for any type of data. Learn more about how Splunk works by reading through this intro page. You'll find many links here for installing, configuring and customizing your Splunk installation.

Configuration options

Most of Splunk's configurations can be reached through Splunk Web. You can use the CLI as well. Splunk also comes with configuration files for advanced customization.

Note: Many of Splunk's settings can be configured in multiple ways. Decide which works best for your setup. Some advanced features may only be configured through configuration files.

Installation and upgrade

Installing Splunk is easy and fast. Here are instructions for installing, upgrading, or backing up an existing copy.

Data inputs

Splunk can receive data in a variety of ways. Each configuration change can be affected via:

Read on for a brief description of each input type.

Note: For a more in-depth description of inputs, read how input configuration works.

Distributed data

Configure distributed inputs and outputs across your network. Send data between one Splunk instance and another, or third party software. For an overview on all the available configuration options, see How data distribution works.

Indexing

Splunk takes all data from inputs and sends it to an indexing pipeline. Data is then broken up into separate events via segmentation rules. Most data is segmented and timestamped correctly. However, you may wish to configure Splunk to index your data in particular ways. Learn more about how indexing works.

Here are some things you might want to consider:

Configuration for indexing is set mostly through props.conf and transforms.conf

Fields

Fields are a useful aspect of Splunk's search interface. You can use Splunk's built-in fields that are enabled by default. Here's a list of Splunk's default fields, including links to more in-depth documentation:

You can also create your own fields. Custom fields are useful for:

To learn more about creating custom fields, see how fields work.

Search

Splunk's search interface is useful for tracking down different aspects of your data. Here are a few things you can do with your searches:

For a more detailed overview of search, see how search works.

Distributed search

In a distributed set up, you may want to search across multiple instances of Splunk. Enable distributed search to federate searches across your entire Splunk deployment. Read more about how distributed search works.

Security

Secure your Splunk Server with the following security configuration options. Here's a brief overview of the available features. For a more detailed overview, see security options.

Authentication

Splunk includes several authentication options, including:

Audit

Splunk 3.2 includes new audit features. Use the following options to enable separate auditing configurations:

Data management

Splunk Servers often index large amounts of data each day. You may want to enable advanced settings to handle the following data management scenarios.

NOTE: Many data management settings are enabled on a per-index basis, using indexes.conf. To learn more about indexes, see how indexes work.

Deployment server

In a distributed set up, enable one or more Splunk instances as deployment servers. A deployment server pushes out configuration changes to other Splunk instances.

For a complete overview of all deployment options, read the Deployment manual. For instructions on configuring and enabling the deployment server and clients, read the Admin manual section on the deployment server.

Performance tuning

The following options help you tune Splunk's performance for your environment. Depending on your system and requirements, you may want to change one or more of the following settings:

A more in-depth overview of performance tuning options is available here.

Configuration files

Many of Splunk's advanced configurations and customizations are available only through configuration files. Create configurations by copying files into a custom bundle directory. Learn more about bundle directories and configuring bundle directories.

Applications

Applications are directories of configuration files with specific purposes. For example, Splunk-2-Netcool. Configure your own applications by following these instructions.

You can also share your configuration file directories as applications with the Splunk community on SplunkBase.

Customization

Pimp your Splunk! Everybody's data is a little bit different. Maybe you want to set custom configurations for the system you're running Splunk on. Here are options for personalizing your Splunk instance.

Splunk Web appearance

Change various aspects of Splunk Web's appearance:

Extend Splunk

Splunk includes a REST API. Read the Developer's Guide to learn more about the REST API. To configure additional REST endpoints, use restmap.conf.

Troubleshooting

If there's something you need help with, even after reading the documentation, contact Splunk support.

If there's a feature you don't see here that you want included, file an enhancement request with Splunk support.

We're always interested in your feedback.

Getting Started

Start Splunk

This topic serves only as a brief instruction to starting Splunk. If you are new to Splunk, we recommending reviewing the User Manual first.

Before you start

Before starting Splunk, install the software. Refer to the Installation Manual for system requirements and step-by-step instructions. Make sure you install the correct version of Splunk and that you are installing on a supported filesystem.

Note: If you are upgrading or migrating to 3.2 from a 3.0 or later, refer to the upgrade instructions before continuing.

Start Splunk on non-Windows platforms

Splunk's command line interface is located in $SPLUNK_HOME/bin/. $SPLUNK_HOME refers to the path you installed under. Navigate to this location and run the following command:

# ./splunk start

You must accept Splunk's EULA the first time you start Splunk after a new installation. To bypass this step, start Splunk and accept the license in one step:

# ./splunk start --accept-license

NOTE: There are two dashes before the accept-license option.

Start Splunk on Windows

On Windows, Splunk is installed by default into \Program Files\Splunk

Start and stop the following Splunk processes via the Windows Services Manager:

You can also start, stop, and restart both processes at once by going to \Program Files\Splunk\bin and typing
# splunk.exe [start|stop|restart]

Load Splunk Web in your browser

Navigate to:

http://mysplunkhost:8000

Use whatever host and port you chose during installation.

The first time you login to Splunk with an Enterprise license, use username admin and password changeme. Splunk with a free license does not have access controls.

Administration basics

The $SPLUNK_HOME variable refers to the top level directory of your installation. By default, this is /opt/splunk/.

Add Splunk to your shell path

To save a lot of typing, set a SPLUNK_HOME environment variable and add $SPLUNK_HOME/bin to your shell's path. The example below works for bash users who accepted the default installation location. Use the correct syntax and path for your own installation.

# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

Splunk's CLI

Splunk's command line interface is located in $SPLUNK_HOME/bin/. If you have exported the path and environment variables (as explained above), you can use the splunk command as follows:

# splunk [action] [object] [-parameter value] ....

If you haven't set an environment variable, navigate to $SPLUNK_HOME/bin/ and run commands as follows:

#./splunk [action] [object] [-parameter value] ....

For general help, type:

# splunk help

For a list of commands and options, type:

# splunk help commands

For Splunk with an Enterprise license, administration commands must be authenticated with a username and password. To authenticate for an entire session, type:

# splunk login

This command prompts you for a Splunk username and password. Use the same username and password for the CLI and Splunk Web. By default, the login is set to admin and the password is changeme.

Logout at any time by typing:

# splunk logout

To authenticate a single command, use the -auth parameter:

# splunk search foo -auth username:password

Note: the -auth string must be the last term in the CLI command.

Start/stop Splunk, check status

Ensure that you have added Splunk to your server host's path (as explained above, in "Adding Splunk to your shell path"). Otherwise you must use the ./splunk command.

Start the Server

From a shell prompt on the Splunk sever host, run this command:

# splunk start

Alternately, start either splunkd (to load back-end configuration) or Splunk Web (to load web configuration):

# splunk start splunkd

# splunk start splunkweb

Or restart Splunk (splunkd or Splunk Web) by running:

# splunk restart

# splunk restart splunkd

# splunk restart splunkweb

Stop the Server

To shut down Splunk, run this command:

# splunk stop

Also available for splunkd and Splunk Web:

# splunk stop splunkd

# splunk stop splunkweb

Check if Splunk is running

To check if Splunk is running, type this command at the shell prompt on the sever host:

# splunk status

You should see this output:

splunkd is running (PID: 3162).
splunk helpers are running (PIDs: 3164).
splunkweb is running (PID: 3216).

Or you can use ps to check for running Splunk processes:

# ps aux | grep splunk | grep -v grep

Solaris users, type -ef instead of aux:

# ps -ef | grep splunk | grep -v grep

Help

Help is available in several forms.

Help Options

Change defaults

Changing the admin default password

Splunk with an Enterprise license has a default administration account and password. It is highly recommended that you change the default. You can do this via Splunk's CLI or Splunk Web.

Note: CLI commands assume you have set a Splunk environment variable. If you have not, navigate to $SPLUNK_HOME/bin and run the ./splunk command.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/users.jpg

via Splunk CLI

The Splunk CLI command is:

# splunk edit user

Note: You must authenticate with the existing password before it can be changed. Log into Splunk via the CLI or use the -auth parameter.

For example:

# splunk edit user admin -password foo -auth admin:changeme

This command changes the admin password from changeme to foo.

Changing network ports

Splunk uses two ports. They default to:

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/ports.jpg

via Splunk CLI

To change the port settings via the Splunk CLI, use the CLI command set.

# splunk set web-port 9000

This command sets the Splunk Web port to 9000.

# splunk set splunkd-port 9089

This command sets the splunkd port to 9089.

Changing the default Splunk server name

The Splunk server name setting controls both the name displayed within Splunk Web and the name sent to other Splunk Servers in a distributed setting.

The default name is taken from either the DNS or IP address of the Splunk Server host.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/ports.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set servername foo

This command sets the servername to foo.

Changing the datastore location

The datastore is the top-level directory where the Splunk Server stores all indexed data, user accounts, and working files.

Note: If you change this directory, the server does not migrate old datastore files. Instead, it starts over again at the new location.

To migrate your data to another directory follow the instructions in Move an index.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/datastore.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set datastore-dir /var/splunk/

This command sets the datastore directory to /var/splunk/.

Set minimum free disk space

The minimum free disk space setting controls how low disk space in the datastore location can fall before Splunk stops indexing.

Splunk resumes indexing when more space becomes available. For detailed information on how to manage Splunk server disk usage, see Disk usage.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/datastore.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set minfreemb 2000

This command sets the minimum free space to 2000 MB.

Find and index data

There are several methods to get your data into Splunk. Add data via Splunk Web, Splunk's CLI, a configuration file, with scripts, or 3rd party software.

Here's a brief intro on getting data into Splunk. For more detailed instructions, follow any of the links above.

Add Data

When you first log into Splunk Web, you're given three options to begin indexing data:

Option 1: Upload a file
Option 2: Start tailing /var/log
Option 3: Start watching /etc

There are many other ways to specify data inputs in Splunk. This section is a high-level description of these techniques. For more detailed methods, see the data inputs section.

Tail a file

When you specify a file to tail, Splunk processes the entire file and then watches the file and processes additions to it. When you give a directory name to process, Splunk recursively searches all subdirectories looking for files resembling log files. You can explicitly include or exclude files with whitelisting and blacklisting.

Tailing files via Splunk Web

Tailing files via the CLI

Use the splunk add command. These commands assume you have set a Splunk environment variable. If you have not, you must navigate to $SPLUNK_HOME/bin and run the ./splunk command.

For example:

splunk add tail /var/log/

This command tails all files in /var/log/.

Find logfiles

Splunk has a built-in CLI command to search for potential log files to index:

splunk find "searchpath1;searchpath2;..."

find searches logs within the specified searchpaths. You can narrow the search by defining restrictions in $SPLUNK_HOME/etc/findlogs.ini. Restrictions can include types of directories and file to ignore, maximum file size, and modification date. After logs are found, you can index some, all, or none of the files. If you answer "Some", Splunk will prompt you file-by-file.

Add more users

There are three default user roles and three different authentication methods to choose from when you set up Splunk with an Enterprise license. Users authenticate with Splunk's built-in system (described below), LDAP or scripted authentication (for third-party auth systems). Either method works with Splunk's roles system.

You must be logged in as a Splunk administrator to add or edit user accounts. The default Admin account password is changeme.

Note: Splunk with a Free license does not contain access control features.

Lost admin password

If you lose the password to your admin account, contact Splunk Support for assistance.

Splunk local users

A Splunk Admin can create new users either via Splunk Web or Splunk's CLI. Users can be mapped to Splunk's default roles or any custom roles via authorize.conf

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_addusers/adminbutton.jpg

http://www.splunk.com/assets/doc-images/30_admin1_addusers/users.jpg

via Splunk CLI

From the CLI, use the following commands to add, edit, remove or list users.

add user username [-parameter value] ...
edit user username [-parameter value]  ...
remove user username [-parameter value]  ...
list user username [-parameter value]  ...

Required (Default) Parameter:

username -- the name of the Splunk user account to manage.
full-name -- real name of user in quotes, for example "Nikola Tesla" - required when adding a new user.

Optional Parameters:

full-name -- real name of user in quotes, for example "Nikola Tesla"
password -- the password to set for the account
role -- either user, power or admin

Example

This example assumes you have set a Splunk environment variable. If you have not, you must navigate to $SPLUNK_HOME/bin and run the ./splunk command.

# splunk edit user newbie -password f8h2.$R -auth admin:d3cidr

This example authenticates as user "admin" to change the password for user "newbie."

Note: You must be logged in as an Admin to make any changes regarding users. Login either via the splunk login command, or use -auth, as exemplified above.

Start searching

Now you're ready to start using Splunk's search capabilities. Here are a few pages to help you start searching:

  1. Search reference.
  2. Search syntax.
  3. Search tutorial.

Data Inputs

How input configuration works

Specify data inputs via Splunk's CLI or Splunk Web. You may also use inputs.conf (read more about how to configure inputs via inputs.conf). Changes made via Splunk Web or the Splunk CLI are written to $SPLUNK_HOME/etc/bundles/local/inputs.conf. Configure Windows inputs via inputs.conf as well.

Read on for a description of Splunk's data input types, including their purpose and behavior.

Files and directories

Data inputs can come from files and directories. Data in files can be processed in live or batch mode. Use tail input for active log files. Use batch input for closed or archived data.

Tail

Splunk's tail behaves like the UNIX tail command. Specify a path to a file or directory and Splunk's tail processor consumes any new input. If subdirectories exist within the specified directory, Splunk recursively examines them for log files. Splunk automatically adds any new files into the index.

In addition, when tailing a file:

Note: If you are tailing large files or archives, removing the input does not stop those files being indexed. This does stop files from being checked again, but all the initial content will be indexed. To stop all in-process data, you must restart the Splunk server.

When tailing a directory:

Note: If the specified file or directory does not exist, the Splunk server will not check to see if it is created later. Splunk only checks for files and directories each time the Splunk server starts (or is restarted). So be sure to explicitly add new files as inputs when they become available if you don't want to restart the server. When tailing a file, the entire path dir/filename must not exceed 1024 characters.

Batch upload and watch

Splunk's batch processing module watches any specified directory on the local Splunk server's file system and then processes the entirety of any new file that appears. You can also upload archived files directly into Splunk Web. If necessary, Splunk unpacks and uncompresses files before indexing. Keep in mind that Splunk needs adequate disk space to uncompress these files. This processing can take more time than processing a live or uncompressed file.

By default, Splunk's batch processor is located in $SPLUNK_HOME/var/spool/splunk. You can set up your own watch directory as well.

Note: This method does not watch files it has already seen, so it's not designed for live logfiles -- just rotated archive copies.

In addition, when batch uploading or watching, Splunk can:

FIFO queues

A FIFO (AKA named pipe) is a queue of data maintained in memory. File systems can write log messages directly to a FIFO. Splunk then accesses the FIFO as though it were a file. When choosing the FIFO data input method, consider the following:

Note: FIFOs are not recommended for application servers forwarding data to Splunk in a distributed setting. Tail is a more reliable, stable method.

Network ports

UDP and TCP ports can feed data into the Splunk Server. UDP and TCP behave differently, and these behaviors affect how data arrives for processing. When configuring network ports, keep in mind that you cannot use ports lower than 1024 if you have not installed Splunk as root.

UDP

UDP is a best effort protocol. This means that you might not get messages if the network is clogged, or has a hiccup. You also can't be absolutely sure the messages aren't spoofed or altered in transit. UDP should be reserved for logging implementations focused on day-to-day troubleshooting rather than compliance or security.

Splunk with an Enterprise license can read directly from the network on any UDP port. Use this configuration to make Splunk act directly as a syslog server by reading remote syslog events on UDP port 514. You can also send any other UDP source of logging data, including SNMP.

Like all network streaming approaches, direct UDP input is higher performance than reading files from disk.

TCP

TCP is a reliable, high-performance choice for many situations, as this protocol includes checks to ensure that data has arrived safely and intact. Splunk with an Enterprise license can receive data on any TCP port, allowing Splunk to receive remote data from syslog-ng and other syslog implementations that use TCP for security or reliability. TCP is the foundation of Splunk's distributed data access.

Note: If the sending process buffers data such that events are broken into multiple pieces, Splunk may interpret the parts as multiple events. This is more likely if events are being generated intermittently, as there may be long pauses (several seconds or longer) between blocks of buffered data. If you notice truncated events, try forcing the process to send events atomically.

Scripted inputs

Configure Splunk to run shell commands on a schedule, and then index the output. For example:

See Configure scripted inputs for details on how to set this up.

Indexing properties

Splunk can process any data, regardless of format and it automatically learns event boundaries, classifies events and sources, and finds timestamps. However, sometimes you may want to customize Splunk's default processing. Change processing settings and indexing properties in props.conf.

Some attributes within props.conf can be customized by defining new stanzas in other configuration files. For example, transforms.conf defines regex-based rules for extracting fields, correlating events and performing other transformations. Segmenters.conf and outputs.conf can also define attribute values referenced by props.conf.

Common use cases for custom indexing properties include:

Configure inputs via Splunk Web

Follow these instructions to configure data inputs via Splunk Web. You can also configure data inputs via Splunk's CLI or a configuration file.

Configuration

Files and directories

FIFO queues

Network ports

With a Splunk Enterprise license, you can define input from any TCP or UDP port.

Configure inputs via the CLI

In addition to using Splunk Web or editing inputs.conf, you can also configure data inputs at Splunk's Command Line Interface (CLI).

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.

Note: If you get stuck, Splunk's CLI has built-in help. Access the main CLI help page by typing splunk help. Individual commands, objects, and parameters have their own help pages as well -- type splunk help [command | object | parameter name].

Data input commands

Use Splunk CLI data commands to perform actions on data sources. Commands and data sources take various parameters depending on the combination you use. There are five different commands to configure data inputs in the CLI:

Command Command syntax Action
add add [tail|watch|fifo|tcp|udp] source [-parameter value] ... Add a specified data input to Splunk.
edit edit [tail|watch|fifo|tcp|udp] source [-parameter value] ... Edit a data input was previously added.
remove remove [tail|watch|fifo|tcp|udp] source Remove a previously added data input.
list list [tail|watch|fifo|tcp|udp] List the currently configured data inputs of a specified type.
spool spool source Copy a file into Splunk via the sinkhole directory.

Data input types

Specify a data input type to use with a data input command.

Data input type Definition
tail Continuously monitor a file or directory for new input.
watch Monitor an archive directory for new files.
fifo A FIFO or named pipe to index from.
tcp A TCP socket (network input) to monitor.
udp A UDP socket (network input) to monitor.

Input type parameters

Change the configuration of each data input type by defining the parameters below. Optional parameters have the syntax: -parameter value. Use only one -hostname, -hostregex or -hostsegmentnum per command.

Tail

Required parameters

source Path to the file or directory to monitor for new input.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
hostregex Specify a regular expression on the source file path to set as the host field value for events from the input source.
hostsegmentnum Set the number of segments of the source file path to set as the host field value for events from the input source.
active-only (T | F) True or False. Set true to tell Splunk to only keep indexing files that have write-permissions enabled.
follow-only (T | F) True or False. Default False. When set to True, Splunk will read from the end of the source (like the "tail -f" Unix command).

Example

Tail only writable files in /var/log/.

./splunk add tail /var/log/
./splunk edit tail /var/log -active-only true

Watch

Required parameters

source Path to a directory to watch for new input.

Optional parameters

method Set the method to bring files into Splunk (symlink or copy). Default is symlink.
sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
hostregex Specify a regular expression on the source file path to set as the host field value for events from the input source.
hostsegmentnum Set the number of segments of the source file path to set as the host field value for events from the input source.

Example

Watch a directory and set host and sourcetype field values for each event that's indexed.

./splunk add watch /mnt/archive -hostsegmentnum 3
./splunk edit watch /mnt/archive -sourcetype myApp

FIFO

Required parameters

source Path to a FIFO or named pipe to index.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
hostregex Specify a regular expression on the source file path to set as the host field value for events from the input source.
hostsegmentnum Set the number of segments of the source file path to set as the host field value for events from the input source.

Example

Configure a FIFO input and set the host and sourcetype field values for each event that's indexed.

./splunk add fifo /var/run/syslogfifo -sourcetype linux_messages_syslog
./splunk edit fifo /var/run/syslogfifo -hostname web01

TCP/UDP

Required parameters

source Port number to listen for data to index.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
remotehost Specify an IP address to exclusively accept data from.
resolvehost Set True of False (T | F). Default is False. Set True to use DNS to set the host field value for events from the input source.

Example

Configure a network input and set the sourcetype field value for each event that's indexed.

./splunk add udp 514 -sourcetype syslog
./splunk edit udp 514 -resolvehost true -auth gwb:d3c1dr

Configure inputs via inputs.conf

Add data inputs via inputs.conf. This allows for more granularity in your configuration than setting up inputs via SplunkWeb or the CLI.

Note: To set dynamic indexing properties for inputs, use props.conf.

Configuration

Add your stanza to $SPLUNK_HOME/etc/bundles/local/inputs.conf. Specify an input type and any number of attribute/value pairs.

[<inputtype>://<path>]
attribute1 = val1
attribute2 = val2
...

Global settings

The following attributes/value pairs are valid for ALL input types

host = <string>

index = <string>

source = <string>

sourcetype = <string>

queue = <string> (parsingQueue, indexQueue, etc)

Input types

The following attributes/value pairs are valid for the specified input types only.

Tail

[tail://<path>]

This directs Splunk to watch all files in the <path> (or just <path> itself if it represents a single file). You must specify the input type and then the path, so put three slashes in your path if you're starting at root. You can use wildcards for the path; see below.

Note: To ensure new events are indexed when you copy over an existing file with new contents, set CHECK_METHOD = modtime in props.conf for the source. This checks the modtime of the file and re-indexes when it changes. Note that the entire file is indexed, which can result in duplicate events.

Wildcards

You can use wildcards to specify your input path for tail input. Use ... for paths and * for files.

Note: In Windows, you must use two backslashes \\ to escape wildcards. Regexes with backslashes in them are not currently supported for _whitelist and _blacklist in Windows.

Specifying wildcards results in an implicit _whitelist created for that stanza. The longest fully qualified path is used as the tail stanza, and the wildcards are translated into regular expressions using the following map:

wildcard regex meaning
* [^/]* anything but /
... .* anything (greedy)
. \. literal .

For example, if you specify

[tail:///foo/bar*.log]

Splunk translates this into
[tail:///foo/]
_whitelist = bar[^/]*\.log

As a consequence, you can't have multiple stanzas with wildcards for files in the same directory.

For example:

[tail:///foo/bar_baz*]
[tail:///foo/bar_qux*]

This results in overlapping stanzas indexing the directory /foo/. Splunk takes the first one, so only files starting with /foo/bar_baz will be indexed. To encompass both sources, manually specify a _whitelist using regular expression syntax for "or":
[tail:///foo]
_whitelist = (bar_baz[^/]*|bar_qux[^/]*)

Note: To set any additional attributes (such as sourcetype) for multiple whitelisted/blacklisted inputs that may have different attributes, use props.conf

Additional attributes

host_regex = <regular expression>

host_segment = <integer>

crcSalt = <string>

followTail = 0|1

_whitelist = <regular expression>

_blacklist = <regular expression>

Batch

[batch://<path>]

Additional attributes


move_policy = (passive_symlink, passive_copy, sinkhole)

host_regex (see tail)
host_segment (see tail)

Note: source = <string> and <KEY> = <string> are not used by batch.

TCP

[tcp://<remote server>:<port>]

Additional attributes

connection_host = [ip | dns]

UDP

[udp://:<port>]

Additional attributes

_rcvbuf = <int>

no_priority_stripping = <true/false>

FIFO

[fifo://<path>]

Scripted Input

[script://<cmd>]

interval = <integer>

passAuth = <username>

Examples

Tail

[tail:///apache/.../logs]

This loads anything in /apache/foo/logs or /apache/bar/logs, etc.

[tail:///apache/*.log]

This loads anything in /apache/ that ends in .log.

Batch (aka Watch)

[batch://system/flight815/*]
move_policy = sinkhole

This example batch loads all files from the directory /system/flight815/. move_policy = sinkhole deletes the files from the directory.

TCP

[tcp://<remote server>:<port>]

This configures Splunk to listen on the specified port. If a connection is made from <remote server>, this stanza is used to configure the input.
If <remote server> is blank, this stanza matches all connections on the specified port.

UDP

[udp://<remote-server>:<port>]

Similar to TCP, except that Splunk listens on a UDP port.

FIFO

[fifo://<path>]

This directs Splunk to read from the FIFO at the specified path.

Configure inputs for Windows

By default, the Splunk Windows port is configured to index your Windows Application, System, and Security event logs. This functionality is not yet exposed in Splunk Web or the CLI. To disable indexing for an event log, use # to comment it out in the following stanza in $SPLUNK_HOME/etc/bundles/local/inputs.conf:

# Windows platform specific input processor.
[WinEventLog:Application] 
[WinEventLog:Security]
[WinEventLog:System]

Note: You must use two backslashes \\ to escape wildcards in stanza names in inputs.conf. Regexes with backslashes in them are not currently supported when specifying paths to files.

Scripted inputs

By configuring inputs.conf, Splunk can also accept events from scripts. Scripted input is useful for command-line tools, such as vmstat, iostat, netstat, top, etc.

Note: Currently, scripted inputs do not get bundled in the deployment server. In the future, Splunk will support this behavior. For now, use your preferred configuration automation tool to push your script directory to your server classes.

Configuration

Note: Your script must be in the bin/ directory underneath your scripts/ directory.

[script://$SCRIPT] 
interval = X 
index = {main, $YOUR_INDEX}
sourcetype = {iostat, vmstat, etc}  OPTIONAL
source = {iostat, vmstat, etc} OPTIONAL
disabled = false

Variables

Example

This example shows the use of the UNIX top command as a data input source.

$ mkdir $SPLUNK_HOME/etc/bundles/scripts
$ #!/bin/sh
 top -bn 1  # linux only - different OSes have different paramaters
chmod +x $SPLUNK_HOME/etc/bundles/scripts/bin/top.sh
$SPLUNK_HOME/etc/bundles/scripts/bin/top.sh
[script:///opt/splunk/etc/bundles/scripts/bin/top.sh]
interval = 5                # run every 5 seconds
sourcetype = top        # set sourcetype to top
source = script://./bin/top.sh   # set source to name of script

Note:

[top]
BREAK_ONLY_BEFORE = GobblyGook
DATETIME_CONFIG = CURRENT

Configure whitelist and blacklist rules

When specifying inputs to monitor in inputs.conf, you can use whitelist and blacklist rules to explicitly tell Splunk to consume ONLY certain files or consume everything EXCEPT certain files. When you define a whitelist, Splunk indexes ONLY the files in that list. Alternately, when you define a blacklist, Splunk ignores the files in that list and consumes everything else. These settings are independent of each other.

Whitelist and blacklist rules use regular expression syntax to define the match on the file name. Also, your rules must be contained within a configuration stanza, for example [monitor://<path>]); those outside a stanza (global entries) are ignored.

Important: Define whitelist and blacklist entries with exact regex syntax; the "..." wildcard is not supported.

Whitelist (allow) files

To define the files you want Splunk to exclusively index, add the following line to your monitor stanza in $SPLUNK_HOME/etc/system/local/inputs.conf:

_whitelist = $YOUR_CUSTOM_REGEX

For example, if you want Splunk to monitor only files with the .log extension:

[monitor:///mnt/logs]
    _whitelist = .*\.log

Blacklist (ignore) files

To define the files you want Splunk to exclude from indexing, add the following line to your monitor stanza in $SPLUNK_HOME/etc/system/local/inputs.conf:

_blacklist = $YOUR_CUSTOM_REGEX

For example, if you want Splunk to ignore and not monitor only files with the .txt extension:

[monitor:///mnt/logs]
    _blacklist = .*\.txt

If you want Splunk to ignore and not monitor all files with either the .txt extension or the .gz extension:

[monitor:///mnt/logs]
    _blacklist = \.(txt|gz)$

Verify your lists

To verify that your whitelist and blacklist rules are configured properly, run the listtails utility found in your $SPLUNK_HOME/bin directory. listtails reads in the configuration of inputs.conf in all bundle directories, scans the directories and shows you the exact list of files that Splunk will tail when you restart.

Note: The listtails utility requires you to first run the command source setSplunkEnv.

Log file rotation

Splunk recognizes when a file that it is tailing (such as /var/log/messages) has been rolled (/var/log/messages1) and will not read the rolled file in a second time.

Note: Splunk does not recognize tar or gzip files produced by logrotate. You can explicitly set blacklist rules for .tar or .gz to prevent Splunk from reading these files as new logfiles, or you can configure logrotate to move these files into a directory you have not told Splunk to read.

How log rotation works

The tailing processor picks up new files and reads the first and last 256 bytes of the file. This data is hashed into a begin and end cyclic redundancy check (CRC). Splunk checks new CRCs against a database that contains all the CRCs of files Splunk has seen before. The location Splunk last read in the file is also stored.

There are three possible outcomes of a CRC check:

1. There is no begin and end CRC matching this file in the database. This is a new file and will be picked up and consumed from the start. Splunk updates the database with new CRCs and seekptrs as the file is being consumed.

2. The begin CRC is present and the end CRC are present but the size of the file is larger than the seekPtr Splunk stored. This means that, while Splunk has seen the file before, there has been information added to it since it was last read. Splunk opens the file and seeks to the previous end of the file and starts reading from there (so Splunk will only grab the new data and not anything it has read before).

3. The begin CRC is present but the end CRC does not match. This means the file has been changed since Splunk last read it and some of the portions it has read in already are different. In this case there is evidence that the previous data Splunk read from has been changed. In this case Splunk has no choice but to read the whole file again.

Strip syslog headers before processing

Remove syslog headers from non-syslog events that have been passed through syslog to Splunk, such as log4j events from a log4j-to-syslog appender. Splunk ships with a regex to do this for you in $SPLUNK_HOME/etc/bundles/default/transforms.conf. Overwrite or change any of the default attributes and values by creating a transforms.conf in $SPLUNK_HOME/etc/bundles/local/ or your own custom bundle directory. For more information on configuration files in general, see how configuration files work.

Configuration

transforms.conf

In $SPLUNK_HOME/etc/bundles/default/transforms.conf:

# This will strip out date stamp, host, process with pid and just get the
# actual message
[syslog-header-stripper-ts-host-proc]
REGEX         = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s.*?:\s(.*)$
FORMAT        = $1
DEST_KEY      = _raw

Additional strippers found in this file include:

props.conf

In $SPLUNK_HOME/etc/bundles/local/props.conf:

[syslog]
TRANSFORMS= syslog-header-stripper-ts-host-proc

This example turns on the built-in regex for remote syslog inputs.

[syslog]
TRANSFORMS-strip-syslog= syslog-header-stripper-ts-host-proc

Add a name onto the TRANSFORMS declarations. There are no special keywords. TRANSFORMS-the-cake-is-a-lie works just as well.

Example

If you have a central syslog server (syslog1.idkfa.kom) receiving events from multiple servers, you can forward the events to a Splunk Server and index them based on the original host (doom1.idkfa.kom) and original timestamp (07:37:15). For this example the events come to Splunk via UDP port 514 and look like this:

Mar 30 14:29:35 syslog1.idkfa.kom Mar 30 07:37:15 doom1.idkfa.kom sshd[7728]: Connection closed by ::ffff:192.168.1.101

Create this configuration stanza in props.conf:

[syslog]
TIME_PREFIX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s[^\s]*\s
TRANSFORMS-strip-syslog= syslog-header-stripper-ts-host

Determine what files Splunk is tailing

When you configure inputs, you may want to know what specific files Splunk will read prior to starting Splunk for indexing. This is especially true when configuring whitelisting/blacklisting rules. Splunk includes a listtails utility which reads in the configuration of inputs.conf in all bundles, scans your directories and shows you the exact list of files what Splunk will tail when you restart. This allows you to make changes to inputs.conf and verify if the blacklist/whitelist filtering is correct.

Run listtails

To use the listtails utility:
1. Navigate to $SPLUNK_HOME/bin/.
2. Run the command ./splunk cmd listtails.

Index SNMP events with Splunk

The most effective way to index SNMP events is to use snmptrapd to write them to a FIFO.

First, configure snmptrapd to write to a FIFO rather than to a file on disk.

# mkfifo /var/run/snmp-fifo
# snmptrapd -o /var/run/snmp-fifo

Then, configure the Splunk Server to add the FIFO as a data input.

log4j

The best way to index log4j files is to set up a standard log4j-syslog appender on your log4j host. Then configure the Splunk Server's properties to strip the syslog header prior to other processing, so Splunk doesn't think the logs are single-line syslog entries.

See the entry on stripping syslog headers for instructions on stripping the syslog headers.

Data Distribution

How data distribution works

Splunk servers running on any supported OS platform can forward data to one another (as well as to other systems) in real time. This setup allows data inputs gathered on one Splunk server in a specific environment to be sent to another Splunk server for indexing and search. Also, Splunk servers can forward data to groups of other Splunk servers, to enable horizontal scaling via clustered indexing. Splunk servers can also clone data to multiple groups of other Splunk servers to provide for data redundancy in high availability environments.

Data distribution covers all configurations in which one Splunk server (the forwarder) is sending data to one or more Splunk servers (the receivers) prior to being indexed. The forwarder can also index data locally.

Note: All Splunk instances in a distributed cluster must be running the same version of Splunk, although they can be running on any variety of support OSes. Each receiving Splunk server must have a unique, valid Splunk Enterprise license.

Forwarding

Forwarding is the simplest setup for forwarding and receiving. Forwarding refers to any server that sends data to another server for indexing.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/dataforward.jpg

Learn how to enable forwarding and receiving.

Routing

With routing enabled, the forwarder matches conditions based on patterns in the events themselves to selectively send some events to one other server and other events to another server.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/datarouting.jpg

Learn how to enable data routing.

Cloning

Cloning refers specifically to a forwarder sending every event to two or more other Splunk servers to provide for data redundancy.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/datacloning.jpg

Learn how to enable cloning.

Data balancing

Data balancing refers to data that is sent in a balanced fashion to groups of servers. This set up supports large volumes of data. All of the forwarders send data to some number of receivers, and the receivers index data in a round-robin fashion.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/balance.jpg

Data balanced target groups are made up of multiple servers. Learn how to set up data balancing.

Buffering during data balancing

If a server becomes inaccessible during data balancing, Splunk continues to send events to all accessible servers.

Eventually, Splunk stops trying to send to an unresponsive server, and notes that the server has gone off line. If all servers are inaccessible, Splunk writes to a buffer on the forwarder's side.

Data buffering values are set in outputs.conf on the forwarding side.

Target groups

Rather than output data to one receiver, forwarders can send to target groups. Target groups are made of one or more receiving servers:

[target group 1] 
server 1, server 2

[target group 2]
server 3

[target group 3]
server 4, server 5, server 6

Cloning sends every event to all target groups.

Routing sends specific events to one target group and different events to other target groups.

You can also set up default groups, which receive all the data not sent to target groups. If more than one group is specified, Splunk clones events to all listed default groups.

defaultGroup=<groupname1>,<groupname2>...

Learn more about target group configuration.

Security

Any Splunk server can route some or all of its incoming data in real time to other Splunk servers and to other systems via TCP, either in the clear text or via SSL. Learn how to set up SSL.

Send to 3rd party systems

By default, data is routed between Splunk servers as cooked data -- meaning events have been parsed and tagged. However, Splunk can be configured to either receive or send raw data in order to interact with third party systems.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/thirdparty.jpg

Learn how to configure Splunk to send to or receive from third party software.

Distributed search

Splunk servers can distribute search requests to other Splunk servers and merge the results back to the user. Distributed search combines with balanced indexing to provide horizontal scaling, allowing you to search and index hundreds of gigabytes or terabytes per day. Additionally, distributed search allows select users to correlate data across different data silos.

http://www.splunk.com/assets/doc-images/30_admin13_forwardreceive/dsearch.jpg

Learn more about distributed search.

Configuration files for data distribution

Enable forwarding and receiving

Set up forwarding and receiving via Splunk Web or Splunk's CLI. To set up more sophisticated forwarding configurations, see this page on configuring outputs.conf.

You can set up two types of forwarders: standard and lightweight. If you configure a standard forwarder, it indexes the data before forwarding it to the receiving Splunk host. When you configure a lightweight forwarder, it sends un-indexed data to the receiving Splunk host. If you are using both types of forwarders, you must specify a different port for each type.

You must set up receiving before setting up forwarding. This way, the Splunk receiving host is prepared for the forwarded data.

Once you have enabled a Splunk instance to forward or receive data, you can configure additional settings, such as routing, cloning, filtering or data balancing. Configuration changes are done on the forwarder side, on the host that is reading the data input.

Note: An Enterprise license is required on each receiver node. Splunk instances that are forwarding can continue to use the free license. For customers with a valid support agreement that require authentication for all Splunk instances please contact support and request a forwarder license. This special forwarder license can be re-used on all forwarding instances.

Receiving

via Splunk Web

via the CLI

Enable receiving from Splunk's CLI. To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. Also, add Splunk to your path and use the splunk command.

To log in:

./splunk login
Splunk username: admin
Password:

To enable receiving:

# ./splunk enable listen 42099 -auth admin:changeme
Listening for Splunk data on TCP port 42099.

To disable receiving:

# ./splunk disable listen -auth admin:changeme
No longer listening for Splunk TCP data.
You need to restart the Splunk Server for your changes to take effect.

Forwarding

You must first configure your receiving Splunk host using the instructions above before configuring forwarders.

via Splunk Web

To begin forwarding data:

via Splunk CLI

Enable forwarding from the Splunk CLI. Navigate to your $SPLUNK_HOME/bin directory on the forwarding server and log in to the CLI. Also add Splunk to your path and use the splunk command.

./splunk login
Splunk username: admin
Password:

To enable forwarding:

# ./splunk add forward-server 10.2.2.2:9999 -auth admin:changeme

To disable forwarding:

# ./splunk remove forward-server 10.2.2.2:9999 -auth admin:changeme

Lightweight forwarding

If you have installed Splunk on a server generating event data, you may want to forward events to a central Splunk server for indexing. This decreases the workload on the forwarding server. To further reduce the work performed on the forwarding side, enable lightweight forwarding. With a lightweight forwarder, all optional processing is moved to the indexing server. Specifically a lightweight forwarding modifies the server to:

You must first configure your receiving Splunk host using the instructions above before configuring forwarders.
Additionally, if you have deployed both standard and lightweight forwarders, you must ensure that each type of forwarder is sending to its own port on the receiver.

With lightweight forwarding, timestamp and host processing still happen on the forwarding side so that this data is accurate.

Configuration

Turn lightweight forwarding on and off with Splunk's CLI. To use Splunk's CLI, navigate to $SPLUNK_HOME/bin/ and use the ./splunk command. You can also add Splunk to your path and use the splunk command.

To enable lightweight forwarding, use this CLI command on the forwarding server:

./splunk set server-type forwarder

To disable lightweight forwarding, use this CLI command on the forwarding server:

./splunk set server-type default

To use a scripted input on your lightweight forwarder, you need to re-enable the exec processor. To do this, go into $SPLUNK_HOME/etc/modules/input/exec and copy the existing config.xml.disabled to config.xml. This enables the module and on restart it will be inserted into the pipeline.

Transplant Parsing from Forwarder

By default, the lightweight forwarder still parses data with props.conf (i.e. character encoding, timestamp extraction, line-merging) on the forwarder and then sends the parsed data to be indexed. Although parsing is not nearly as resource intensive as indexing, you may still want to avoid doing it on the forwarder.

To disable parsing on the forwarder, configure inputs.conf for each input, or as a global setting:

queue=indexQueue

On the receiving side, the default for splunktcp input is to skip parsing and send data directly to be indexed. To change this, inputs.conf must specify:

[splunktcp://<remote server>:<port>]
queue=parsingQueue

Configure outputs.conf

Configure outputs.conf to send to multiple groups of one or more servers, called target groups. Also, you can set up a default group, made up of one or more target groups, which receives all the data not sent to target groups. If there is more than one group specified in the default group, Splunk clones events to all listed default groups.

Note: While forwarding, events are stored in memory. If any receiver goes down, Splunk buffers the events in memory on the forwarder. Also, by default, time extraction is based on the timestamp in the event, not when Splunk receives the event. If you want to change this default behavior while forwarding, please configure your forwarder to turn off timestamping, in which case Splunk uses the time the forwarder saw the event.

Configuration

Default group and global settings

Add your default group stanza to $SPLUNK_HOME/etc/bundles/local/outputs.conf on the forwarding server.

[tcpout]
defaultGroup= Group1, Group2, ...
attribute1 = val1
attribute2 = val2
...

If you have no default group, set global settings in the [tcpout] stanza.

Note: Settings for your default group are global and inherited by all target groups. Override these settings by creating explicit rules for each target group.

Target groups

Add any number of target group stanzas to $SPLUNK_HOME/etc/bundles/local/outputs.conf on the forwarding server.

[tcpout:$TARGET_GROUP]
server=$IP:$PORT, $IP2:$PORT2...
attribute1 = val1
attribute2 = val2
...

Note: If your target group is made up of more than one $IP:$PORT, the forwarder sends events in a round robin between these URIs.

Optional attributes

There are a number of optional attributes you can set in outputs.conf.

Queue settings

Your data stream enters a queue as it leaves the forwarder. There are a few queue settings you can tweak in outputs.conf.

Single server

Add any number of single server stanzas to $SPLUNK_HOME/etc/bundles/local/outputs.conf on the forwarding server. Use single server configuration to set up SSL and backoff settings (see below). Servers indicated in single server stanzas must also be a part of a target group in order to send data.

[tcpout-server://$IP:$PORT]
attribute1 = val1
attribute2 = val2
...

Backoff settings

Backoff settings are server specific, meaning they must be set in a [tcpout-server://$IP:$PORT] stanza. They cannot be set for a target or default group.

If one of the target group servers becomes unreachable, you can configure the forwarder to retry the connection. If a connection needs to be retried, the forwarder uses backoffAtStartup or initialBackoff as the number of seconds to wait. After this time expires, the forwarder doubles the number of seconds over and over again until reaching maxBackoff. When this is reached, the forwarder stops doubling the number of seconds in between retries and uses the same maxBackoff seconds. It retries at this frequency maxNumberOfRetriesAtHighestBackoff times or forever if that value is -1.

Example

Specify a target group for an IP:PORT which consists of a single receiver. This is the simplest possible configuration; it sends data to the host at 10.1.1.197 on port 9997.

[tcpout:group1]
server=10.1.1.197:9997

Specify a target group for a hostname which consists of a single receiver.

[tcpout:group2]
server=myhost.Splunk.com:9997

Specify a target group made up of two receivers. In this case, the data is balanced (round-robin) between these two receivers. Specify as many receivers as you wish here. Ccombine host name and IP if you wish.

[tcpout:group3]
server=myhost.Splunk.com:9997,10.1.1.197:6666

Send every event to a receiver at foo.Splunk.com:9997 and send heartbeats every 45 seconds with a maximum queue size of 100,500 events.

[tcpout:group4]
server=foo.Splunk.com:9997
heartbeatFrequency=45
maxQueueSize=100500

Set the hearbeat frequency to 15 for each group and clone the events to groups indexer1 and indexer2. Also, index all this data locally as well.

[tcpout]
heartbeatFrequency=15
indexAndForward=true

[tcpout:indexer1]
server=Y.Y.Y.Y:9997

[tcpout:indexer2]
server=X.X.X.X:6666

Data balance between Y.Y.Y.Y and X.X.X.X.

[tcpout:indexerGroup]
server=Y.Y.Y.Y:9997, X.X.X.X:6666

Clone events between two data balanced groups.

[tcpout:indexer1]
server=A.A.A.A:1111, B.B.B.B:2222

[tcpout:indexer2]
server=C.C.C.C:3333, D.D.D.D:4444

Set up routing

Enable routing to forward data from one Splunk server to another based on content. For example, data may be routed to systems based on sourcetype, a custom indexed field, or the content of the raw event. Routing allows you to specifically distribute events to any system.

First, decide which events to route to which servers. Then edit the props.conf, transforms.conf and outputs.conf files on the forwarding servers.

Configuration

props.conf

Edit $SPLUNK_HOME/etc/bundles/local/props.conf and set a TRANSFORMS-routing= attribute:

[<spec>]
TRANSFORMS-routing=$UNIQUE_STANZA_NAME

<spec> can be:

$UNIQUE_STANZA_NAME should match the name of your stanza in transforms.conf.

transforms.conf

Edit $SPLUNK_HOME/etc/bundles/local/transforms.conf and set rules to match your props.conf stanza:

[$UNIQUE_STANZA_NAME]
REGEX=$YOUR_REGEX
DEST_KEY=_TCP_ROUTING
FORMAT=$UNIQUE_GROUP_NAME

outputs.conf

Edit $SPLUNK_HOME/etc/bundles/local/outputs.conf and set which tcpout outputs go to which servers or groups:

[tcpout:$UNIQUE_GROUP_NAME]
server=$IP:$PORT

Examples

Basic example

The following example sends all events with sourcetype="syslog" to one target group, all events that contain the word error to another target group, and everything else to a third target group.

props.conf

Edit $SPLUNK_HOME/etc/bundles/local/props.conf and set a TRANSFORMS-routing= attribute:

[default]
TRANSFORMS-routing=errorRouting

[syslog]
TRANSFORMS-routing=syslogRouting

transforms.conf

Edit $SPLUNK_HOME/etc/bundles/local/transforms.conf and set errorRouting and syslogRouting rules:

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

[syslogRouting]
REGEX=.
DEST_KEY=_TCP_ROUTING
FORMAT=syslogGroup

outputs.conf

Edit $SPLUNK_HOME/etc/bundles/local/outputs.conf and set which tcpout outputs go to with servers or groups:

[tcpout]
defaultGroup=everythingElseGroup

[tcpout:syslogGroup]
server=10.1.1.197:9997

[tcpout:errorGroup]
server=10.1.1.200:9999

[tcpout:everythingElseGroup]
server=10.1.1.250:6666

Complex example

This examples combines routing, data balancing and target group specific parameters. This outputs.conf sends all events with sourcetype="syslog" to one balanced target group, all events that contain the word error to a different target group, and clones everything else to two target groups. The syslogGroup uses a persistent queue which lives in the /tmp directory and is capped at a maximum on disk size of 100MB. The heartbeat frequency for all target groups is dialed down to 10 seconds.

Note: the props.conf and transforms.conf are the same as the example above.

outputs.conf

[tcpout]
defaultGroup=everythingElseGroup1, everthingElseGroup2
heartbeatFrequency=10

[tcpout:syslogGroup]
server=10.1.1.197:9997, 10.1.1.198:7777
usePersistentQueue=true
blockOnQueueFull=true
persistentQueuePath=/tmp
maxPersistentQueueSizeInMegs=100

[tcpout:errorGroup]
server=10.1.1.200:9999

[tcpout:everythingElseGroup1]
server=10.1.1.240:6666

[tcpout:everythingElseGroup2]
server=10.1.1.245:5555

Filtering and routing

Set up Splunk to filter out unwanted events before forwarding and indexing. Edit props.conf and transforms.conf on the forwarding side to eliminate unnecessary data before forwarding. Use the $SPLUNK_HOME/etc/bundles/README/props.conf.example and ../transforms.conf.example as examples, or create your own props.conf and transforms.conf. Make any changes in $SPLUNK_HOME/etc/bundles/local/, or your own custom bundle directory. For more information on configuration files in general, see