This document last updated: 12/01/08 05:12pm

Print Admin Manual

How Splunk Works

Overview of Splunk

Splunk is search software for any type of data. Learn more about how Splunk works by reading through this introductory page. You'll find many links here for installing, configuring and customizing your Splunk installation.

Configuration options

Splunk has several options for configuration: a Web interface (Splunk Web), a command line interface (the CLI), and configuration files. Most of Splunk's configuration can be accomplished by using the Admin page of Splunk Web, and the CLI. Configure advanced settings through configuration files.

Installation and upgrade

Installing Splunk is easy and fast. These instructions show you how to install, upgrade, or back up an existing copy of Splunk.

Important: It's a good idea to back up your current instance before you upgrade.

Data sources

Splunk is capable of receiving data in a variety of ways. Read on for a brief description of each input type. For a more in-depth description of inputs, read how input configuration works.

Windows

Splunk for Windows comes with its own set of configuration files for setting up Windows-specific inputs, including Windows registry and WMI. Read more about configuring Windows inputs.

Distributed data

Configure distributed inputs and outputs across your network. Send data between one Splunk instance and another, or third party software. For an overview on all the available configuration options, see How data distribution works.

Indexing

Splunk takes all data from inputs and sends it to an indexing pipeline. Data is then broken up into separate events via segmentation rules. Most data is segmented and timestamped correctly. However, you may wish to configure Splunk to index your data in particular ways. Learn more about how indexing works.

Here are some things you might want to consider:

Configuration for indexing is set mostly through props.conf and transforms.conf.

Fields

Fields are a useful aspect of Splunk's search interface. You can use Splunk's built-in fields that are enabled by default. Here's a list of Splunk's default fields, including links to more in-depth documentation:

You can also create your own fields. Custom fields are useful for:

To learn more about creating custom fields, see how fields work.

Search

Splunk's search interface is useful for tracking down different aspects of your data. Here are a few things you can do with your searches:

For a more detailed overview of search, see how search works.

Distributed search

In a distributed set up, you may want to search across multiple instances of Splunk. Enable distributed search to federate searches across your entire Splunk deployment. Read more about how distributed search works.

Security

Secure your Splunk server with the following security configuration options. Here's a brief overview of the available features. For a more detailed overview, see security options.

Authentication

Splunk includes several authentication options, including:

Audit

Use the following options to enable separate auditing configurations:

Data management

Splunk servers often index large amounts of data each day. You may want to enable advanced settings to handle the following data management scenarios.

Note: Many data management settings are enabled on a per-index basis, using indexes.conf. To learn more about indexes, see how indexes work.

Deployment server

In a distributed set up, enable one or more Splunk instances as deployment servers. A deployment server pushes out configuration changes to other Splunk instances.

For a complete overview of all deployment options, read the Deployment manual. For instructions on configuring and enabling the deployment server and clients, read the Admin manual section on the deployment server.

Performance tuning

The following options help you tune Splunk's performance for your environment. Depending on your system and requirements, you may want to change one or more of the following settings:

A more in-depth overview of performance tuning options is available here.

Configuration files

Many of Splunk's advanced configurations and customizations are available only through configuration files. Create configurations by copying files into a custom application directory. Learn more about application directories and configuring application directories.

Applications

Applications are directories of configuration files with specific purposes. Configure your own applications by following these instructions.

You can also share your configuration file directories as applications with the Splunk community on SplunkBase.

Customization

Pimp your Splunk! Everybody's data is a little bit different. Maybe you want to set custom configurations for the system you're running Splunk on. Here are options for personalizing your Splunk instance.

Splunk Web appearance

Change various aspects of Splunk Web's appearance:

Extend Splunk

Splunk includes a REST API. Read the Developer's Guide to learn more about the REST API. To configure additional REST endpoints, use restmap.conf.

Troubleshooting

If there's something you need help with, even after reading the documentation, contact Splunk support.

If there's a feature you don't see here that you want included, file an enhancement request with Splunk support.

We're always interested in your feedback.

Getting Started

Start Splunk

This topic serves only as a brief instruction to starting Splunk. If you are new to Splunk, we recommending reviewing the User Manual first.

Before you start

Before starting Splunk, install the software. Refer to the Installation Manual for system requirements and step-by-step instructions. Make sure you install the correct version of Splunk and that you are installing on a supported filesystem.

Start Splunk on non-Windows platforms

Splunk's command line interface is located in $SPLUNK_HOME/bin/. $SPLUNK_HOME refers to the path you installed under. Navigate to this location and run the following command:

# ./splunk start

You must accept Splunk's EULA the first time you start Splunk after a new installation. To bypass this step, start Splunk and accept the license in one step:

# ./splunk start --accept-license

NOTE: There are two dashes before the accept-license option.

Start Splunk on Windows

On Windows, Splunk is installed by default into \Program Files\Splunk

Start and stop the following Splunk processes via the Windows Services Manager:

You can also start, stop, and restart both processes at once by going to \Program Files\Splunk\bin and typing
# splunk.exe [start|stop|restart]

Load Splunk Web in your browser

Navigate to:

http://mysplunkhost:8000

Use whatever host and port you chose during installation.

The first time you login to Splunk with an Enterprise license, use username admin and password changeme. Splunk with a free license does not have access controls.

Administration basics

The $SPLUNK_HOME variable refers to the top level directory of your installation. By default, this is /opt/splunk/.

Add Splunk to your shell path

To save a lot of typing, set a SPLUNK_HOME environment variable and add $SPLUNK_HOME/bin to your shell's path.

This example works for Linux/BSD/Solaris users who accepted the default installation location:

# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

This example works for Mac users who accepted the default installation location:

# export SPLUNK_HOME=/Applications/Splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

Splunk's CLI

Splunk's command line interface is located in $SPLUNK_HOME/bin/. If you have exported the path and environment variables (as explained above), you can use the splunk command as follows:

# splunk [action] [object] [-parameter value] ....

If you haven't set an environment variable, navigate to $SPLUNK_HOME/bin/ and run commands as follows:

#./splunk [action] [object] [-parameter value] ....

For general help, type:

# splunk help

For a list of commands and options, type:

# splunk help commands

For Splunk with an Enterprise license, administration commands must be authenticated with a username and password. To authenticate for an entire session, type:

# splunk login

This command prompts you for a Splunk username and password. Use the same username and password for the CLI and Splunk Web. By default, the login is set to admin and the password is changeme.

Logout at any time by typing:

# splunk logout

To authenticate a single command, use the -auth parameter:

# splunk search foo -auth username:password

Note: the -auth string must be the last term in the CLI command.

Start/stop Splunk, check status

Ensure that you have added Splunk to your server host's path (as explained above, in "Adding Splunk to your shell path"). Otherwise you must use the ./splunk command.

Start the Server

From a shell prompt on the Splunk sever host, run this command:

# splunk start

Alternately, start either splunkd (to load back-end configuration) or Splunk Web (to load web configuration):

# splunk start splunkd

# splunk start splunkweb

Or restart Splunk (splunkd or Splunk Web) by running:

# splunk restart

# splunk restart splunkd

# splunk restart splunkweb

Stop the Server

To shut down Splunk, run this command:

# splunk stop

Also available for splunkd and Splunk Web:

# splunk stop splunkd

# splunk stop splunkweb

Check if Splunk is running

To check if Splunk is running, type this command at the shell prompt on the sever host:

# splunk status

You should see this output:

splunkd is running (PID: 3162).
splunk helpers are running (PIDs: 3164).
splunkweb is running (PID: 3216).

Or you can use ps to check for running Splunk processes:

# ps aux | grep splunk | grep -v grep

Solaris users, type -ef instead of aux:

# ps -ef | grep splunk | grep -v grep

Help

Help is available in several forms.

Help Options

Change defaults

Changing the admin default password

Splunk with an Enterprise license has a default administration account and password. It is highly recommended that you change the default. You can do this via Splunk's CLI or Splunk Web.

Note: CLI commands assume you have set a Splunk environment variable. If you have not, navigate to $SPLUNK_HOME/bin and run the ./splunk command.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/users.jpg

via Splunk CLI

The Splunk CLI command is:

# splunk edit user

Note: You must authenticate with the existing password before it can be changed. Log into Splunk via the CLI or use the -auth parameter.

For example:

# splunk edit user admin -password foo -auth admin:changeme

This command changes the admin password from changeme to foo.

Changing network ports

Splunk uses two ports. They default to:

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/ports.jpg

via Splunk CLI

To change the port settings via the Splunk CLI, use the CLI command set.

# splunk set web-port 9000

This command sets the Splunk Web port to 9000.

# splunk set splunkd-port 9089

This command sets the splunkd port to 9089.

Changing the default Splunk server name

The Splunk server name setting controls both the name displayed within Splunk Web and the name sent to other Splunk Servers in a distributed setting.

The default name is taken from either the DNS or IP address of the Splunk Server host.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/ports.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set servername foo

This command sets the servername to foo.

Changing the datastore location

The datastore is the top-level directory where the Splunk Server stores all indexed data, user accounts, and working files.

Note: If you change this directory, the server does not migrate old datastore files. Instead, it starts over again at the new location.

To migrate your data to another directory follow the instructions in Move an index.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/datastore.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set datastore-dir /var/splunk/

This command sets the datastore directory to /var/splunk/.

Set minimum free disk space

The minimum free disk space setting controls how low disk space in the datastore location can fall before Splunk stops indexing.

Splunk resumes indexing when more space becomes available. For detailed information on how to manage Splunk server disk usage, see Disk usage.

via Splunk Web

http://www.splunk.com/assets/doc-images/30_admin1_changedefaults/adminbutton.jpg

http://www.splunk.com/assets/doc-images/3_2admin1_changedefaults/datastore.jpg

via Splunk CLI

To change the server name via the CLI, type the following:

# splunk set minfreemb 2000

This command sets the minimum free space to 2000 MB.

Find and index data

There are many ways to set up data inputs in Splunk. This section is a high-level description of these techniques. For more detailed methods, see the data inputs section.

Here's a brief intro on getting data into Splunk.

Monitor a file

When you first log in to Splunk Web, you're provided a link to begin monitoring /var/log locally. You can monitor other files and directories you're interested in. When you specify a file to monitor, Splunk processes the entire file and then watches the file and processes additions to it. When you monitor a directory, Splunk recursively searches all subdirectories looking for files resembling log files. You can explicitly include or exclude files with whitelisting and blacklisting.

Monitor files via Splunk Web

Manage your indexed files and add new files to your index from the Admin > Data Inputs: Files & Directories page.

1. To access the Admin page, click the Admin link in the upper right-hand corner.
The Admin page opens to the Server settings page.

2. From the navigation links on the left, click Data Inputs.
The Admin > Data Inputs: All page opens.

3. From the navigation links on the left or the table of input types, click Files & Directories.
The Admin > Data Inputs: FIles & Directories page opens.

4. Click New Input.
The Admin > Data Inputs: Files & Directories: New Input opens.

Monitor files via the CLI

Use the splunk add command. These commands assume you have set a Splunk environment variable. If you have not, you must navigate to $SPLUNK_HOME/bin and run the ./splunk command.

For example:

splunk add monitor /var/log/

This command monitors all files in /var/log/.

Crawl for inputs

Splunk 3.3 introduces the new crawl feature. Crawl your file system for potential logs and data to index. Read more about Using crawl and Configuring crawl.

Add more users

There are three default user roles and three different authentication methods to choose from when you set up Splunk with an Enterprise license. Users authenticate with Splunk's built-in system (described below), LDAP or scripted authentication (for third-party auth systems). Either method works with Splunk's roles system.

You must be logged in as a Splunk administrator to add or edit user accounts. The default Admin account password is changeme.

Note: Splunk with a Free license does not contain access control features. To access this page, you must run Splunk with an Enterprise license. For more information, read About Splunk licenses.

Lost admin password

If you lose the password to your admin account, contact Splunk Support for assistance.

Splunk local users

A Splunk Admin can create new users either via Splunk Web or Splunk's CLI. Users can be mapped to Splunk's default roles or any custom roles via authorize.conf

via Splunk Web

via Splunk CLI

From the CLI, use the following commands to add, edit, remove, or list users.

add user [-parameter value] ...
edit user [-parameter value]  ...
remove user [-parameter value]  ...
list user

Required (default) Parameters:
username -- the name of the Splunk user account to manage.
full-name -- the full name of the user in quotes, for example "Nikola Tesla".
role -- either User, Power, or Admin.

Note: The role names are case sensitive.

Optional Parameters:
password -- the password to set for the account.


Examples

The following are examples of editing a user's properties and adding a new user. Only Admin roles can modify user properties. To login, use the splunk login command or -auth, as exemplified in these examples.

Note: These examples assume you have set a Splunk environment variable. If you have not, navigate to $SPLUNK_HOME/bin and run the ./splunk command.

Example 1

Let's say, as an admin on a Splunk server, you want to change the password for another user. The syntax for this looks something like:

# splunk edit user <username> -password <newpassword> -auth <your_username>:<your_password>

Note: When editing a specific user's properties, you can list the user without the -username parameter.

Therefore, to authenticate as user admin to change the password for user newbie:

# splunk edit user newbie -password f8h2.$R -auth admin:adminpw

Example 2

Now, as an admin on a Splunk server, you want to add a new user with more than one role. The syntax for this looks something like:

# splunk add user -username <username> -full-name "First Last" -role <role1> -role <role2> -password <password> -auth <your_username>:<your_password>

Therefore, to add a new user deep, with Everybody and Admin permissions:

# splunk add user -username deep -full-name "the deep" -role Everybody -role Admin -password foobar -auth admin:adminpw

Start searching

Now you're ready to start using Splunk's search capabilities. Here are a few pages to help you start searching:

  1. Search reference.
  2. Search syntax.
  3. Search tutorial.

Data Inputs

How input configuration works

Splunk consumes any data you point it at. Before indexing data, you must add your data source as an input. The source is then listed as one of Splunk's default fields (whether it's a file, directory or network port).

Note: Splunk looks for the inputs it is configured to monitor every 24 hours starting from the time it was last restarted. This means that if you add a stanza to monitor a directory or file that doesn't exist yet, it could take up to 24 hours for Splunk to start indexing its contents. To ensure that your input is immediately recognized and indexed, add the input via Splunk Web or by using the add command in the CLI.

Data input methods

Specify data inputs via the following methods:

Most data sources can be specified via Splunk Web. For more extensive configuration options, use inputs.conf. Changes made via Splunk Web or the Splunk CLI are written to $SPLUNK_HOME/etc/system/local/inputs.conf. Configure Windows inputs via inputs.conf as well.

Sources

Splunk accepts data inputs from a wide range of sources. Here's a basic overview of your options. Read on through the Data Inputs and Data Distribution sections of this manual for configuration specifics.

Files and directories

Data inputs include files and directories. Use monitor for continuous, non-destructive inputs from files and directories. Use batch input for one time, destructive file loading. Destructive file loading means that the original files are deleted when Splunk is done indexing them. Keep this in mind when using batch input.

You can also configure Splunk's file system change monitor to watch for changes in your file system. However, you cannot currently use both monitor and file system change monitor to follow the same directory or file. If you want to see changes in a directory, use file system change monitor. If you want to index new events in a directory, use monitor.

To configure files and directories, see files and directories.

To configure file system change monitor, see the page on file system change monitor.

Monitor

Specify a path to a file or directory and Splunk's monitor processor consumes any new input. You can also specify a mounted or shared directory, as long as the Splunk server can see the directory. If the specified directory contains subdirectories, Splunk recursively examines them for new files. Splunk only checks for files and directories each time the Splunk server starts/restarts, so be sure to add new sources when they become available if you don't want to restart the server. You can also use crawl to discover new sources

When using monitor:

Note: Splunk looks for the inputs it is configured to monitor every 24 hours starting from the time it was last restarted. This means that if you add a stanza to monitor a directory or file that doesn't exist yet, it could take up to 24 hours for Splunk to start indexing its contents. To ensure that your input is immediately recognized and indexed, add the input via Splunk Web or by using the add command in the CLI.

Important: To avoid performance issues, Splunk recommends that you set followTail=1 in inputs.conf if you are deploying Splunk to systems containing significant quantities of historical data. Setting followTail=1 for a monitor input means that any new incoming data is indexed when it arrives, but anything already in files on the system when Splunk was first started will not be processed.

Batch upload

Upload files directly through Splunk Web. If necessary, Splunk uncompresses files before indexing.

Use the batch processor at the CLI or in inputs.conf to load files once and destructively. By default, Splunk's batch processor is located in $SPLUNK_HOME/var/spool/splunk. If you move a file into this directory, Splunk indexes it and deletes it. For continuous, non-destructive loading of files, use monitor.

FIFO queues

Caution: Due to their vulnerability, FIFOs are not recommended. Monitor is a more reliable, stable method. Support FIFO inputs is deprecated and will be removed in a future release of Splunk.

A FIFO (AKA named pipe) is a queue of data maintained in memory. File systems can write log messages directly to a FIFO. Splunk then accesses the FIFO as though it were a file. FIFO access is very fast, but FIFOs are vulnerable when there are processing disruptions because the in-memory data may be lost.

To configure FIFO cues, see this page.

Network ports

You can configure Splunk with an Enterprise license to listen on any network port. This is the best method to send data to your Splunk server from any machine (see data distribution for more information). When configuring network ports, keep in mind that you cannot use ports lower than 1024 if you have not installed Splunk as root.

To configure network ports, see this page.

UDP

UDP is a best effort protocol, so you might not get messages if the network is clogged or goes down. You also can't be absolutely sure the messages aren't spoofed or altered in transit. Use UDP for day-to-day troubleshooting rather than compliance or security.

Splunk with an Enterprise license can read directly from the network on any UDP port. Use this configuration to make Splunk act directly as a syslog server by reading remote syslog events on UDP port 514. You can also send any other UDP source of logging data.

TCP

TCP is a reliable, high-performance choice for many situations, as TCP checks to ensure that data has arrived safely and intact. Splunk with an Enterprise license can receive data on any TCP port, allowing Splunk to receive remote data from syslog-ng and other syslog implementations that use TCP for security or reliability. TCP is the foundation of Splunk's data distribution architecture.

Scripted inputs

Configure Splunk to run shell commands on a schedule, and then index the output.

For example:

See configure scripted inputs for details on setting this up.

Windows data sources

By default, Splunk for Windows indexes all Windows Application, System, and Security event logs. Splunk for Windows can also monitor and index changes to your registry and accept WMI data input. For more information on configuring Splunk for Windows, see this page.

Crawl

Discover new inputs automatically. Crawl uses rules you configure to traverse any given directory structure. Splunk adds new inputs you find via crawl to inputs.conf.

Data processing

Once Splunk consumes data, it sends it to the universal processing pipeline, where it further processes your data. Splunk automatically learns event boundaries, classifies events and sources, and finds timestamps. However, you may want to customize Splunk's default processing. Change processing settings and indexing properties via props.conf.

Some attributes within props.conf can be customized by defining new stanzas in other configuration files. For example, transforms.conf defines regex-based rules for extracting fields, correlating events and performing other transformations. Segmenters.conf and outputs.conf can also define attribute values referenced by props.conf.

Common use cases for custom indexing properties include:

Files and directories

Point Splunk at a file or a directory. If you specify a directory, Splunk consumes everything in the directory. Use monitor for continuous, non-destructive inputs from files and directories. Use batch input for one time, destructive file loading, which means that Splunk deletes any batch files once it indexes them.

Monitor

Specify a path to a file or directory and Splunk's monitor processor consumes any new input. You can also specify a mounted or shared directory, including network filesystems, as long as the Splunk server can see the directory. If the specified directory contains subdirectories, Splunk recursively examines them for new files.

Splunk only checks for files and directories each time the Splunk server starts/restarts, so be sure to add new inputs when they become available if you don't want to restart the server. If you want Splunk to find potential new inputs automatically, use crawl.

When using monitor:

Note: You cannot currently use both monitor and file system change monitor to follow the same directory or file. If you want to see changes in a directory, use file system change monitor. If you want to index new events in a directory, use monitor.

Batch

Upload files directly through Splunk Web. If necessary, Splunk uncompresses files before indexing. Use the batch processor at the CLI or in inputs.conf to load files once and destructively. By default, Splunk's batch processor is located in $SPLUNK_HOME/var/spool/splunk. If you move a file into this directory, Splunk indexes it and deletes it. For continuous, non-destructive loading of files, use monitor.

Splunk Web

Add inputs from files and directories via Splunk Web.

1. Click Admin in the upper right-hand corner of Splunk Web.

2. Then click Data Inputs.

3. Pick files and directories.

4. Click New Input to add an input.

5. Under Data access, pick Monitor a directory.

You can also:

6. Specify the pathname to the file or directory. If you select Upload, use the Browse... button.
To monitor a shared network drive, enter the following: <myhost><mypath> (or \\<myhost>\<mypath> on Windows). Make sure your Splunk server can see the mounted drive.

7. Under the Host heading, select the host name. You have several choices if you are using Monitor or Batch methods. Learn more about setting host value.
Note: Host only sets the host field in Splunk. It does not direct Splunk to look on a specific host on your network.

8. Now set the Source Type. Source type is a default field added to events. Source type is used to determine processing characteristics such as timestamps and event boundaries. Learn more about source type.

9. After specifying the source, host, and source type, click Submit.

CLI

Monitor files and directories via Splunk's Command Line Interface (CLI). To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. Or add Splunk to your path and use the splunk command.

If you get stuck, Splunk's CLI has built-in help. Access the main CLI help by typing splunk help. Individual commands have their own help pages as well -- type splunk help <command>.

The following commands are available for input configuration via the CLI:

Command Command syntax Action
add add monitor $SOURCE [-parameter value] ... Add inputs from $SOURCE.
edit edit monitor $SOURCE [-parameter value] ... Edit a previously added input for $SOURCE.
remove remove monitor $SOURCE Remove a previously added $SOURCE.
list list monitor List the currently configured monitor.
spool spool source Copy a file into Splunk via the sinkhole directory.

Change the configuration of each data input type by setting additional parameters. Parameters are set via the syntax: -parameter value.

Note: You can only set one -hostname, -hostregex or -hostsegmentnum per command.

Required parameters

source Path to the file or directory to monitor for new input.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
hostregex Specify a regular expression on the source file path to set as the host field value for events from the input source.
hostsegmentnum Set the number of segments of the source file path to set as the host field value for events from the input source.
active-only (T | F) True or False. Set true to tell Splunk to only keep indexing files that have write-permissions enabled.
follow-only (T | F) True or False. Default False. When set to True, Splunk will read from the end of the source (like the "tail -f" Unix command).

Example

The following example shows how to monitor files in /var/log/:

Add /var/log/ as a data input:

./splunk add monitor /var/log/

Edit the input you just added:

./splunk edit monitor /var/log -active-only true

This tells Splunk to monitor only files that are still open for writing.

Inputs.conf

To add an input, add a stanza for it to inputs.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. If you have not worked with Splunk's configuration files before, read how configuration files work before you begin.

You can set any number of attributes and values following an input type. If you do not specify a value for one or more attributes, Splunk uses the defaults that are preset in $SPLUNK_HOME/etc/system/default/ (noted below).

Monitor

[monitor://<path>]
<attrbute1> = <val1>
<attrbute2> = <val2>
...

This type of input stanza (monitor) directs Splunk to watch all files in the <path> (or just <path> itself if it represents a single file). You must specify the input type and then the path, so put three slashes in your path if you're starting at root. You can use wildcards for the path; see below.

Note: To ensure new events are indexed when you copy over an existing file with new contents, set CHECK_METHOD = modtime in props.conf for the source. This checks the modtime of the file and re-indexes when it changes. Note that the entire file is indexed, which can result in duplicate events.

host = <string>

index = <string>

sourcetype = <string>

source = <string>

queue = <string> (parsingQueue, indexQueue, etc)

host_regex = <regular expression>

host_segment = <integer>

crcSalt = <string>

followTail = 0|1

_whitelist = <regular expression>

_blacklist = <regular expression>

Wildcards

You can use wildcards to specify your input path for monitored input. Use ... for paths and * for files.

Note: In Windows, you must use two backslashes \\ to escape wildcards. Regexes with backslashes in them are not currently supported for _whitelist and _blacklist in Windows.

Specifying wildcards results in an implicit _whitelist created for that stanza. The longest fully qualified path is used as the monitor stanza, and the wildcards are translated into regular expressions using the following map:

wildcard regex meaning
* [^/]* anything but /
... .* anything (greedy)
. \. literal .

For example, if you specify

[monitor:///foo/bar*.log]

Splunk translates this into
[monitor:///foo/]
_whitelist = bar[^/]*\.log

As a consequence, you can't have multiple stanzas with wildcards for files in the same directory.

For example:


[monitor:///foo/bar_baz*]
[monitor:///foo/bar_qux*]

This results in overlapping stanzas indexing the directory /foo/. Splunk takes the first one, so only files starting with /foo/bar_baz will be indexed. To encompass both sources, manually specify a _whitelist using regular expression syntax for "or":
[monitor:///foo]
_whitelist = (bar_baz[^/]*|bar_qux[^/]*)

Note: To set any additional attributes (such as sourcetype) for multiple whitelisted/blacklisted inputs that may have different attributes, use props.conf.

Examples

To load anything in /apache/foo/logs or /apache/bar/logs, etc.

[monitor:///apache/.../logs]

To load anything in /apache/ that ends in .log.
[monitor:///apache/*.log]

Batch

[batch://<path>]
move_policy = sinkhole
<attrbute1> = <val1>
<attrbute2> = <val2>
...

Use batch to set up a one time, destructive input of data from a source. For continuous, non-destructive inputs, use monitor.
Note: You must set move_policy = sinkhole. This loads the file destructively. Do not use this input type for files you do not want to consume destructively.

host = <string>

index = <string>

sourcetype = <string>

source = <string>

queue = <string> (parsingQueue, indexQueue, etc)

host_regex = <regular expression>

host_segment = <integer>

Note: source = <string> and <KEY> = <string> are not used by batch.

Example

This example batch loads all files from the directory /system/flight815/.

[batch://system/flight815/*]
move_policy = sinkhole

FIFO

Caution: Data sent via FIFO is not persisted in memory and can be an unreliable method for data sources. To ensure your data is not lost, use monitor instead.

Splunk Web

Add inputs from FIFOs via Splunk Web.

1. Click Admin in the upper right-hand corner of Splunk Web.

2. Then click Data Inputs.

3. Pick files and directories.

4. Click New Input to add an input.

5. Under Source, type in the path to the FIFO.

6. Under the Host heading, accept the default host name or enter a new hostname/IP address. Learn more about setting host value.
Note: Host only sets the host field in Splunk. It does not direct Splunk to look on a specific host on your network.

7. Now set the Source Type. Source type is a default field added to events. Source type is used to determine processing characteristics such as timestamps and event boundaries. Learn more about setting source type. Choose:

8. After specifying the source, host, and source type, click Submit.

CLI

Add a FIFO via Splunk's Command Line Interface (CLI). To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. Or add Splunk to your path and use the splunk command.

If you get stuck, Splunk's CLI has built-in help. Access the main CLI help by typing splunk help. Individual commands have their own help pages as well -- type splunk help <command>.

The following commands are available for input configuration via the CLI:

Command Command syntax Action
add add fifo $SOURCE [-parameter value] ... Add inputs from $SOURCE.
edit edit fifo $SOURCE [-parameter value] ... Edit a previously added input for $SOURCE.
remove remove fifo $SOURCE Remove a previously added $SOURCE.
list list fifo List the currently configured $SOURCE.

Required parameters

source Path to a FIFO or named pipe to index.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
hostregex Specify a regular expression on the source file path to set as the host field value for events from the input source.
hostsegmentnum Set the number of segments of the source file path to set as the host field value for events from the input source.

Example

This example shows how to enable a FIFO input, then set the host and sourcetype.

1. Add the FIFO /var/run/syslogfifo and set the sourcetype to linux_messages_syslog.

./splunk add fifo /var/run/syslogfifo -sourcetype linux_messages_syslog

2. Edit the input configuration to set the host to web01.

./splunk edit fifo /var/run/syslogfifo -hostname web01

inputs.conf

To add an input, add a stanza for it to inputs.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. If you have not worked with Splunk's configuration files before, read how configuration files work before you begin.

You can set any number of attributes and values following an input type. If you do not specify a value for one or more attributes, Splunk uses the defaults that are preset in $SPLUNK_HOME/etc/system/default/ (noted below).

[fifo://<path>]

This input stanza type directs Splunk to read from a FIFO at the specified path.

host = <string>

index = <string>

sourcetype = <string>

source = <string>

queue = <string> (parsingQueue, indexQueue, etc)

Network ports

With a Splunk Enterprise license, you can enable input on any TCP or UDP port. Splunk consumes any data sent on these ports. TCP is the protocol underlying Splunk's data distribution, which is the recommended method for sending data from any remote machine to your Splunk server.

Important: In version 3.3.3 of Splunk, default syslog processing via UDP does not correctly handle line-breaks. To work around this issue, add _linebreaker = _linebreaker to the UDP stanza in $SPLUNK_HOME/etc/system/local/inputs.conf. This issue was resolved in 3.3.4.

Splunk Web

Add inputs from network ports via Splunk Web.

1. Click Admin in the upper right-hand corner of Splunk Web.

2. Then click Data Inputs.

3. Pick Network Ports - Display and access configuration for UDP and TCP ports.

4. Click New Input to add an input.

5. Under the Source heading, select Protocol of UDP or TCP.

6. Accept the default port, 9998, or enter another port number.

7. Specify whether this port should accept connections from all hosts or one host. If you specify one host, enter the IP address of the host.

8. Now set the Source Type. Source type is a default field added to events. Source type is used to determine processing characteristics such as timestamps and event boundaries. Learn more about setting source type. Choose:

9. After specifying the source, host, and source type, click Submit.

CLI

Monitor files and directories via Splunk's Command Line Interface (CLI). To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. Or add Splunk to your path and use the splunk command.

If you get stuck, Splunk's CLI has built-in help. Access the main CLI help by typing splunk help. Individual commands have their own help pages as well -- type splunk help <command>.

The following commands are available for input configuration via the CLI:

Command Command syntax Action
add add monitor $SOURCE [-parameter value] ... Add inputs from $SOURCE.
edit edit monitor $SOURCE [-parameter value] ... Edit a previously added input for $SOURCE.
remove remove monitor $SOURCE Remove a previously added data input.
list list monitor List the currently configured monitor.
spool spool source Copy a file into Splunk via the sinkhole directory.

Change the configuration of each data input type by setting additional parameters. Parameters are set via the syntax: -parameter value.

Required parameters

source Port number to listen for data to index.

Optional parameters

sourcetype Specify a sourcetype field value for events from the input source.
index Specify the destination index for events from the input source.
hostname Specify a host name to set as the host field value for events from the input source.
remotehost Specify an IP address to exclusively accept data from.
resolvehost Set True of False (T | F). Default is False. Set True to use DNS to set the host field value for events from the input source.

Example

Configure a network input, then set the sourcetype:

Important: In version 3.3.3 of Splunk, default syslog processing via UDP does not correctly handle line-breaks. To work around this issue, add _linebreaker = _linebreaker to the UDP stanza in $SPLUNK_HOME/etc/system/local/inputs.conf. This issue was resolved in 3.3.4.

Check the Spunk Wiki for information about the best practices for using UDP when configuring Syslog input.

./splunk add udp 514 -sourcetype syslog
./splunk edit udp 514 -resolvehost true -auth admin:changeme

Note: Splunk must be running as root to watch ports under 1024.

inputs.conf

To add an input, add a stanza for it to inputs.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. If you have not worked with Splunk's configuration files before, read how configuration files work before you begin.

You can set any number of attributes and values following an input type. If you do not specify a value for one or more attributes, Splunk uses the defaults that are preset in $SPLUNK_HOME/etc/system/default/ (noted below).

TCP

[tcp://<remote server>:<port>]
<attrbute1> = <val1>
<attrbute2> = <val2>
...

This type of input stanza tells Splunk to listen to <remote server> on <port>. If <remote server> is blank, Splunk listens to all connections on the specified port.

host = <string>

index = <string>

sourcetype = <string>

source = <string>

queue = <string> (parsingQueue, indexQueue, etc)

connection_host = [ip | dns]

UDP

Important: In version 3.3.3 of Splunk, default syslog processing via UDP does not correctly handle line-breaks. To work around this issue, add _linebreaker = _linebreaker to the UDP stanza in $SPLUNK_HOME/etc/system/local/inputs.conf. This was resolved in 3.3.4.

[udp://:<port>]
<attrbute1> = <val1>
<attrbute2> = <val2>
...

This type of input stanza is similar to the TCP type, except that it listens on a UDP port.

host = <string>

index = <string>

sourcetype = <string>

source = <string>

queue = <string> (parsingQueue, indexQueue, etc)

_rcvbuf = <int>

no_priority_stripping = true

no_appending_timestamp = true

Scripted inputs

By configuring inputs.conf, Splunk can accept events from scripts. Scripted input is useful for command-line tools, such as vmstat, iostat, netstat, top, etc.

Note: Currently, scripted inputs do not get sent via the deployment server. In the future, Splunk will support this behavior. For now, use your preferred configuration automation tool to push your script directory to your server classes.

Note: On Windows platforms, use of text-based scripts such those in perl and python can be handled via the use of an intermediary window batch (.bat) file.

Caution: Scripted input-launched scripts inherit Splunk's environment, so be sure to
clear environment variables which may affect your script's operation. The only environment variable that's likely to cause problems is the library path (most commonly known as LD_LIBRARY_PATH on linux/solaris/freebsd).

Configuration

Configure inputs.conf, using the following attributes:

[script://$SCRIPT] 
interval = X 
index = <index>
sourcetype = <iostat, vmstat, etc>  OPTIONAL
source = <iostat, vmstat, etc> OPTIONAL
disabled = <true | false>

If you want the script to run continuously, write the script to never exit and set it on a short interval. This helps to ensure that if there is a problem the script gets restarted. Splunk keeps track of scripts it has spawned and will shut them down upon exit.

Example

This example shows the use of the UNIX top command as a data input source.

$ mkdir $SPLUNK_HOME/etc/apps/scripts
$ #!/bin/sh
 top -bn 1  # linux only - different OSes have different paramaters
chmod +x $SPLUNK_HOME/etc/apps/scripts/bin/top.sh
$SPLUNK_HOME/etc/apps/scripts/bin/top.sh
[script:///opt/splunk/etc/apps/scripts/bin/top.sh]
interval = 5                # run every 5 seconds
sourcetype = top        # set sourcetype to top
source = script://./bin/top.sh   # set source to name of script

props.conf

You may need to modify props.conf:

For example, adding the following to $SPLUNK_HOME/etc/apps/scripts/default/props.conf forces all lines into a single event:

[top]
BREAK_ONLY_BEFORE = GobblyGook

Since there is no timestamp in the top output we need to tell Splunk to use the current time. This is done in props.conf by setting:

DATETIME_CONFIG = CURRENT

Whitelist and blacklist rules

When specifying inputs to monitor in inputs.conf, you can use whitelist and blacklist rules to explicitly tell Splunk to consume ONLY certain files or consume everything EXCEPT certain files. When you define a whitelist, Splunk indexes ONLY the files in that list. Alternately, when you define a blacklist, Splunk ignores the files in that list and consumes everything else. These settings are independent of each other.

Whitelist and blacklist rules use regular expression syntax to define the match on the file name. Also, your rules must be contained within a configuration stanza, for example [monitor://<path>]; those outside a stanza (global entries) are ignored.

Important: Define whitelist and blacklist entries with exact regex syntax; the "..." wildcard is not supported.

Note: We recommend that you blacklist all compressed files.

Note: Instead of whitelisting or blacklisting your data inputs, you can filter specific events and send them to different queues or indexes. Read more about filtering and routing events to different queues and filtering and routing events to alternate indexes. You can also use the crawl feature to predefine files you want Splunk to index or not index automatically when they are added to your filesystem.

Whitelist (allow) files

To define the files you want Splunk to exclusively index, add the following line to your monitor stanza in $SPLUNK_HOME/etc/system/local/inputs.conf:

_whitelist = $YOUR_CUSTOM_REGEX

For example, if you want Splunk to monitor only files with the .log extension:

[monitor:///mnt/logs]
    _whitelist = .*\.log$

You can whitelist multiple files in one line, using the "|" (OR) operator. For example, to whitelist filenames that contain query.log OR my.log:

_whitelist = query\.log$|my\.log$

Or, to whitelist exact matches:

_whitelist = /query\.log$|/my\.log$

Note: The "$" anchors the regex to the end of the line. There is no space before or after the "|" operator.

Blacklist (ignore) files

To define the files you want Splunk to exclude from indexing, add the following line to your monitor stanza in $SPLUNK_HOME/etc/system/local/inputs.conf:

_blacklist = $YOUR_CUSTOM_REGEX

Important: If you create a _blacklist line for each file you want to ignore, Splunk activates only the last filter.

If you want Splunk to ignore and not monitor only files with the .txt extension:

[monitor:///mnt/logs]
    _blacklist = \.(txt)$

If you want Splunk to ignore and not monitor all files with either the .txt extension OR the .gz extension (note that you use the "|" for this):

[monitor:///mnt/logs]
    _blacklist = \.(txt|gz)$

Verify your lists

To verify that your whitelist and blacklist rules are configured properly, run the listtails utility found in your $SPLUNK_HOME/bin directory. listtails reads in the configuration of inputs.conf in all application directories, scans the directories, and displays an exact list of files that Splunk will monitor when you restart.

In your $SPLUNK_HOME/bin directory, run:

./splunk cmd listtails

Crawl

Use crawl to search your filesystem for new data sources to add to your index. Configure one or more types of crawlers in crawl.conf to define the type of data sources to include in or exclude from your results.

Configuration

Edit $SPLUNK_HOME/etc/system/local/crawl.conf to configure one or more crawlers that browse your data sources when you run the crawl command. Define each crawler by specifying values for each of the crawl attributes. Enable the crawler by adding it to crawlers_list.

Crawl logging

The crawl command produces a log of crawl activity that's stored in $SPLUNK_HOME/var/log/splunk/crawl.log. Set the logging level with the logging key in the [default] stanza of crawl.conf:

[default]
logging = <warn | error | info | debug>

Enable crawlers

Enable a crawler by listing the crawler specification stanza name in the crawlers_list key of the [crawlers] stanza.

Use a comma-separated list to specify multiple crawlers.

Enable crawlers that are defined in the stanzas: [file_crawler], [port_crawler], and [db_crawler].

[crawlers]
crawlers_list = file_crawler, port_crawler, db_crawler

Define crawlers

Define a crawler by adding a definition stanza in crawl.conf. Add additional crawler definitions by adding additional stanzas.

Example crawler stanzas in crawl.conf:

[Example_crawler_name]
....

[Another_crawler_name]
....

Add key/value pairs to crawler definition stanzas to set a crawler's behavior. The following keys are available for defining a file_crawler:

bad_directories_list Specify directories to exclude.
bad_extensions_list Specify file extensions to exclude.
bad_file_matches_list Specify a string, or a comma-separated list of strings that filenames must contain to be excluded. You can use wildcards (examples: foo*.*,foo*bar, *baz*).
packed_extensions_list Specify extensions of compressed files to include. Leave this empty if you don't want to add any zipped files.
collapse_threshold Specify the minimum number of files a source must have to be considered a directory.
days_sizek_pairs_list Specify a comma-separated list of age (days) and size (kb) pairs to constrain what files are crawled. For example: days_sizek_pairs_list = 7-0, 30-1000 tells Splunk to crawl only files last modified within 7 days and at least 0kb in size, or modified within the last 30 days and at least 1000kb in size.
big_dir_filecount Set the maximum number of files a directory can have in order to be crawled. crawl excludes directories that contain more than the maximum number you specify.
index Specify the name of the index to add crawled file and directory contents to.
max_badfiles_per_dir Specify how far to crawl into a directory for files. If Splunk crawls a directory and doesn't find valid files within the specified max_badfiles_per_dir, then Splunk excludes the directory.
root Specify directories for a crawler to crawl through.

Example

Here's an example crawler called simple_file_crawler may look like:

[simple_file_crawler]
bad_directories_list= bin, sbin, boot, mnt, proc, tmp, temp, home, mail, .thumbnails, cache, old
bad_extensions_list= mp3, mpg, jpeg, jpg,  m4, mcp, mid
bad_file_matches_list= *example*, *makefile, core.*
packed_extensions_list= gz, tgz, tar, zip
collapse_threshold= 10
days_sizek_pairs_list= 3-0,7-1000, 30-10000
big_dir_filecount= 100
index=main
max_badfiles_per_dir=100

Windows inputs

Configure Splunk for Windows to index your Windows Application, System, and Security event logs. Splunk for Windows can also monitor and index changes to your registry and accept WMI data input. This functionality is not yet exposed in Splunk Web or the CLI.

When you run the Splunk Windows installer, you are given the option to set up indexing and/or monitoring for the event logs, the registry, and for WMI. If you choose to do this, the default values for these settings are assumed. Once you have completed the installation, you can then make changes to the default values set by the installation process.

If you want to make changes to the default values, edit a copy of inputs.conf in $SPLUNK_HOME\etc\system\local\. You only have to provide values for the attributes you want to change within the stanza. For more information about how to work with Splunk configuration files, refer to How configuration files work.

Configure indexing for Windows event logs

Windows event logs are from binary format *.evt files and cannot be monitored like a flat file. The settings for which event logs to index are in the following stanza in inputs.conf:

# Windows platform specific input processor.
[WinEventLog:Application] 
[WinEventLog:Security]
[WinEventLog:System]

You can configure Splunk to read non-default Windows event logs as well, but you must import them to the Windows Event Viewer first, and then add them to your local copy of inputs.conf, (usually in $SPLUNK_HOME\etc\system\local\inputs.conf) as follows:

[WinEventLog:DNS Server]
disabled = 0
[WinEventLog:Directory Service]
disabled = 0
[WinEventLog:File Replication Service]
disabled = 0

To disable indexing for an event log, add disabled = 1 below its listing in the stanza in $SPLUNK_HOME\etc\system\local\inputs.conf.

Configure Windows registry monitoring input

The global settings for Windows registry monitoring are in the following stanza in inputs.conf:

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.py]
interval = 60
sourcetype = WinRegistry
source = WinRegistry
disabled = 0

Note: The Splunk registry input monitoring script (splunk-regmon.py) is configured as a scripted input. Do not change this value.

The Windows registry monitoring functionality uses two additional configuration files that are described in Windows registry input. You may wish to review that page before proceeding.

Note: You must use two backslashes \\ to escape wildcards in stanza names in inputs.conf. Regexes with backslashes in them are not currently supported when specifying paths to files.

Windows Management Interface (WMI) input

Splunk supports WMI (Windows Management Interface) data input for agentless access to Windows performance data and event logs. This means you can pull event logs from all the Windows servers and desktops in your environment without having to install anything on those machines.

The Splunk WMI data input can connect to multiple WMI providers and pull data from them. The WMI data input runs as a separate process (splunk-wmi.exe) on the Splunk server. It is configured as a scripted input in $SPLUNK_HOME\etc\system\default\inputs.conf. Do not edit this file.

Note: This feature is only available on the Windows versions of Splunk and is NOT enabled by default. To enable it, add the following line to $SPLUNK_HOME\etc\system\local\inputs.conf:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.py]
disabled = 0

Important: There is an issue with stopping and restarting Splunk currently affecting users of remote WMI polling. If one or more of your WMI sources is unavailable at the time that you stop Splunk, Splunk will not come back up unless you wait for the splunk-wmi.exe process to exit, or kill it manually. To avoid this issue, do not unnecessarily list non-existent/non-functioning machines in wmi.conf.

Security and remote access considerations

Splunk requires privileged access to index many Windows data sources, including WMI, Event Log, and the registry. This includes both the ability to connect to the box, as well as permissions to read the appropriate data once connected. To access WMI data, Splunk must run as a user with permissions to perform remote WMI connections. This user name must be a member of an Active Directory domain and must have appropriate privileges to query WMI. Both the Splunk server making the query and the target systems being queried must be part of this Active Directory domain.
Note: If you installed Splunk as the LOCAL SYSTEM user, WMI remote authentication will not work; this user has null credentials and Windows servers normally disallow such connections.

There are several things to consider:

Test access to WMI

Follow these steps to test the configuration of the Splunk server and the remote machine:

1. Log into the machine Splunk runs on as the user Splunk runs as.
2. Click Start -> Run and type wbemtest. The wbemtest application starts.
3. Click Connect and type \\<server>\root\cimv2, replacing <server> with the name of the remote server. Click Connect. If you are unable to connect, there is a problem with the authentication between the machines.
4. If you are able to connect, click Query and type select * from win32_service. Click Apply. After a short wait, you should see a list of running services. If this does not work, then the authentication works, but the user Splunk is running as does not have enough privileges to run that operation.

Configure WMI input

Look at wmi.conf to see the default values for the WMI input. If you want to make changes to the default values, edit a copy of wmi.conf in $SPLUNK_HOME\etc\system\local\. Only set values for the attributes you want to change for a given type of data input. Refer to How configuration files work for more information about how Splunk uses configuration files.

[settings]
initial_backoff = 5
max_backoff = 20
max_retries_at_max_backoff = 2
result_queue_size = 1000
checkpoint_sync_interval = 2
heartbeat_interval = 500

[WMI:AppAndSys]
server = foo, bar
interval = 10
event_log_file = Application, System, Directory Service
disabled = 0

[WMI:LocalSplunkWmiProcess]
interval = 5
wql = select * from Win32_PerfFormattedData_PerfProc_Process where Name = "splunk-wmi"
disabled = 0

The [settings] stanza specifies runtime parameters. The entire stanza and every parameter within it are optional. If the stanza is missing, Splunk assumes system defaults.

You can specify two types of data input: event log, and raw WQL (WMI query language) The event log input stanza contains the event_log_file parameter, and the WQL input stanza contains wql.

The common parameters for both types are:

WQL-specific parameters:

Event log-specific parameter:
event_log_file: specify a comma-separated list of log files to poll in the event_log_file parameter. File names that include spaces are supported, as shown in the example.

Fields for WMI data

All events received from WMI have the source set to wmi.

The host is identified automatically from the data received.

Windows registry input

Splunk supports the capture of Windows registry settings and lets you monitor changes to the registry. You can know when registry entries are added, updated, and deleted. When a registry entry is changed, Splunk captures the name of the process that made the change and the key path from the hive to the entry being changed.

The Windows registry input monitor application runs as a process called splunk-regmon.exe.

Note: This feature is not currently supported on Windows 2000 due to an issue with a Windows 2000 dll (PSAPI.DLL).

Warning: Do not stop or kill the splunk-regmon.exe process manually; this could result in system instability. To stop the process, stop the Splunk server process from the Windows Task Manager or from within Splunk Web.

How it works

Windows registries can be extremely dynamic (thereby generating a great many events). Splunk provides a two-tiered configuration for fine-tuning the filters that are applied to the registry event data coming into Splunk.

Splunk Windows registry monitoring uses two configuration files to determine what to monitor on your system, sysmon.conf and regmon-filters.conf, both located in $SPLUNK_HOME\etc\system\local\. These configuration files work as a hierarchy: