Documentation: 3.2.1
|  Print Version Contents
This page last updated: 09/03/08 02:09pm

Configure inputs via inputs.conf

Add data inputs via inputs.conf. This allows for more granularity in your configuration than setting up inputs via SplunkWeb or the CLI.

Note: To set dynamic indexing properties for inputs, use props.conf.

Configuration

Add your stanza to $SPLUNK_HOME/etc/bundles/local/inputs.conf. Specify an input type and any number of attribute/value pairs.

[<inputtype>://<path>]
attribute1 = val1
attribute2 = val2
...

Global settings

The following attributes/value pairs are valid for ALL input types

host = <string>

  • Set the host value of your input to a static value.
  • "host =" is automatically prepended to the value when this shortcut is used.

index = <string>

  • Set the index where events from this input will be stored.
  • "index =" is automatically prepended to the value when this shortcut is used.

source = <string>

  • Set the source name of events from this input.
  • "source =" is automatically prepended to the value when this shortcut is used.

sourcetype = <string>

  • Set the sourcetype name of events from this input.
  • "sourcetype =" is automatically prepended to the value when this shortcut is used.

queue = <string> (parsingQueue, indexQueue, etc)

  • Specify where the input processor should deposit the events that it reads.
  • Can be any valid, existing queue in the pipeline.

Input types

The following attributes/value pairs are valid for the specified input types only.

Tail

[tail://<path>]

This directs Splunk to watch all files in the <path> (or just <path> itself if it represents a single file). You must specify the input type and then the path, so put three slashes in your path if you're starting at root. You can use wildcards for the path; see below.

Note: To ensure new events are indexed when you copy over an existing file with new contents, set CHECK_METHOD = modtime in props.conf for the source. This checks the modtime of the file and re-indexes when it changes. Note that the entire file is indexed, which can result in duplicate events.

Wildcards

You can use wildcards to specify your input path for tail input. Use ... for paths and * for files.

  • ... recurses through directories until the match is met. This means that /foo/.../bar will match foo/bar, foo/1/bar, foo/1/2/bar, etc. but only if bar is a file.
    • To recurse through a subdirectory, use another .... For example /foo/.../bar/....
  • * matches anything in that specific path segment. It cannot be used inside of a directory path; it must be used in the last segment of the path. For example /foo/*.log matches /foo/bar.log but not /foo/bar.txt or /foo/bar/test.log.
  • Combine * and ... for more specific matches:
    • foo/.../bar/* matches any file in the bar directory within the specified path.

Note: In Windows, you must use two backslashes \\ to escape wildcards. Regexes with backslashes in them are not currently supported for _whitelist and _blacklist in Windows.

Specifying wildcards results in an implicit _whitelist created for that stanza. The longest fully qualified path is used as the tail stanza, and the wildcards are translated into regular expressions using the following map:

wildcard regex meaning
* [^/]* anything but /
... .* anything (greedy)
. \. literal .

For example, if you specify

[tail:///foo/bar*.log]

Splunk translates this into
[tail:///foo/]
_whitelist = bar[^/]*\.log

As a consequence, you can't have multiple stanzas with wildcards for files in the same directory.

For example:

[tail:///foo/bar_baz*]
[tail:///foo/bar_qux*]

This results in overlapping stanzas indexing the directory /foo/. Splunk takes the first one, so only files starting with /foo/bar_baz will be indexed. To encompass both sources, manually specify a _whitelist using regular expression syntax for "or":
[tail:///foo]
_whitelist = (bar_baz[^/]*|bar_qux[^/]*)

Note: To set any additional attributes (such as sourcetype) for multiple whitelisted/blacklisted inputs that may have different attributes, use props.conf

Additional attributes

host_regex = <regular expression>

  • If specified, the regex will extract host from the filename of each input.
  • Specifically, the first group of the regex is used as the host.
  • If the regex fails to match, the default "host =" attribute is used.

host_segment = <integer>

  • If specified, the '/' separated segment of the path will be set as host.
  • If the value is not an integer, or is less than 1, the default "host =" attribute is used.

crcSalt = <string>

  • If set, this string will be added to the CRC.
  • This can be used to force Splunk to consume files that have matching CRCs.
  • If the string <SOURCE> is specified, then the full source path will be added to the CRC.

followTail = 0|1

  • If set to 1, tailing will begin at the end of the file (like tail -f).
  • This will only apply to files the first time they are picked up.
  • After that, Splunk's internal file position records keep track of the file.

_whitelist = <regular expression>

  • If set, files from this path are tailed only if they match the specified regex.

_blacklist = <regular expression>

  • If set, files from this path are NOT tailed if they match the specified regex.

Batch

[batch://<path>]
  • Same as tailing, except Splunk uses the batch file loader.
  • This is for files that are closed for writing.
  • For open files, use tail.

Additional attributes


move_policy = (passive_symlink, passive_copy, sinkhole)

  • Set the file handling policy.
  • The "sinkhole" policy deletes the files as they are read.
  • The other two methods link or copy the files into a separate directory.
  • Defaults to passive_symlink.

host_regex (see tail)
host_segment (see tail)

Note: source = <string> and <KEY> = <string> are not used by batch.

TCP

[tcp://<remote server>:<port>]
  • Configure Splunk to listen on a specific port.
  • If a connection is made from <remote server>, this stanza is used to configure the input.
  • If <remote server> is blank, this stanza matches all connections on the specified port.

Additional attributes

connection_host = [ip | dns]

  • If "ip" is set, the TCP input processor will rewrite the host with the ip address of the remote server.
  • If "dns" is set, the host will be rewritten with the DNS entry of the remote server.
  • Defaults to ip.

UDP

[udp://:<port>]
  • Similar to TCP, except that it listens on a UDP port.

Additional attributes

_rcvbuf = <int>

  • Specify the receive buffer for the UDP port.
  • If the value is 0 or negative, it will be ignored.
  • The default value for Splunk is 1MB (the default in the OS varies).

no_priority_stripping = <value>

  • If this attribute is set to any value, then Splunk does NOT strip the <priority> syslog field from received events.
  • This key should not be included if you want to strip <priority>.

FIFO

[fifo://<path>]
  • This directs Splunk to read from a FIFO at the specified path.

Scripted Input

[script://<cmd>]
  • Will run the command "cmd" at a configured interval and index the output.
  • The command must reside in $SPLUNK_HOME/etc/bundles/$YOUR_BUNDLE/bin directory.

interval = <integer>

  • How often to execute the specified command (in seconds).
  • If interval is not specified, it will default to 60 seconds.

passAuth = <username>

  • User that this script should be run as.
  • If a username is provided, Splunk generates an auth token for that user and passes it to the script via stdin.

Examples

Tail

[tail:///apache/.../logs]

This loads anything in /apache/foo/logs or /apache/bar/logs, etc.

[tail:///apache/*.log]

This loads anything in /apache/ that ends in .log.

Batch (aka Watch)

[batch://system/flight815/*]
move_policy = sinkhole

This example batch loads all files from the directory /system/flight815/. move_policy = sinkhole deletes the files from the directory.

TCP

[tcp://<remote server>:<port>]

This configures Splunk to listen on the specified port. If a connection is made from <remote server>, this stanza is used to configure the input.
If <remote server> is blank, this stanza matches all connections on the specified port.

UDP

[udp://<remote-server>:<port>]

Similar to TCP, except that Splunk listens on a UDP port.

FIFO

[fifo://<path>]

This directs Splunk to read from the FIFO at the specified path.
Previous: Configure inputs via the CLI    |    Next: Configure inputs for Windows

Comments

  1. typo fixed.
    Posted by emma on Mar 25 2008, 11:00am

  2. Typo in "The "sinkhole" policy deletse the files as they are read."
    Posted by meeas on Mar 10 2008, 12:54pm

Log in to comment.