About configuration files

About configuration files

Splunk's configuration information is stored in .conf files. These files are located under your Splunk installation directory (referred to in the documentation as $SPLUNK_HOME) under etc/system, etc/apps, and etc/users.

When you make a change to a configuration setting (we call them attributes) in Splunk Manager, Splunk's Web-based management interface (within Splunk Web), the change gets written to the relevant configuration file. This change is written to a copy of the appropriate configuration file in $SPLUNK_HOME/etc/system/local, and the default value of the attribute is left alone in $SPLUNK_HOME/etc/system/default.

You can do a lot of your configuration from within Manager, but for some of the more sophisticated possible customizations, you must edit these configuration files by hand. As Splunk matures, more and more of its options are available for configuration from within Manager, but until then, configuration files are the path to Splunk enlightenment.

The following is the relevant directory structure that exists under $SPLUNK_HOME/etc/system:

• $SPLUNK_HOME/etc/system/default
  • contains the pre-configured configuration files. Do not modify the files in default.
• $SPLUNK_HOME/etc/system/local
  • Local changes on a site-wide basis go here, for example settings you wish to be available to all apps.
• $SPLUNK_HOME/etc/apps/search/local/
  • Edits for search-time settings in the default splunk search interface go here. Settings for custom apps that you build go in the appropriate app directory.
• $SPLUNK_HOME/etc/system/README
  • Supporting documentation. This directory contains example and spec configuration files for you to refer to as you create your own configuration files. For most configuration files, there are two reference files; .spec and .example. For example, inputs.conf.spec and inputs.conf.example. The .spec file is a specification of syntax, including which attributes and variables are available. The .example files are helpful examples of real-world usage. These files are all found in the $SPLUNK_HOME/etc/system/README directory.

Note: The most accurate and up-to-date list of settings available for a given configuration file is in the .spec file for that configuration file. You can find the latest version of the .spec and .example files in the Configuration file reference in this manual, or in $SPLUNK_HOME/etc/system/README.

Creating and editing configuration files on non-UTF-8 OSes

Splunk expects configuration files to be in ASCII/UTF-8. If you are editing or creating a configuration file on an OS that is non-UTF-8, you must ensure that the editor you are using is configured to save in ASCII/UTF-8.

Configuration file precedence

Splunk configuration files layer on top of one-another. Multiple instances of the same configuration file and the stanzas within that file (with differing setting values) can and will exist in your deployment. Splunk determines which files and stanzas to use based on their locations within the directory structure.

What this means is that when you edit a configuration file, you should not edit the values in the copy of that file in $SPLUNK_HOME/etc/system/default, but instead make a copy of it and put it in a directory that Splunk will look at (and use the attribute values from) before the default directory. Most people use the $SPLUNK_HOME/etc/system/local directory, but you can create your own directory under /system as well. You can get fancy with the configuration files and layer several versions on top of one-another, with different attribute values filtering through and being used by Splunk as described later in this topic, but most deployments just use the $SPLUNK_HOME/etc/system/local directory to make configuration changes.

Another reason not to edit the copies of the configuration files $SPLUNK_HOME/etc/system/defaultis that when you upgrade Splunk, all your changes will be overwritten. Changes you make to files in other (/local or custom) directories are not overwritten and will continue to take effect post-upgrade.

Configuration files are evaluated in the following order:

• local -- local changes and preferences are evaluated first.
• user-created directories -- these are evaluated in alphabetical order.
• default -- Splunk's default settings are evaluated last.

Troubleshooting configuration precedence and other issues

Splunk's configuration file system supports many overlapping configuration files in many different locations. The price of this level of flexibility is that figuring out which value for which configuration option is being used in your Splunk installation can sometimes be quite complex. If you're looking for some tips on figuring out what configuration setting is being used in a given situation, check out "Troubleshooting configurations" in this manual.

Order of precedence

Directories are evaluated in the following order:

$SPLUNK_HOME/etc/users/*
$SPLUNK_HOME/etc/system/local/*
...
$SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/apps/A/local/*
...
$SPLUNK_HOME/etc/apps/Z/local/*
$SPLUNK_HOME/etc/apps/A/default/*
...
$SPLUNK_HOME/etc/apps/Z/default/*
$SPLUNK_HOME/etc/system/default/*

with the following considerations:

• /etc/users/ is considered when that user logs in and in search time.
• /etc/users/ is ignored for back end processing; at line merging, date extraction and index-time regexing time, no user context exists.
• /etc/apps/ is considered within the context of the specified app (props, inputs, transforms, etc.)
• /etc/system/ is considered as a global attribute (indexes, server, authorize, authentication, etc.)

Numbered directories are evaluated in the following order:

$SPLUNK_HOME/etc/apps/myapp1
$SPLUNK_HOME/etc/apps/myapp10
$SPLUNK_HOME/etc/apps/myapp2
$SPLUNK_HOME/etc/apps/myapp20
...

Attribute precedence

Precedence is applied attribute-by-attribute. What this means is that if the file props.conf exists in /etc/system/default and also in /etc/system/local, the props.conf file in /system/default does not just override or replace the entire props.conf file. If the same attribute/value pair exists in both the /system props.conf and the /local props.conf, the local props.conf overrides just that attribute.

For example, if $SPLUNK_HOME/etc/system/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = t2rss-error

And $SPLUNK_HOME/etc/apps/t2rss/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

Both the sourcetype assignment in local and the line merging attributes in t2rss apply. However, if both local and t2rss have a sourcetype assignment for source::/opt/Locke/Logs/error*, the assignment in local overrides t2rss.

Precedence within sets of stanzas affecting the same target

When two or more stanzas specify a behavior that affects the same item, items are evaluated in ASCII order. For example, if you specify in props.conf the following:

[source::.../bar/baz]
attr = val1

[source::.../bar/*]
attr = val2

the first stanza's value for attr is not used, because the second stanza's path is higher in the ASCII order and takes precedence.

Overriding ASCII priority in props.conf

Starting in version 4.0.7 of Splunk, there is a way to override this default order. Use the priority key to specify a higher or lower priority for a given attribute.

For example, suppose we have a source:

    source::az

and the following colliding patterns:

    [source::...a...]
    sourcetype = a

    [source::...z...]
    sourcetype = z

In this case, the settings provided by the pattern "source::...a..." take precedence over those provided by "source::...z...", and sourcetype will have the value "a".

To override this default ASCII ordering, use the priority key:

    [source::...a...]
    sourcetype = a
    priority = 5

    [source::...z...]
    sourcetype = z
    priority = 10

Assigning a higher priority to the second stanza causes sourcetype to have the value "z".

If not specified, the default value for the priority key is 0.

You can use the priority key to resolve collisions between sourcetype patterns and between host patterns. Note, however, that the priority key will not affect precedence across spec types. For example, source patterns take priority over host and sourcetype patterns, regardless of priority key values.

Important: The supported range of values for this setting is 0-99. You can set it higher, but Splunk does not recommend this.

Precedence for events with multiple attribute assignments

Beyond the above rules for precedence, there is an additional precedence issue that affects only props.conf. props.conf sets attributes for processing individual events by host, source or source type (and sometimes event type). So it's possible for one event to have the same attribute set differently for the default fields: host, source or sourcetype. The precedence order is:

• source
• host
• sourcetype

You may want to override default props.conf settings. For example, you are tailing mylogfile.xml, which by default is labeled sourcetype = xml_file. This configuration will re-index the entire file whenever it changes, even if you manually specify another sourcetype, because the property is set by source. To override this, add the explicit configuration by source:

[source::/var/log/mylogfile.xml]
CHECK_METHOD = endpoint_md5