How do configuration files work?

Splunk's configurations are affected via configuration files. Even configurations set up through Splunk Web or the CLI are written out to configuration files. Set up more advanced configurations in configuration files, or make an application. Learn more about application configuration, including best practices.

Once you have created a working application for a single Splunk server, you can then distribute it to target servers through the Splunk deployment server or share them with others through SplunkBase.

Restart your Splunk server to apply any changes you make to the configuration files. Changes to how Splunk processes index data do not affect data that is already indexed.

Note the following exceptions to this:

Enable configurations changes made to transforms.conf by typing the following search in Splunk Web:

| extract reload=T

Bounce authentication.conf via the Admin -> Server section of Splunk Web.

Configuration file directory structure

There are two general configuration file directories in $SPLUNK_HOME/etc/:

$SPLUNK_HOME/etc/system/
- The system directory contains default shipped Splunk configurations and user-created content to override these settings.
$SPLUNK_HOME/etc/apps/
- The apps directory stores downloaded and custom built applications.
- Store custom configurations in custom directories underneath ../apps/.

NOTE: There is also a legacy directory in $SPLUNK_HOME/etc/bundles to support prior versions' configurations and the deployment server.

Both system/ and the application directories in apps/ have the same directory structure:

default/
- Settings in default/ should not be changed.
local/
- Make all custom edits here, including overriding settings in default/.
readme/
- Supporting documentation.
bin/
- Scripts that support the application, such as searchscripts, custom web scripts, and REST endpoint handlers.
static/
- Files served by the HTTP server and other static files (non-executable).

For example:

    apps/
        myapp1/
            default/
            local/
            static/
            bin/
        myapp2/
            default/
            local/
            static/
            bin/

Your Splunk server ships with several such directories, including:

default - contains the pre-configured configuration files. Do not modify the files in default.

Note: Not all configuration files appear in default/.

local - stores modifications you make through the web interface or command line. You can make file edits here, or in a custom application directory.

Note: If you edit files that are also written to by Splunk Web, your edits may be overridden if someone else is editing Splunk Web at the same time.

learned - this set of configurations are settings created by the Splunk Server as it trains on incoming data.
readme - this directory contains example and spec configuration files that can help you create your own configuration files. For each configuration file, there are two reference files; .spec and .example. For example, inputs.conf.spec and inputs.conf.example. The .spec file is a specification of syntax, including which attributes and variables are available. The .example files are helpful examples of real-world usage. These files are all found in the $SPLUNK_HOME/etc/system/README directory.

Configuration file precedence

Configuration files live in multiple places: default, local and any custom application directories you create. Configuration files are evaluated in the following order:

local -- local changes and preferences are evaluated first.
user-created directories -- these are evaluated in alphabetical order.
default -- Splunk's default settings are evaluated last.

NOTE: Any configurations set in $SPLUNK_HOME/etc/bundles take precedence over configurations in $SPLUNK_HOME/etc/system or $SPLUNK_HOME/etc/apps.

Example

Directories are evaluated in the following order:

$SPLUNK_HOME/etc/bundles/local/*
$SPLUNK_HOME/etc/bundles/A/*
...
$SPLUNK_HOME/etc/bundles/Z/* $SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/apps/A/local/*
...
$SPLUNK_HOME/etc/apps/Z/local/*
$SPLUNK_HOME/etc/apps/A/default/*
...
$SPLUNK_HOME/etc/apps/Z/default/*
$SPLUNK_HOME/etc/system/default/*

Numbered directories are evaluated in the following order:

$SPLUNK_HOME/etc/apps/myapp1
$SPLUNK_HOME/etc/apps/myapp10
$SPLUNK_HOME/etc/apps/myapp2
$SPLUNK_HOME/etc/apps/myapp20
...

Attribute precedence

Precedence is applied attribute-by-attribute. That is, if the file props.conf exists in local and a user created configuration file directory, the props.conf file in local does not override or replace the entire props.conf file. If the same attribute/specification exists in both the local props.conf and the user-created props.conf, the local props.conf overrides the attribute.

For example, if $SPLUNK_HOME/etc/system/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = t2rss-error

And $SPLUNK_HOME/etc/apps/t2rss/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

Both the sourcetype assignment in local and the line merging attributes in t2rss apply. However, if both local and t2rss have a sourcetype assignment for source::/opt/Locke/Logs/error*, the assignment in local overrides t2rss.

Precedence rules for events with multiple attribute assignments

Beyond the above rules for precedence, there is an additional precedence issue that affects only props.conf. props.conf sets attributes for processing individual events by host, source or sourcetype (and sometimes eventtype). So it's possible for one event to have the same attribute set differently for the default fields: host, source or sourcetype. The precedence order is:

source
host
sourcetype

You may want to override default props.conf settings. For example, you are tailing mylogfile.xml, which by default is labeled sourcetype = xml_file. This configuration will re-index the entire file whenever it changes, even if you manually specify another sourcetype, because the property is set by source. To override this, add the explicit configuration by source:

[source::/var/log/mylogfile.xml]
CHECK_METHOD = endpoint_md5

Previous: 64-bit operating systems | Next: Configure application directories

Comments

No comments have been submitted.