This document last updated: 05/02/08 05:05pm

Print Developer Manual

Overview

Overview

These topics provide tools for extending and customizing Splunk. Most custom Splunk development involves one or more of the following:

• Extension via the Splunk REST API
• Implementation of custom configurations
• Changing the appearance of Splunk Web

The following information and examples are available:

• Splunk's REST API reference
• Information and instructions for using applications to customize Splunk configurations
• Information and instructions for changing the appearance of Splunk Web (sometimes referred to as "skinning" or "rebranding")

Visit the Splunk developer community for more examples.

Splunk Architecture

Splunk is a high performance, scalable software server written in C/C++ and Python. It indexes and searches logs and other IT data in real time. Splunk works with data generated by any application, server or device. After downloading, installing, and starting Splunk, you'll find two Splunk Server processes running on your host, splunkd and splunkweb.

• splunkd is a distributed C/C++ server that accesses, processes and indexes streaming IT data and also handles search requests. splunkd processes and indexes your data by streaming it through a series of pipelines, each made up of a series of processors.
  • Pipelines are single threads inside the splunkd process, each configured with a single snippet of XML.
  • Processors are individual, reusable C or C++ functions that act on the stream of IT data passing through a pipeline. Pipelines can pass data to one another via queues. splunkd supports a command line interface for searching and viewing results.

• splunkweb is a Python-based application server providing the Splunk Web user interface. It allows users to search and navigate IT data stored by Splunk servers and to manage your Splunk deployment through the browser interface.

splunkweb and splunkd can both communicate with your web browser via REST.

• Splunk's data store manages the original raw data in compressed format as well as the indexes into the data. Data can be deleted or archived based on retention period or maximum data store size.

• Splunk Servers can communicate with one another using a TCP-based protocol to forward data from one server to another and to distribute searches across multiple servers.

• Configuration bundles are directories of files that contain configuration settings including, user accounts, saved searches, data inputs and processing properties to easily create specific Splunk environments.

• Modules are files that add new functionality to Splunk by adding to or modifying existing processors and pipelines. They can include C++ code libraries.

REST API

Splunk's REST API

REST is a programming method that provides simple access to Web-based resources. If you'd like to know more about REST methods, Wikipedia has an article on it titled Representational State Transfer.

Using REST Methods

HTTP has an uniform interface for accessing resources, which consists of URIs, methods, status codes, headers, and content distinguished by MIME type.

The most important HTTP methods are POST, GET, PUT and DELETE. These are often compared with the CREATE, READ, UPDATE, DELETE (CRUD) operations associated with database technologies.

The following table associates several common HTTP verbs with similar database operations, however the meaning of the HTTP verbs do not correspond directly with a single database operation. For example, an HTTP PUT is used to set the value of a resource and may result in either a creation or update as needed.

Splunk REST endpoint mappings

Starting in version 3.2, Splunk's REST endpoints are served via SSL off the splunkd process using the URL format: https://hostname:port/services/, where hostname is your Splunk server's hostname, and port is the port number on which the splunkd process is listening.

For example, if you are logged into the local machine and Splunk is running on the default ports, you can use https://localhost:8089/services/ to access the REST endpoints. Remember, your mileage may vary depending on the machine's hostname, ports, registered certificates, and firewall settings.

Note: All examples in this documentation assume you are logged into the local machine and that Splunk is running on the default ports

HTTP ports Splunk uses

Splunk listens on the following ports:

• Splunk Web listens on port 8000 by default
• splunkd listens on port 8089 by default

Connections to splunkd, splunkd are encrypted by default.

What you need to know about searching with Splunk

Before you can build effective extensions to Splunk using the REST API, you should understand some basics about how Splunk works. You're building queries, so you'll need to understand the Splunk search language. Splunk is an IT Data search engine. It allows you to index any format of IT data from any source in real time from all of your applications, servers, and devices, and then search, navigate, alert, and report on all this data in real time.

What's the data type that you're getting back?

• might be search results, or events
• might be timechart data

This is important because if you want to draw a graph, you'll want it run through the timechart operator.

How are you handling time ranges?

Time values are passed in as header parameters.
You can pass time values as starttime and endtime in epoch seconds (which you must do if you pass them this way), or you can pass them in the search string itself.
To see how that works, use Splunk Web to build queries. Try searching for something over a custom time range. If you go to the saved search window, you can copy in the query to use.
You can also specify times relative to "now". Here is the BNF for the relative time arguments:

<rel_time> ::= "now" | ("-"|"+")<integer><unit>
<unit> ::= "s" | "m" | "h" | "d" | "w" | "m" | "y"

If "now" is given, the unadulterated result of a call to the system call "time" is returned. Otherwise, the current time is converted into a broken down time, and the specified quantity is subtracted or added from that component of the broken down time.
For example, suppose "now" is 10/9/2007, 07:32:15, the relative specifier "+2d" corresponds to 10/11/2007, 07:32:15.

What extracted fields are available for use in your query?

You should know what these are. Any fields that are extracted at search time are available. Be aware that when you search, multiple field extractions are being created and returned to the interface, although you may not see them all.

The following query gives you the number of occurrences and distinct values of each field in the most recent <maxresults> of sourcetype=foo
> sourcetype=foo | stats count(*) dc(*)

You can append | transpose to this search to make the output easier to read.
If you want this information over all results, perform the same search using the CLI dispatch command, which is useful for long-running searches.

Other important topics

• Configuration files

Be sure you understand how configuration files work.

• Users

Any authentication that you invoke will use REST to authenticate against users you create in Splunk.
Note: If you're using the free version of Splunk, the username and password fields will be blank, as you cannot create users in the free version.

• Saved searches

You can call saved searches from within your code, but if it's possible for someone else to delete your saved search, your code will fail.

Login

/services/auth/login

Provides user authentication

POST

Returns a session key to be used when making REST calls to splunkd

Form Arguments

username - The Splunk account username
password - The corresponding password

Response Status

200 - User successfully authenticated
400 - Username or password not provided
401 - Login credentials failed

Response Body

// sample response to /services/auth/login

<response>
    <sessionKey>87ad615df536e3</sessionKey>
</response>

Legacy methods

/services/invokeapi/legacy_method_name

Provides access to legacy invokeAPI method calls.

GET

Calls the invokeAPI method, as defined in legacy_method_name.

The invokeAPI method calls originally used XML as the input parameter. In order map XML into flat querystring arguments, the following translation is used:

* <key>foo</key> ==> key=foo
* <key attr1="bar">foo</key> ==> key=foo&key@attr1=bar
* <key><key2>foo</key2></key> ==> key.key2=foo

So, for example:

<call name="getUserInfo">
    <params>
        <key1>foo</key1>
        <key2 attr1="bar">baz</key2>
        <key3>
            <key4>boo</key4>
        </key3>
    </params>
</call>

is called by converting the XML node and attribute names into serialized strings:

/services/invokeapi/getUserInfo?key1=foo&key2=baz&key2@attr1=bar&key3.key4=boo

Query Arguments

<legacy_arg>
The key name corresponds to the XML parameter, as defined by the XML structure. The value passed is the value assigned to the key name.

Response Status

200 - Method executed successfully
401 - Login credentials failed
500 - There was an error; see body contents for messages

Response Body

// The return content is arbitrary XML returned by each method.

// sample response to /services/invokeapi/getUserInfo?userId=1
<user>
  <id>1</id>
  <name>admin</name>
  <password>********</password>
  <realName>Administrator</realName>
  <userType>
    <role>Admin</role>
  </userType>
</user>

POST

Identical to GET, except that arguments are passed via form arguments.

Properties

/services/properties

Provides access to the configuration values

GET

Returns an Atom feed of top level configuration files

Response Status

200 - OK

Response Body

// sample response to /services/properties

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">
    <title>properties</title>
    <id>https://localhost:8089/services/properties</id>
    <updated>2008-01-29T11:40:58-0800</updated>
    <generator version="31758"/>
    <author>
        <name>Splunk</name>
    </author>
    <entry>
        <title>alert_actions</title>
        <id>https://localhost:8089/services/properties/alert_actions</id>
        <updated>2008-01-29T11:40:58-0800</updated>
        <link href="https://localhost:8089/services/properties/alert_actions" rel="alternate"/>

    </entry>
    <entry>
        <title>api</title>
        <id>https://localhost:8089/services/properties/api</id>
        <updated>2008-01-29T11:40:58-0800</updated>
        <link href="https://localhost:8089/services/properties/api" rel="alternate"/>
    </entry>
</feed>

/services/properties/file_name

Provides access to the configuration values for a specific file

GET

Returns an Atom feed of stanzas contained in file_name

Response Status

200 - OK

Response Body

// sample response to /services/properties/alert_actions

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">
    <title>alert_actions</title>
    <id>https://localhost:8089/services/properties/alert_actions</id>
    <updated>2008-01-29T11:43:21-0800</updated>
    <generator version="31758"/>
    <author>

        <name>Splunk</name>
    </author>
    <entry>
        <title>default</title>
        <id>https://localhost:8089/services/properties/alert_actions/default</id>
        <updated>2008-01-29T11:43:21-0800</updated>
        <link href="https://localhost:8089/services/properties/alert_actions/default" rel="alternate"/>

    </entry>
    <entry>
        <title>email</title>
        <id>https://localhost:8089/services/properties/alert_actions/email</id>
        <updated>2008-01-29T11:43:21-0800</updated>
        <link href="https://localhost:8089/services/properties/alert_actions/email" rel="alternate"/>
    </entry>

    <entry>
        <title>rss</title>
        <id>https://localhost:8089/services/properties/alert_actions/rss</id>
        <updated>2008-01-29T11:43:21-0800</updated>
        <link href="https://localhost:8089/services/properties/alert_actions/rss" rel="alternate"/>
    </entry>
</feed>

POST

Creates a new stanza within file_name

Form Arguments

__stanza - The name of the stanza to create

Response Status

201 - Stanza was successfully created; will be followed by header Location: /services/properties/[stanza_name]
303 - Stanza already exists; will be followed by header Location: /services/properties/[stanza_name]
400 - Form arguments were invalid

Response Body

[no response body, unless error occurs]

/services/properties/file_name/stanza_name

Provides access to the configuration values for a stanza within a specific file

GET

Returns an Atom feed of key/value pairs contained in the stanza

Response Status

200 - OK
404 - Stanza was not found in file_name

Response Body

// sample response to /services/properties/alert_actions/email

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">
    <title>email</title>
    <id>https://localhost:8089/services/properties/alert_actions/email</id>
    <updated>2008-01-29T11:55:24-0800</updated>
    <generator version="31758"/>
    <author>
        <name>Splunk</name>
    </author>
    <entry>
        <title>format</title>
        <id>https://localhost:8089/services/properties/alert_actions/email/format</id>
        <updated>2008-01-29T11:55:24-0800</updated>

        <link href="https://localhost:8089/services/properties/alert_actions/email/format" rel="alternate"/>
        <content type="text">csv</content>
    </entry>
    <entry>
        <title>from</title>
        <id>https://localhost:8089/services/properties/alert_actions/email/from</id>
        <updated>2008-01-29T11:55:24-0800</updated>

        <link href="https://localhost:8089/services/properties/alert_actions/email/from" rel="alternate"/>
        <content type="text">splunk</content>
    </entry>
    <entry>
        <title>inline</title>
        <id>https://localhost:8089/services/properties/alert_actions/email/inline</id>
        <updated>2008-01-29T11:55:24-0800</updated>

        <link href="https://localhost:8089/services/properties/alert_actions/email/inline" rel="alternate"/>
        <content type="text">false</content>
    </entry>
</feed>

POST

Adds or updates key/value pairs in the current stanza. One or more key/value pairs may be passed at one time to this endpoint.

Form Arguments

<key_name> - The argument name is the key to update; the value is the value to be set

Response Status

200 - Key value was successfully added/updated
400 - Form request was badly formed
404 - The stanza was not found
409 - One or more of the input values failed validation

Response Body

[upon successful write (HTTP 200), the response will be identical to the GET response; non-200 response will be standard message format]

PUT

Adds or overwrites the entire stanza block. If the stanza doesn't already exist, it will be created. The PUT method is useful for adding inline comments.

Request

<raw_payload> - The raw text of the stanza, excluding the stanza header declaration

Response Status

200 - Stanza was updated
201 - Stanza was created; will be followed by header Location: /services/properties/[stanza_name]. This is redundant, but follows spec.
404 - The file_name was not found

Response Body

[upon successful write (HTTP 20x), the response will be identical to the GET response; non-200 response will be standard message format]

/services/properties/file_name/stanza_name/key_name

Provides access to individual key/value within a stanza

GET

Returns the value of the key in plain text.

Response Status

200 - OK
404 - Key/stanza/file was not found

Response Body

<code>
// sample response to /services/properties/alert_actions/email/format

csv
</code>

POST

Updates an existing key value

Form Arguments

value - The argument name is the key to update; the value is the value to be set

Response Status

200 - Key value was successfully added/updated
400 - Form request was badly formed
404 - The stanza was not found
409 - The input value failed validation

Response Body

[upon successful write (HTTP 200), the response will be identical to the GET response; non-200 response will be standard message format]

PUT

Adds a new key to the stanza, or updates an existing key

Request

<raw_payload> - The raw value of the key

Response Status

200 - Key was updated
201 - Key was created
404 - The file_name or stanza_name was not found
409 - The value failed validation

Response Body

[upon successful write (HTTP 20x), the response will be identical to the GET response; non-200 response will be standard message format]

Search

/services/search/jobs

Provides listing for for jobs

GET

Returns a list of current searches. Optional filter arguments can be passed to specify searches. The user id is implied by the authetication to the call.

Response Status

200 - OK

Response Body

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">
    <title>jobs</title>
    <id>https://localhost:8089/services/search/jobs</id>
    <updated>2008-01-11T14:02:00-0800</updated>
    <generator version="30887"/>
    <author>
        <name>Splunk</name>
    </author>
    <entry>
        <title>search *</title>
        <id>https://localhost:8089/services/search/jobs/1014004140</id>
        <published>2008-01-11T14:02:00-0800</published>
        <updated>2008-01-11T14:02:00-0800</updated>
        <link href="https://localhost:8089/services/search/jobs/1014004140" rel="alternate"/>
        <link href="https://localhost:8089/services/search/jobs/1014004140/events" rel="events"/>
        <link href="https://localhost:8089/services/search/jobs/1014004140/results" rel="results"/>
        <link href="https://localhost:8089/services/search/jobs/1014004140/timeline" rel="timeline"/>
        <link href="https://localhost:8089/services/search/jobs/1014004140/summary" rel="summary"/>
        <link href="https://localhost:8089/services/search/jobs/1014004140/control" rel="control"/>
        <s:messages>
            <s:msg type="ERROR">my error text</s:msg>
            <s:msg type="WARN">my error text</s:msg>
        </s:messages>
        <author>
            <name>admin</name>
        </author>
        <content type="text/xml">
            <s:dict>
                <s:key name="sid">1014004140</s:key>
                <s:key name="eventCount">1234</s:key>
                <s:key name="cursorTime">2008-01-11T19:24:00-0800</s:key>
                <s:key name="isDone">0</s:key>
                <s:key name="isFinalized">1</s:key>
                <s:key name="keywords">none</s:key>
                <s:key name="isPaused">0</s:key>
                <s:key name="isStreaming">0</s:key>
                <s:key name="ttl">24.00 hours</s:key>
            </s:dict>
        </content>
    </entry>
</feed>

POST

Starts a new search job on Splunk server

Form Variables

search -This is the search language string that will be executed taking results from the local and remote servers.
remote_server_list - (comma separated list) (default: empty) This is the list of (possibly wildcarded) servers from which raw events should be pulled. This same server list is to be used in subsearches.
start_time - The earliest (inclusive), respectively, time bounds for the search. The time string can be either a UTC time (with fractional seconds), a relative time specifier (to now) or a formatted time string.
end_time - The latest (exclusive), respectively, time bounds for the search. The time string can be either a UTC time (with fractional seconds), a relative time specifier (to now) or a formatted time string.
time_format - Used to convert a formatted time string from {start,end}_time into UTC seconds. It defaults to ISO-9601.
status_buckets - [int] (default: 300) The most status buckets to generate.
with_event_types - {1,0} (default: 0) Specifies whether events should be processed for event types.
max_count - (default: 10000) The number of events that can be accessible in any given status bucket. Also, in transforming mode, the maximum number of results to store. Specifically, in all calls, offset+count <= max_count.
timeout - [int] (default: 86400) The number of seconds to keep this search after processing has stopped.
enable_eventtypes - Specfies whether eventtypes should be assigned to events. This option causes searches to be significantly slower and should be used with discretion.

Response Status

200 - OK
404 - something is very wrong

Response Body

<?xml version="1.0" encoding="UTF-8"?>
<response>
<sid>12345</sid>
</response>

/services/search/jobs/search_id

Represents an active search in the search system.

GET

Returns summary information about the search job

Response Status

200 - OK
404 - Search job id was not found on this server

Response Body

<?xml version="1.0" encoding="UTF-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">
    <title>search *</title>
    <id>https://localhost:8089/services/search/jobs/1014004140</id>
    <updated>2008-01-11T14:02:00-0800 (mtime)</updated>
    <published>2008-01-11T14:02:00-0800 (start)</published>
    <link href="https://localhost:8089/services/search/jobs/1014004140/events" rel="events"/>
    <link href="https://localhost:8089/services/search/jobs/1014004140/results" rel="results"/>
    <link href="https://localhost:8089/services/search/jobs/1014004140/timeline" rel="timeline"/>
    <link href="https://localhost:8089/services/search/jobs/1014004140/summary" rel="summary"/>
    <link href="https://localhost:8089/services/search/jobs/1014004140/control" rel="control"/>
    <s:messages>
        <s:msg type="ERROR">my error text</s:msg>
        <s:msg type="WARN">my error text</s:msg>
    </s:messages>
    <author>
        <name>admin</name>
    </author>
    <content type="text/xml">
        <s:dict>
            <s:key name="sid">1014004140</s:key>
            <s:key name="eventCount">1234</s:key>
            <s:key name="cursorTime">2008-01-11T19:24:00-0800</s:key>
            <s:key name="isDone">0</s:key>
            <s:key name="isFinalized">1</s:key>
            <s:key name="keywords">none</s:key>
            <s:key name="isPaused">0</s:key>
            <s:key name="isStreaming">0</s:key>
            <s:key name="ttl">24.00 hours</s:key>
        </s:dict>
    </content>
</entry>
 </code>           

DELETE

Deletes the current search job

Response Status

200 - OK
404 - Search job id was not found on this server

/services/search/jobs/search_id/events

Represents the raw events returned by the current search

GET

This is the primary method for a client to fetch a set of untransformed events. If the dispatched search includes a transforming command, the events here are those that would be transformed, not the final transformed results.

Request Query

offset - The first result (inclusive) from which to begin returning data. This value is 0-indexed. Default value is 0.
count - The maximum number of results to return. If value is set to 0, then all available results are returned. Default value is 100
start_time - The earliest (inclusive), respectively, time bounds for the results to be returned. If not specified, the range applies to all results found.
end_time - The latest (exclusive), respectively, time bounds for the results to be returned. If not specified, the range applies to all results found.
time_format - Used to convert a formatted time string from {start,end}_time into UTC seconds. It defaults to %m/%d/%Y:%H:%M:%S
field_list - [comma separated list] (default: *) A list of the fields to return for the event set.
max_lines - (default: 0) (NI) The most lines that any single event's _raw field should contain. The value 0 is used to specify no limit.
truncation_mode - {abstract,truncate} (default: abstract) (NI) Specifies how "max_lines" should be achieved.
output_mode - {csv,text,xml,json} (default: text) Specifies what format the output should be returned in.
segmentation - (default: raw) The type of segmentation to perform on the data. This will include an option to perform k/v segmentation.

Response Status

200 - Search events returned
204 - Search job was found, but the server has not finished preparing the events yet; retry your request.
404 - Search job id was not found on this server

Response Body

// sample JSON output
// https://localhost:8089/services/search/jobs/1234/events?output_mode=json

[
    {
        "_cd": "0:4374557",
        "_index": "main",
        "_kv": "1",
        "_meta": " date_second::36 date_hour::19 date_minute::11 date_year::2008 date_month::january date_mday::21 date_wday::monday date_zone::-480 punct::_[//:::_-]____\\\"@...\\\"...",
        "_raw": "I [21/Jan/2008:19:11:36 -0800] Added remote printer \"HPLaserJ@10.1.1.123\"...",
        "_serial": "0",
        "_time": "1200971496",
        "date_hour": "19",
        "date_mday": "21",
        "date_minute": "11",
        "date_month": "january",
        "date_second": "36",
        "date_wday": "monday",
        "date_year": "2008",
        "date_zone": "-480",
        "host": "decider.local",
        "linecount": "1",
        "punct": "_[//:::_-]____\"@...\"...",
        "source": "/var/log/cups/error_log",
        "sourcetype": "cups_error"
    },
    {
        "_cd": "0:4374549",
        "_index": "main",
        "_kv": "1",
        "_meta": " date_second::36 date_hour::19 date_minute::11 date_year::2008 date_month::january date_mday::21 date_wday::monday date_zone::-480 punct::_[//:::_-]____\\\"@...\\\"...",
        "_raw": "I [21/Jan/2008:19:11:36 -0800] Added remote printer \"HPLaserJ@10.1.5.65\"...",
        "_serial": "1",
        "_time": "1200971496",
        "date_hour": "19",
        "date_mday": "21",
        "date_minute": "11",
        "date_month": "january",
        "date_second": "36",
        "date_wday": "monday",
        "date_year": "2008",
        "date_zone": "-480",
        "host": "decider.local",
        "linecount": "1",
        "punct": "_[//:::_-]____\"@...\"...",
        "source": "/var/log/cups/error_log",
        "sourcetype": "cups_error"
    }
]

/services/search/jobs/search_id/results

This is the primary method for a client to fetch a set of TRANSFORMED events. If the dispatched search does not include a transforming command, the effect is the same as get_events, albeit with fewer options.

GET

Request Query

offset - The first result (inclusive) from which to begin returning data. This value is 0-indexed. Default value is 0.
count - The maximum number of results to return. If value is set to 0, then all available results are returned. Default value is 100
field_list - [comma separated list] (default: *) A list of the fields to return for the event set.
output_mode - {csv,text,xml,json} (default: text) Specifies what format the output should be returned in.

Response Status

200 - Search events returned
204 - Search job was found, but the server has not finished preparing the events yet; retry your request.
404 - Search job id was not found on this server

Response Body

    // sample JSON output
    // https://localhost:8089/services/search/jobs/1234/results?output_mode=json

    [
    {
    "_cd": "0:4374557",
    "_index": "main",
    "_kv": "1",
    "_meta": " date_second::36 date_hour::19 date_minute::11 date_year::2008 date_month::january date_mday::21 date_wday::monday date_zone::-480 punct::_[//:::_-]____\\\"@...\\\"...",
    "_raw": "I [21/Jan/2008:19:11:36 -0800] Added remote printer \"HPLaserJ@10.1.1.123\"...",
    "_serial": "0",
    "_time": "1200971496",
    "date_hour": "19",
    "date_mday": "21",
    "date_minute": "11",
    "date_month": "january",
    "date_second": "36",
    "date_wday": "monday",
    "date_year": "2008",
    "date_zone": "-480",
    "host": "decider.local",
    "linecount": "1",
    "punct": "_[//:::_-]____\"@...\"...",
    "source": "/var/log/cups/error_log",
    "sourcetype": "cups_error"
    },
    {
    "_cd": "0:4374549",
    "_index": "main",
    "_kv": "1",
    "_meta": " date_second::36 date_hour::19 date_minute::11 date_year::2008 date_month::january date_mday::21 date_wday::monday date_zone::-480 punct::_[//:::_-]____\\\"@...\\\"...",
    "_raw": "I [21/Jan/2008:19:11:36 -0800] Added remote printer \"HPLaserJ@10.1.5.65\"...",
    "_serial": "1",
    "_time": "1200971496",
    "date_hour": "19",
    "date_mday": "21",
    "date_minute": "11",
    "date_month": "january",
    "date_second": "36",
    "date_wday": "monday",
    "date_year": "2008",
    "date_zone": "-480",
    "host": "decider.local",
    "linecount": "1",
    "punct": "_[//:::_-]____\"@...\"...",
    "source": "/var/log/cups/error_log",
    "sourcetype": "cups_error"
    }
    ]

/services/search/jobs/search_id/timeline

Provides "timeline" output of the so-far-read untransformed events.

GET

Returns the timeline data

Request Query

time_format
Used to convert a formatted time string from {start,end}_time into UTC seconds. It defaults to %m/%d/%Y:%H:%M:%S

Response Status

200 - OK
204 - Search id was found, but the server has not finished preparing the events yet; retry your request.
404 - Search id was not found on server

Response Body

<?xml version="1.0"?>
<timeline c="478586" cursor="1143878400">
    <bucket c="2" t="1143878400.000" d="2588400" f="1">2006-04-01T00:00:00.000-08:00</bucket>
    <bucket c="0" t="1146466800.000" d="2678400" f="1">2006-05-01T00:00:00.000-07:00</bucket>
    <bucket c="0" t="1149145200.000" d="2592000" f="1">2006-06-01T00:00:00.000-07:00</bucket>
    ...
    <bucket c="37620" t="1191222000.000" d="2678400" f="1">2007-10-01T00:00:00.000-07:00</bucket>
    <bucket c="108760" t="1193900400.000" d="2595600" f="1">2007-11-01T00:00:00.000-07:00</bucket>
    <bucket c="102507" t="1196496000.000" d="2678400" f="1">2007-12-01T00:00:00.000-08:00</bucket>
    <bucket c="67179" t="1199174400.000" d="2678400" f="1">2008-01-01T00:00:00.000-08:00</bucket>
</timeline>

/services/search/jobs/search_id/summary

Provides "getFieldsAndStats" output of the so-far-read untransformed events.

GET

Returns the summary output

Request Query

start_time - The earliest (inclusive), respectively, time bounds for the results to be returned. If not specified, the range applies to all results found.
end_time - The latest (exclusive), respectively, time bounds for the results to be returned. If not specified, the range applies to all results found.
time_format - Used to convert a formatted time string from {start,end}_time into UTC seconds. It defaults to %m/%d/%Y:%H:%M:%S
field_list - [comma separated list] (default: *) A list of the fields to return for the event set.
top_count - (default: 10) For each key, this number of the most frequent items will be returned.

Response Status

200 - Action was executed successfully
403 - Not authorized to execute action
404 - Search id was not found on server

/services/search/jobs/search_id/control

Provides job control handle for current search

POST

Executes a job control command

Request Form

action - The control action to execute
pause - Suspends the execution of the current search
unpause - Resumes the execution of the current search, if paused
finalize - Stops the search, and provides intermediate results to the /results endpoint
cancel - Stops the current search and deletes the result cache

Response Status

200 - Action was executed successfully
403 - Not authorized to execute action
404 - Search id was not found on server

Streaming

/services/streams/search

Provides synchronous event search streaming service

GET

Executes a simple search, i.e. no pipe support

Query Arguments

q - The simple search string (no leading 'search' operator) to execute

Response Status

200 - Method executed successfully

Response Body

// The return content is raw event text in streaming format.  
// There is no formatting, or timestamping on the data.  
// Close the client connection to stop the search.

/services/streams/livetail

Provides synchronous data input tailing service

GET

Streams raw data being received by Splunk

Query Arguments

q - The simple search string (no leading 'search' operator) to apply to the incoming data stream

Response Status

200 - Method executed successfully

Response Body

// The return content is raw event text in streaming format.  
// There is no formatting, or timestamping on the data.  
// Close the client connection to stop the search.

Configuration files

How do configuration files work?

Most of Splunk's advanced configurations are affected via configuration files. A bundle directory is a directory of one or more configuration files. The majority of Splunk's functionality can be set up through configuration files in any number of bundle directories. Learn more about bundle configuration, including best practices.

All bundles are housed in $SPLUNK_HOME/etc/bundles/. Bundles can be separated based on functionality, or collapsed into a single subdirectory containing each user-crafted configuration file. Bundles make packaging customizations easy. Once you have created a working bundle for a single Splunk server, you can then distribute it to target servers through the Splunk deployment server or share them with others through SplunkBase.

Configuration file directory structure

Configuration file directories are placed into a Splunk installation as subdirectories of $SPLUNK_HOME/etc/bundles/. Your Splunk Server ships with several such directories, including:

• default - contains the pre-configured configuration files. Do not modify the files in default.

Note: Not all configuration files appear in default/.

• local - stores modifications you make through the web interface or command line. You can make file edits here, or in a custom bundle directory.

Note: If you edit files that are also written to by Splunk Web, your edits may be overridden if someone else is editing Splunk Web at the same time.

• learned - this set of configurations are settings created by the Splunk Server as it trains on incoming data.
• readme - this directory contains example and spec configuration files that can help you create your own configuration files. For each configuration file, there are two reference files; .spec and .example. For example, inputs.conf.spec and inputs.conf.example. The .spec file is a specification of syntax, including which attributes and variables are available. The .example files are helpful examples of real-world usage. These files are all found in the $SPLUNK_HOME/etc/bundles/README directory.

Configuration file precedence

Configuration files live in multiple places: default, local and any configuration file directories you create. It is possible for an attribute to exist in more than one file. The evaluation order of configuration files is:

• local
• user-created directories
  • these are evaluated in alphabetical order
• default

Precedence is applied attribute-by-attribute. That is, if the file props.conf exists in local and a user created configuration file directory, the props.conf file in local does not override or replace the entire props.conf file. If the same attribute/specification exists in both the local props.conf and the user-created props.conf, the local props.conf overrides the attribute.

For example, if $SPLUNK_HOME/etc/bundles/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = t2rss-error

And $SPLUNK_HOME/etc/bundles/t2rss_bundle/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

Both the sourcetype assignment in local and the line merging attributes in t2rss_bundle apply. However, if both local and t2rss_bundle had a sourcetype assignment for source::/opt/Locke/Logs/error*, the assignment in local overrides t2rss_bundle.

Precedence rules for events with multiple attribute assignments

Beyond the rules discussed above for precedence, there is an additional precedence issue that affects the props.conf file alone. props.conf sets attributes for processing individual events by host, source or sourcetype (and sometimes eventtype). So it's possible for one event to have the same attribute set differently for its host, source or sourcetype. The precedence order is:

• source
• host
• sourcetype

Create Regular Expressions

Use these parameters to create regular expressions that can be invoked by properties to match patterns in data and metadata.

Splunk uses the PCRE library for Perl Compatible Regular Expressions. For additional information on building your own regular expressions, see the perlre documentation as well as the PCRE man pages for differences between PCRE and Perl 5. (Search for "PCRECOMPAT(3)".)

Filename

• transforms.conf
  • within any subdirectory of $SPLUNK_HOME/etc/bundles/

Format

[<regex name>]
attribute1 = val1
attribute2 = val2
...

Attributes

If the same <spec> is found in two bundle directories, the following precedence rules apply:

• Attributes in the "local" bundle are read first.
• Attributes in other bundles except "default" are loaded in alphabetical order of bundle name.
• Attributes in the "default" bundle are read last.

Overriding is performed attribute by attribute.

SOURCE_KEY = <string> (_raw)

Specifies which key in the event should be the source of the string on which to perform the regex.

LOOKAHEAD = <integer> (256)

Specifies how far into the string to match.

REGEX = <regular expression> (Empty)

If set, this is the regular expression that will operate on the data.

FORMAT = <string> ($1)

A string with arguments specified by $n, where n represents the output of the n-th parenthesized grop of the REGEX. If the regex does not have n groups, the matching fails. The special identifier $0 represents what was in the DEST_KEY before this regex was performed.

DEFAULT_VALUE = <string> (Empty)

If set, and the regex fails, this value will be written to the DEST_KEY.

DEST_KEY = <string> (<regex name>)

Specifies where the formatted result of running the regex should be stored. Any key prefixed by '_' will not be stored when it reaches the index.

REPEAT_MATCH = <bool> (False)

Specifies whether the regular expression should be run several times on the SOURCE_KEY. If set, it will start wherever the last match stopped, and will continue until no more matches are found.

WRITE_META = <string> (False)

Specifies that whatever was written to DEST_KEY should also be appended to the "meta data" for the event.

Splunk Reserved Keys

As data is streamed through Splunk pipelines, it lives in a set of keys. Some keys are indexed and/or stored and others are neither.

You can create new keys, any you create without _ prefixes are indexed in entirety (no major/minor segmentation) but not stored. Any key prefixed by '_' will not be stored when it reaches the index.

_raw

The raw text of the event. It is indexed with major/minor segmentation. It is intended to preserve the integrity of original content. You can change _raw with regexes, but you will destroy data integrity.

_meta

The metadata for the event. It is indexed with space delimited segmentation.

_time

The timestamp of the event, in seconds since 1/1/1970 UTC. It is in the primary index but not stored.

queue

The queue to be used by the next queue output processor. It is neither indexed nor stored.

cluster

Used to create meta-events. It is indexed in entirety but not stored.

MetaData:Host

The host associated with the event. The value must be prefixed by "host::" It is indexed in entirety but not stored.

_MetaData:Index

The index where the event should be stored. It is neither indexed nor stored.

MetaData:Source

The source associated with the event. The value must be prefixed by "source::" It is indexed in entirety but not stored.

MetaData:Sourcetype

The sourcetype of the event. The value must be prefixed by "sourcetype::" It is indexed in entirety but not stored.

Examples

This creates a meta event for each IP address via regex.

[access-ip]
DEST_KEY      = cluster
REGEX         = (\d+\.\d+\.\d+\.\d+)
DEFAULT_VALUE = 
PREPEND       = ip::
WRITE_META    = False

To make use of the meta events, you can Splunk for anything and add the modifer index::metaevents. Or, if you search for two values that are in separate events but in the same meta event (because both events contain the same IP value), the meta event will appear in your results as if it were a real log event.

Add & Override Properties

These parameters add or reconfigure processing properties inside the Splunk server. Properties are exposed by C++ code within individual processors.

Filename

• props.conf (within any subdirectory of $SPLUNK_HOME/etc/bundles/)

Format

[<spec>]
attribute1 = val1
attribute2 = val2
...

Spec

<sourcetype>

the sourcetype of an event

host::<host>

where <host> is the host for an event

reportinghost::<host>

where <host> is the reporting host for an event

source::<source>

where <source> is the source for an event

Attributes

If the same <spec> is found in two bundle directories, the following precedence rules apply.

• Attributes in the "local" bundle are read first.
• Attributes in other bundles except "default" are loaded in alphabetical order of bundle name.
• Attributes in the "default" bundle are read last.

Overriding is performed attribute by attribute.

Linemerging

SHOULD_LINEMERGE = <bool> (True)

When set to true Splunk can combine several input lines into a single event, based on the following configuration attributes.

These are used only when SHOULD_LINEMERGE = True

AUTO_LINEMERGE = <bool> (True)

Directs Splunk to use automatic learning methods to determine where to break a stream of lines into events.

MAX_EVENTS = <integer> (256)

Specifies the maximum number of input lines that will be added to any event. Splunk will break after the specified number of lines are read.

The following two conditions are checked in the order below. If the first matches, the second is not tested.

MUST_BREAK_AFTER = <regular expression> (Empty)

When set, and the regular expression matches the current line, Splunk is guaranteed to create a new event for the next input line. Splunk may still break before the current line if another rule matches.

MUST_NOT_BREAK_AFTER = <regular expression> (Empty)

When set and the current line matches the regular expression, Splunk will not break on any subsequent lines until the MUST_BREAK_AFTER expression matches.

The following four conditions are checked in the order below. If one matches, then the remainder are not tested.

BREAK_ONLY_BEFORE = <regular expression> (Empty)

When set, Splunk will create a new event if and only if it encounters a new line that matches the regular expression.

BREAK_ONLY_BEFORE_DATE = <bool> False

When set, Splunk will create a new event if and only if it encounters a new line with a date.

BREAK_BEFORE_DATE = <bool> (True)

When set to true, Splunk will create a new event when it encounters a new line with a timestamp. It will also emit an event containing all lines since the last new event was created. Note that Splunk will still create a new event unless there is a rule that says not to break or the automatic learning method determines that splunk should not break.

MUST_BREAK_BEFORE = <regular expression> (^\s*$)

Note Deprecated. When set, Splunk will create a new event when it encounters a new line that matches the regular expression. This is similar to BREAK_BEFORE_DATE.

Timestamp extraction configuration

DATETIME_CONFIG = <filename relative to SPLUNK_HOME> (/etc/datetime.xml)

Specifies the file to configure the timestamp extractor. This configuration may also be set to "NONE" to prevent the timestamp extractor from running or "CURRENT" to assign the current system time to each event.

MAX_TIMESTAMP_LOOKAHEAD = <integer> (150)

Specifies how far into an event splunk should look for a timestamp.

Typing configuration

TYPING_CONFIG = <filename relative to SPLUNK_HOME>

(/etc/event-types/current/default.xml)

Specifies the file to configure the event typer. This configuration may also be set to "NONE" to prevent the event typer from running.

AUTO_TAG = <bool> (False)

Specifies whether to automatically tag new event types with important keywords from events of that event type.

Class configuration

TRANSFORMS-<class> = <"regex name","regex name",...>

see transforms.conf.spec

Define regular expressions in transforms.conf files. Splunk configures classes of regular expressions for each event.

For each class, Splunk takes the configuration from the highest precedence configuration block. This means that if a particular class is specified for a source, it will override the same class if it is specified for a sourcetype. Similarly, if a particular class is specified in the local bundle for a sourcetype, it will override that class for the default bundle for that sourcetype.

Class names ("-annotation") must be unique within a Splunk instance, so the new configuration does not override an existing configuration. Always add a class name rather than use "TRANSFORMS" alone to avoid replacing Splunk default behavior.

The following is an example TRANSFORMS class in the default bundle for all sourcetypes:

TRANSFORMS-annotation = filetype,loglevel,os,browser,language,ip,email,url

This configuration uses the "test-pipeline" regex (defined in transforms.conf) for sourcetype "access_log":

[access_log]
TRANSFORMS-test = test-pipeline

Source Type configuration

sourcetype = <string> (Empty)

If set for a [source::...] block, it will cause that source to be assigned the specified sourcetype.

Examples

If host matches nyc* then set Eastern Time Zone.

[host::nyc.*]
TZ = EST-5EDT01:00:00,M4.1.0/02:00:00,M10.5.0/02:00:00

If the sourcetype is apache_error, linemerge source data into multi-line events.

[apache_error]
SHOULD_LINEMERGE = True

If the source matches, turn on auto-tagging. The tags will be based on useful-looking words in events during typing process.

[source::.*datatotag.*]
AUTO_TAG = 1

Use the shipped syslog-regex in regex-props.conf to extract meta events.

[source::.*sysloglikesource.*]
REGEXES = syslog-regex

Use the regex access-ip in overlay-regex-props.conf to extract meta-events.

[source::.*access.*]
REGEXES = access-ip

Add Input Configurations

These parameters add or reconfigure input sources from which the Splunk server will access and index data.

Filename

• inputs.conf (within any subdirectory of $SPLUNK_HOME/etc/bundles/)

Format

[:]
attribute1 = val1
attribute2 = val2
...

Any attributes not specified are set to the values for the [default] entry.

Input Types

Tail File

This directs Splunk to use the tail file input module to watch all files in the directory.

Batch File

Same as tail file, except Splunk uses the batch file input module.

FIFO

This directs splunk to read from the fifo at the specified path.

TCP

This configures Splunk to listen on the specified port. If a connection is made from , this stanza is used to configure the input. If is blank, this stanza matches all connections on the specified port.

UDP

Similar to TCP, except that it listens on a UDP port.

Distributed

This is the same as TCP, except the remote server is assumed to be a Splunk server. For splunktcp, the host or connection_host will be used if the remote Splunk server does not set a host, or if the host is set to host::localhost.

Attributes

Every input type has these possible options:

• host =

This is a shortcut for MetaData:Host =. It sets the host of events from this input to be the specified string. "host::" is automatically prepended to the value when this shortcut is used.

• index =

This is a shortcut for _MetaData:Index = . It sets the index where events from this input will be stored.

• source =

This is a shortcut for MetaData:Source = . It sets the source name of events from this input to be the specified string. "source::" is automatically prepended to the value when this shortcut is used.

• sourcetype =

This is a shortcut for MetaData:Sourcetype = . It sets the source type name of events from this input to be the specified string. "sourcetype::" is automatically prepended to the value when this shortcut is used.

• reportinghost =

This is a shortcut for Reportinghost = . It sets the reporting host name of events from this input to be the specified string. "reportinghost::" is automatically prepended to the value when this shortcut is used.

• =

This generically associates with for all events from this input.

Additional Attributes

Each input type has additional attributes specific to the type:

Tail

• host_regex =

If specified, the tail file input module will use the specified regualar expression to extract the host from the filename of each input. Specifically the first group of the regex is used as the host. If the regex fails to match, the "host =" attribute is used as the host.

• host_segment =

If specified, the tail file input module will use the specified '/' separated segment of the path as the host of each input. If the value is not an integer, or is less than 1, the "host =" attribute is used as the host.

• _whitelist =

If specified, files from this path will be tailed only if they match the specified regular expression.

• _blacklist =

If specified, files from this path will not be tailed if they match the specified regular expression. If a file is inadvertently specified for both _whitelist and _blacklist, _blacklist prevails and the file will not be tailed.

Batch

• move_policy = {passive_symlink, passive_copy, sinkhole} (passive_symlink)

This specifies the policy to be used to handle the files. The "sinkhole" policy will delete the files as they are read, while the other two methods will link or copy the files into a separate directory.

• host_regex (see Tail)

• host_segment (see Tail)

Batch input ignores these parameters:

• source

UDP

• connection_host = {ip,dns,} (ip)

The UDP input processor will rewrite the host with the ip address of the remote server if "ip" is set, the DNS name of the remote server if "dns" is set, and will do nothing to the host for any other string.

Distributed

• queue = (parsingQueue)

This specifies where the TCP input processor should deposit the events that it reads.

Examples

The following are example UDP input configurations. To use one of these configurations, copy the configuration block into inputs.conf in $SPLUNK_HOME/etc/bundles/local/inputs.conf

This configuration directs the server to listen on UDP port 514 for raw data from ANY remote server. The "host" of the data will be set as the IP address of the remote server.

[udp::514]

This configuration directs the server to listen on UDP port 9995 for raw data from ANY remote server. The "host" of the data will be set as the host name of the remote server. All data will also be assigned the sourcetype "log4j" and the source "UDP:9995"

[udp::9995]
connection_host = dns
sourcetype = log4j
source = udp:9995

This configuration directs the server to listen on UDP port 9994 for raw data from 10.1.1.10. All data will be assigned the
host "webhead-1", the sourcetype "access_common" and the the source "10.1.1.10/var/log/apache/access.log"

[udp:10.1.1.10:9994]
host = webhead-1
sourcetype = access_common
source = 10.1.1.10/var/log/apache/access.log

Recognize Source Types

Create new source types to describe your data input sources.

Filename

• props.conf (within any subdirectory of $SPLUNK_HOME/etc/bundles/)

Format

[source::<path>]
sourcetype=mySourceType
...

Create a new configuration with the path to your file (or files) and the name you wish to assign as a sourcetype.

Examples

[source::/var/log/netinfo.log]
sourcetype=netinfo_log

[source::/var/log/httpd/mywebsite_*_log]
sourcetype=mywebsite_log

Splunk Web appearance

How Splunk Uses Skins

The default Splunk Web interface is defined by the HTML, CSS, JavaScript and XSL found under the $SPLUNK_HOME/share/splunk/search directory. Each skin has a CSS file that overrides the default styles for text color, background color, background image, and some other associated

Skins css Files

$SPLUNK_HOME/share/splunk/search_oxiclean/static/css/skins

The .css files in this directory are where most customizations are done. If you put a file in this directory, it will appear in the Themes menu in Splunk Web. Since any file is assumed to be a skin, don't forget to remove temporary files created by your text editor. The standard skins provided are basic.css, the default theme, black.css, the all-black "inverted" or "night mode" theme, and desert.css, the neutral brown-toned theme. You can use any of these files as a base to design your own themes.

Images/skins directory

$SPLUNK_HOME/share/splunk/search_oxiclean/images/skins

This is the standard location for image files, one directory for each skin. You can use the default images in your new skins by specifying their path from $SPLUNK_HOME/share/splunk/search_oxiclean (like background-image:url(/images/skins/basic/menu_arrow_bg.gif.) Put any new images in a new directory to avoid confusing them with the default ones.

Add or remove themes

Copy your new theme's .css file to the $SPLUNK_HOME/share/splunk/search/static/css/skins directory (or create a new one) to add a new theme. To remove an existing theme, delete its .css file. Restart the server to see your changes.

Create a new skin

The easiest way to create a new skin is to copy an existing one to a new .css file and edit the items you wish to change. The standard skin files contain comments to help you identify which items you are interested in, but expect to make many changes scattered throughout the file due to to the complexity of interface elements.

Edit the new file, restart the server, and select your theme from the Preferences menu.

Note: You can also restart Splunk Web only with this command:

# splunk restart splunkweb

Splunk does not validate the skins files. In general, an invalid (or empty) css file will still leave the interface usable, but may not appear as expected.

Examples

This example shows how to change the standard popup menus from white to blue.

The menus have two basic colors, one for the background and one for the highlight. In reality, there are multiple elements that need to have the correct style applied to make it all look like a single color menu. And there are sometimes side-effects to watch out for.

For this example, we will use a medium blue (#6699ff) for the background and a light blue (#99ccff) for the highlight.

Copy basic.css to a new file, blue.css.

First, change the menu background color:

/* POPUP MENUS */
/* basic styles for submenu arrow-icons and highlight/opened states */
.popupMenu ul{
        background-color:#6699ff;  /* was #fff */
}
.popupMenu li.secondary label {
        background-color:#6699ff;  /* was #fff */
        background-image:url(/images/skins/basic/menu_arrow_bg.gif);
        background-position:right;
        background-repeat:no-repeat;
}

Next, change the menu item border to match:

/*---------------------*/
/* border color styles */
/*---------------------*/
.popupMenu ul label {
        border:1px solid #6699ff;  /* was #fff */
}

The check mark is an image, and the one that basic.css (menu_checkbox_bg.gif) uses has a white border. If you prefer something else, you can create a new image or use a different one such as menu_checkbox_boxed_bg.gif:

/* checkboxes and radio buttons in menu options */
.softWrap #softWrap,
.helpActive #helpActive,
.exploded #exploded,
#liteMode {
        background-image:url(/images/skins/basic/menu_checkbox_boxed_bg.gif); /* was menu_checkbox_bg.gif */
        background-repeat:no-repeat;
}

Now the highlight is where it gets complicated.

In basic.css, there are two elements using the same style:

.popupMenu li.secondary label.explicitMouseOver,
.popupMenu li.open label {
        background-color:#fffad1;
}

An open menu item is .popupMenu li.open label, but we don't want to also change .popupMenu li.secondary label.explicitMouseOver. So change these lines so each element has its own style. Then we can apply the new style only the one we are interested in:

.popupMenu li.secondary label.explicitMouseOver,
{
        background-color:#fffad1;
}
.popupMenu li.open label {
        background-color:#99ccff; /* was #fffad1 */
}

And again for the divider between sections of the menu:

.popupMenu ul, 
/* .popupMenu ul li,  handle this below */
.typeAhead,
div.histogramAndTabs,
div#tabs,
div.subsections,
div#topBar .selected,
input.disabled, select.disabled {
    border-color:#ccc;
}

.popupMenu ul li {
        border-color:#99ccff; /* was #ccc */
}

Similarly, the highlight of the selected item shares a style with another element:

label.explicitMouseOver,
.typeAhead label.explicitMouseOver {
        background-color:#fffad1 !important;
}

We can separate the two as we did above, but there is a side-effect to consider. label.explicitMouseOver is also used for other lists, such as the Source Types and Sources lists on the Search page, as well as highlighting segments in a list of events. You can change it, but be aware that it will impact more than just the menu.

If you still want light blue, change only label.explicitMouseOver:

label.explicitMouseOver  {
        background-color:#99ccff !important; /* was #fffad1 */
}

.typeAhead label.explicitMouseOver {
        background-color:#fffad1 !important;
}