This document last updated: 06/06/07 03:06pm

Print Developer Manual

Start Here

Splunk Architecture

Splunk is a high performance, scalable software server written in C/C++ and Python. It indexes and searches logs and other IT data in real time. Splunk works with data generated by any application, server or device. The Splunk Developer API is accessible via SOAP or the command line. After downloading, installing and starting Splunk, you'll find two Splunk Server processes running on your host, splunkd and splunkweb.

1

How-To FAQ

Basic directions for the most commonly requested Splunk modifications that require developer work, rather than reconfiguration.

Rebrand the Splunk Web Interface

Use skins to modify Splunk's appearance with CSS.

Customize the Splunk Web Interface's Layout and Behavior

Use skins to modify Splunk's appearance and behavior with CSS.

Launch Splunk from Your Interface

Use permalinks to open Splunk and run specific searches encoded into each permalink's URL.

Search with Splunk from another application

Choose your API: command line (CLI) or SOAP.

Manage Splunk setting and users from another application

You can control Splunk through two APIs: command line (CLI) or SOAP.

Send Live Splunk alerts to another application

Set up an alert script - a shell script called with contextual parameters whenever a Live Splunk meets its alert threshold.

Preconfigure, customize or tune specific data inputs

Create a bundle with a shrinkwrapped input configuration and/or any properties, regular expressions, or Saved Splunks customized for your data.

Preload Saved and Live Splunks

Create a bundle with the Saved and Live Splunks you wish to install on your Splunk Server instances.

Add processing not available through configuration alone

Create a module that adds one or more new processors to your Splunk Server instances.

Access data not supported by Splunk's built-in input modules

Create a module that includes an input processor to access your data, and then passes it along to Splunk's universal pipeline or a custom pipeline of your own.

Command Line Interface

How Splunk Uses the CLI

Splunk's command line interface provides most of the functionality of the splunkweb GUI, minus a few graphically-oriented features such as plotting Events by Time. It adds several administrative controls, such as starting and stopping the server. Most commands use the same SOAP API calls used by splunkweb. Additional commands work directly through the local Unix operating system.

Most operations take the form:

# splunk [action] [context] [default argument] -parameter

Each command follows one of the following existing sets of actions:

Examples

Below are a few examples of typical commands. Each How-to section includes more specific examples. A formal and complete listing of Splunk command line syntax is in the TK.

Search

# splunk search hoursago::1 sourcetype::linux_messages_syslog NOT success

Login

# splunk login melissa:s33kr1t

Data Input

# splunk add tail /var/log/ -liveonly true

Users

# splunk add user -username algore -full-name "Al Gore" -password pa55word -auth admin:changeme

Splunk-2-Splunk

# splunk enable receive

Summary

# splunk summary sources -sort most-events

Authentication

Splunk Professional requires all commands to be authenticated. The API supports both per-command and per-session authentication.

Command Authentication

Individual commands can be authenticated by including an authentication username and password. One parameter can specify both user and password. If either the username or password aren't specified, the command line prompts the user.

Parameters

Examples

splunk search meta::all -auth admin:changeme
splunk search meta::all -auth admin

Session Authentication

Alternatively, users can use the "login" command to cache their auth credentials on the local filesystem securely. Auth tokens persist until a logout command is issued or the Splunk server is restarted.

Examples

splunk login
splunk logout

Search

The command-line search API supports the exact same syntax as the Splunk box, with additonal parameters.

Actions

Default Argument

Parameters

where range is n items returned from the full results. Example:

splunk search 404 -get sources::0-9

returns the first 10 sources from the specified search.

Example

splunk search -get hosts "smtp NOT success hoursago::1"

By default only 100 events are returned when a search is done from the CLI. This can be changed by adding maxresults:: to your search. For large searches, we recommend you use the "raw" output type to reduce memory usage.

splunk search -output rawevents "meta::all minutesago::120 maxresults::100000 maxtime::300"

User Accounts

Splunk Professional user accounts can be created and maintained through the CLI.

Actions

Default Argument

Parameters

Examples

splunk add user -username algore -full-name "Al Gore" -password pa55word
splunk edit user algore -role user

Saved and Live Splunks (TBD)

This section of the API is still being finalized. [15 Aug 2006]

Data Inputs

Splunk's access to event data from files, FIFOs, network ports, and ODBC databases can be configured and controlled from the CLI, including its methods for setting meta data - host, source, and sourcetype.

Actions

Contexts

Default Argument

Parameters

Add

Edit

Remove

List

Examples 

splunk add tail /var/log/myapp/ -sourcetype myApp
splunk add batch /var/logarchive/ -host-method segment:2
splunk add udp 514
splunk edit odbc mysqlserver:stats -sourcetype mynewtype
splunk remove odbc mysqltable
splunk remove udp 514

Settings

These actions control the Splunk Server's own configuration - what port its Web interface runs on, how much disk space it leaves free, etc.

Actions

Contexts

Default Argument

Parameters

Examples



splunk enable broadcast

splunk edit -mgmt-port 8090
SP-CAAAJ7R

Splunk-2-Splunk

These commands control the forwarding and receiving of data shared among Splunk Server instances. They also control distributed search configurations. You must use a Splunk Professional instance at the receiving end of a Splunk-2-Splunk connection.

Actions

Contexts

Default Argument

Parameters

Examples

splunk add forward 10.1.1.123:9045
splunk enable receive
splunk set dist-search -port 7890

Summaries

Summary info on inputs and events indexed, as seen on the Splunk Server's home page.

Actions

Contexts

Default Argument

Parameters

Examples



splunk summary totalevents

splunk summary sources -sort most-events
SP-CAAAJ7R

Controls

Commands for sysadmins to start and stop Splunk Server processes, and to check the syntax of configuration files for errors.

Actions

Contexts

Default Argument

Parameters

Examples

splunk start

splunk start --nodaemon



splunk restart splunks
ipaq sell | SP-CAAAAJQ

splunk stop splunkweb

SOAP API

How Splunk uses SOAP

Splunk's SOAP API is how the splunkweb process communicates with splunkd process. Anything you can do through Splunk's Web interface, you can do through SOAP. You can make calls directly by building complete SOAP messages via the front-end of your choice, or use the Python Control Layer interface.

Python Control Layer

The Python Control Layer (PCL) allows you to send SOAP requests from your Python code without having to generate the full call yourself. You can still use SOAP directly, but many operations are available in the PCL and more will be added in the future. All of the data input methods in the PCL take a (more or less) standardized Python dictionary object and, in most cases, return a dictionary as well.

As for the methods themselves, they follow the convention:

def <module>_<operation>(args, fromCLI)

where <module> is the input module (e.g. "tail" or "fifo") and <operation> is the operation that is to be performed with the module (e.g. "addFile" or "list"). args is a Python dictionary object. fromCLI is a boolean variable that is set to True if the method is called from the command line tool, but is otherwise defaulted to False for calls from other consumers, such as the search interface.

The PCL Data Input Methods can be found in the control_api.py module, located in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib. To use this module, import the following into your Python code:

control_api.py
cli_common.py
control_exceptions.py

control_api.py defines most of the interface, cli_common.py contains authorization functions and control_exceptions.py contains the exceptions used by the PCL. This directory also contains many additional source files that actually implement the interface. All the Python code that Splunk uses is in these files for you to examine. As more functionality is folded into the PCL, methods will be appropriately exposed through control_api.py.

You can find the list of supported commands in control_api.py (search for "cList" around line 720.) This file defines the interface exactly as Splunk uses it, so this is always the most accurate source for information. The list looks like this:

cList.addCmd(SplunkCmd("add",    "user",             user_add, "username" , proReq = True))
cList.addCmd(SplunkCmd("add",    "fifo",             fifo_add, "source"))

The argument "proReq = True" indicates that this function is only available for a Splunk Professional server. Any request to a Pro server must be authenticated by passing a valid authentication token with the request. Additional details can be found by looking at the function definitions elsewhere in the file or the implementations in the *.py files.

All methods take a dictionary (set of key-value pairs). Methods usually have one or more required key-value pairs and zero or more optional key-value pairs. All keys are required to be in lower-case. There is no case restriction on the values. If a module is unable to find a required key-value pair in the incoming dictionary, an exception will be raised. If the method finds a key-value pair that is not in its set of required or optional key-values, an exception will be raised.

On error the methods will raise exceptions, so calls to the PCL should be wrapped in try/except blocks for error handling purposes.
Most methods will return a dictionary on completion. In most cases, an empty dictionary on return signifies a successful call. In certain cases, the dictionary will contain a "serverRestartReq" key-value signifying that splunkd must be restarted for changes to take effect.

Environment

Splunk requires a number of environment variables to be set, such as SPLUNK_HOME, and the correct PATH to be able to locate splunk files. Source the file setSplunkEnv in the splunk bin directory (usually /opt/bin/splunk) to set up the necessary environment. As long as the environment is correct, you can run your Python code from anywhere on your filesystem.

source /opt/splunk/bin/setSplunkEnv

Authentication

Every call requires an authentication token be passed in the dictionary, even for the free server which does not support users or authentication. In this situation, use a dictionary with an empty string for the authstr.

All requests to a Splunk Professional server must be authenticated, otherwise they will fail. To do this, first call getAuthInfo with a valid username and password. The server responds with an authentication token to be used for subsequent requests. This token is good until you submit a logout request or the next time splunkd is restarted.

Example

This is a minimal example of using the PCL to generate a SOAP call. It has no authentication or error handling but shows the basic structure of a PCL call (in this case to a free server.) It generates a SOAP request for function tail_list and prints the returned dictionary containing a list of all the tails currently configured in the server. Note that it calls the version of Python supplied with Splunk and not any that may be installed on the system. Use Splunk's version of Python to avoid problems with incompatible versions.

#!/opt/splunk/bin/python -u

import splunk.clilib.control_api as ca

argsDict = {
  "authstr" : ""
}

results = ca.tail_list(argsDict)
print results

The output looks like this:

{'dirs': ['/var/log/httpd'], 'dynamic': ['/var/log/httpd/error_log', '/var/log/httpd/access_log'], 'static': ['/var/log/system.log']}

For this minimal configuration, you can see there is one directory (/var/log/httpd) that contains two files (error_log and access_log) and one additional file specified by name (system.log.)

See the following page for more complete examples of making PCL calls from your Python code.

Using the PCL to Make SOAP Requests

Authentication Example

Below is a more complete example of the list_tails function. It authenticates against a Splunk Pro server, which could be elsewhere on the network, passes the authentication token with the list_tails request, and prints the contents of the returned dictionary.

#!/opt/splunk/bin/python -u

# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm

# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions

# for exit()
import sys

# optionally connect to another server at hostname:port
# 8089 is the default Splunk SOAP port
# comm.setURI("https://testserver:8089")

try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")

except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)

# make a dictionary with the auth token
argsDict = {
  "authstr" : authString
}

# PCL call, returns list of tails
results = ca.tail_list(argsDict)

# output the results
for key, values in results.items():
  print "%s: %s" % (key, values)

Run against a Splunk Pro server, the output looks like this:

dirs: ['/var/log/httpd']
dynamic: ['/var/log/httpd/error_log', '/var/log/httpd/access_log']
static: ['/var/log/system.log']

If this example is run against a free server, an exception is raised because it can't handle the call to getAuthInfo.

Traceback (most recent call last):
  File "./listTails.py", line 18, in ?
    authString = comm.getAuthInfo("admin", "changeme")
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 135, in getAuthInfo
    retStr = callAPI(block)
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 180, in callAPI
    raise InvokeAPI, str(root.documentElement.firstChild.data)
splunk.clilib.control_exceptions.InvokeAPI: 'badparams'

The message "badparams" is a generic error. In this case it indicates that the requested operation, authentication against a free server, does not exist.

Search Example

Here is an example of a search. The key "terms" is the same as what you would type in the Splunk Box to do a search. For more on what you can specify here, see the Language Reference. 

#!/opt/splunk/bin/python -u

# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm

# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions

# for exit()
import sys

try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")

except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)

# make a dictionary with the auth token and the search string
argsDict = {
  "authstr" : authString,
  "terms"   : "404"
}

results = ca.search(argsDict)
print results

The output when run against the same minimal Splunk Pro server is this:

127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302

To get the full event, including metadata, specify you want XML by adding "format" to the dictionary:

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "format"  : "xml"
}

Now the result is this:

<queryResult><ids>
</ids>
<results type="events"> <result cd="0:2891"><segtext xml:space="preserve">127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302<meta></meta></segtext>               <timestamp>1170872913</timestamp>               <source cd="2">/var/log/httpd/access_log</source>
                <host cd="1" name="test.splunk.com" >                 <tags></tags>
</host>         <sourcetype cd="2" base="too_small">too_small</sourcetype>
                <type cd="9" wob=" v:226e b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

For more about the search function, you can look at the Python code in the file searchizzle.py. The function definition begins like this:

def search(args, fromCLI):
  paramsReq = ("terms", "authstr")
  paramsOpt = ("output", "format", "get")

which shows the required and optional parameters. "output" is the output type, the default is "splunkui" (and this is almost always the correct choice.) "format" was mentioned above, and "get" is the type of data to return.

The default for get is "events", to return matching events. You could also specify "hosts" to return a list of hosts with matching events and so on. The list of possible values is defined in validGetTypes a few lines below:

  validGetTypes     = ("events", "hosts", "sources", "sourcetypes", "types")

Additional PCL Search Examples

The search function has a number of optional parameters for determining what results are returned. In addition to the standard event results discussed in the previous example, here are some other ways you can use the PCL search function.

output

For a short report of selected matching event statistics, use "output":

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "output"  : "scheduler"
}

If a regular search returns these events:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /test.html HTTP/1.1" 404 305
127.0.0.1 - - [27/Feb/2007:14:00:50 -0800] "GET /favicon.ico HTTP/1.1" 404 307
127.0.0.1 - - [26/Feb/2007:16:16:35 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:40:27 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:37:04 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:29:29 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:27:07 -0800] "GET /favicon.ico HTTP/1.1" 404 302

Then specifying an output type of "scheduler" returns this:

eventCount:          7
hostCount:           1
sourceCount:         1
typeCount:           2
sourceTypeCount:     1
eventTagCount:       0
hostTagCount:        0
starttime:           12/31/1969:16:00:00
endtime:             03/06/2009:12:46:17

get

Use "get" to return a list of hosts, sources, sourcetypes or types that contain matching events. The dictionary shown below returns a list of the event types of the matching events.

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "get"     : "types"
}

returns

eventtype::?9
eventtype::?11

With any of these, you can also specify a format of "xml" to get the output in XML with additional information:

<queryResult><ids>
</ids>
<results type="types">  <result c="5" cd="9"><segtext xml:space="preserve">5 results from eventtype::?9<meta></meta></segtext>          <type cd="9" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:0  ">
                        <tags></tags>
</type>
</result>       <result c="2" cd="11"><segtext xml:space="preserve">2 results from eventtype::?11<meta></meta></segtext>                <type cd="11" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

Creating SOAP Requests

Observing PCL SOAP Calls

You can see exactly how Splunk creates SOAP calls with a small change to the existing PCL code to output the data sent and received. Use these examples to generate your own SOAP messages from the front-end of your choice and parse the data returned from splunkd for your application.

1. Open $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib/cli_common.py
2. Find the following block in the callAPI function:

try:
    retStr = server.invokeAPI(apiData=data)
  except socket.error:
    raise SOAPConnectionException, "Could not connect to splunk server.  
Please ensure that splunkd is running."

3. After the try: line, add (with 4 leading spaces):

    print "BEGIN SOAP CALL:\n%s\nEND SOAP CALL" % data

4. After the SOAPConnectionException line, add (with 2 leading spaces):

  print "BEGIN RETURNED XML:\n%s\nEND RETURNED XML" % retStr

5. Save the file
6. Run a few commands from the GUI to see what is being sent. The results can be found in $SPLUNK_HOME/var/log/splunk/web_access.log. Remember to remove the added code later to avoid filling this file wtih SOAP calls.

Creating a SOAP Call

Here is an example of search via SOAP:

1. Make a request to https://localhost:8089/ (replace with your hostname:port) to generate an authentication token (valid for as long as you leave splunkd running, or until you switch authentication mechanisms):

<call name="userLogin"><params><login><username>admin</username><password>changeme</password></login></params></call>

This returns an authentication token, which you must include in your search call:

<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>

2. Now run the search call, with a few important notes:

<call name="executeQuery"><params><query>SEARCH  meta::all GET events::0-2 
OUTPUT splunkui::2.1 
format::raw</query><user>livesplunkuser</user><queryId>37627661733490256838</queryId>
<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>
</params></call>

This results in the following xml (3 results, one in each <result> block),
which you can parse as you wish:

<queryResult>
<ids>
</ids>
<results type="events"><result cd="0:873197"><segtext xml:space="preserve">Feb 
19 14:25:49 VeeAte last message repeated 1964 times<meta><sg 
c="4003720337">format::l1_::_</sg> </meta></segtext>                
<timestamp>1171923949</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873110"><segtext xml:space="preserve">Feb 19 14:24:49 VeeAte 
last message repeated 1956 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923889</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873023"><segtext xml:space="preserve">Feb 19 14:23:49 VeeAte 
last message repeated 1951 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923829</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
</results><performance>Queryid=3263159498132591711719240037627661733490256838 
user=livesplunkuser result="success" results_returned=9774 
submitted=02/19/2007:14:28:38 time_between_submission_and_execution=0.000 
execution_time=0.020 total_time=0.020</performance>
<query>SEARCH meta::all GET events::0-2 OUTPUT splunkui::2.1 
format::raw</query>
</queryResult>

executeQuery.pl

Here is a perl script that executes SOAP calls:
Note This requires XML::XPath and SOAP::Lite

#!/usr/bin/perl

use XML::XPath;
use SOAP::Lite;

$| = 1;

# URL to Splunk Server Management Port
$splunkd = "https://defiant.splunk.com:8089";

# Login to splunk instance supplied by argument and return auth token.
sub Splunk_SOAP_login {
    my ($url) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);

    $apiData = "<call name=\"userLogin\"><params><login><username>admin</username><password>changeme</password></login></params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    $authToken = $s->invokeAPI($call) -> result;
    return $authToken;
}

## Execute the SOAP invoke api call and return the result
sub Splunk_SOAP_invokeAPI {
    my ($url, $authToken, $callname, $data) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);
    $apiData = "<call name=\"" . $callname . "\"><params>" . $authToken . $data . "</params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    return $s->invokeAPI($call) -> result;
}

# Helper function to build the data portion of a query call.
sub buildQuery {
    my ($searchString) = @_;
    $qdata = "<query>SEARCH $searchString GET events::0-9 OUTPUT maxlines::14 splunkui::2.1 summary::2.1 format::all "timeformat::%m/%d/%Y, %H:%M:%S"</query><user>admin</user><queryId>1234567890</queryId>"; 

    return $qdata;
}

# Obtain an auth token.
$authToken = Splunk_SOAP_login($splunkd);
print("authToken: $authToken\n");

# Build the API call.

# change this to send a different call
# some possible choices are:
#     getLicenseInfo
#     getUserInfo
#     getHostname
#     deleteUser
$callname = "executeQuery";

# for queries, this is the search term
$term = "error";

$query = buildQuery($term);

# Make the call and print results.
$results = Splunk_SOAP_invokeAPI($splunkd, $authToken, $callname, $query);
print $results;

SOAP Message Examples

Here are examples of a full SOAP message, including the XML, SOAP body and envelope.

Login

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 642
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <invokeAPI xmlns="urn:m2c-ManagementService">
            <apiData xsi:type="xsd:string"><call
            name="userLogin"><params><login><username>admin<
            /username><password>changeme</password></login><
        /    params></call></apiData>
        </invokeAPI>
    </soap:Body>
</soap:Envelope>

Execute Query

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 925
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <invokeAPI xmlns="urn:m2c-ManagementService">
            <apiData xsi:type="xsd:string"><call name="executeQuery"><params>
            <auth><userId>1</userId><username>admin</username>
            <authToken>3135052749</authToken></auth>
            <query>SEARCH error GET events::0-9 OUTPUT maxlines::14
            splunkui::2.1 summary::2.1 format::all &quot;timeformat::%m/%d/%Y,
            %H:%M:%S&quot;</query><user>admin</user><
            queryId>1234567890</queryId></params></call>
            </apiData>
        </invokeAPI>
    </soap:Body>
</soap:Envelope>

Bundles

How Bundles Work

Bundles add configuration to splunkd. A bundle is a small directory of files that contains one or more configuration files that together configure the Splunk Server for a specific site or standard environment. Bundles must be placed into a Splunk installations as subdirectories of the $SPLUNK_HOME/etc/bundles directory.

Startup

When splunkd starts, it walks through the subdirectories of the bundles directory in this order:

local
This subdirectory is where admins should add local configuration additions and changes. Local bundles override all other settings.
All other subdirectories execept learned and default
These subdirectories are presumed to be user-added bundles. They're loaded in alphabetical order.
default
These are the bundles shipped by Splunk.
learned
These are settings created by the Splunk Server as it trains on incoming data. Learned configurations take lowest priority after all human-specified settings.

Within each subdirectory, splunkd looks for and loads each of these files.

auth.conf
Authentication settings
inputs.conf
Data inputs - files, network ports, etc.
props.conf
Processing properties - time zones, breaking characters, etc.
regexes.conf
Regular expressions for use by the properties defined in props.conf.
livesplunks.conf, savedsplunks.conf
Live Splunks and their associated Saved Splunks

Typical Uses

Bundles can configure any part of splunkd that you can configure through the splunkweb GUI or command line, as well as more advanced processing parameters.

Authentication
Configuring your selected authentication scheme, such as LDAP
Data input
For example, to turn on a specific TCP port and assign a sourcetype to any data accessed from it.
Processing properties
Customized configurations for specific sources, source types, or hosts.
Regular expressions
Regexes for identifying event boundaries, setting meta data attributes, and performing transformations.
Saved and Live Splunks
Custom Splunks meant for searching a specific set of data, such as Weblogic logs.

Example files

The Splunk Server ships with several files that demonstrate how to create bundle configurations.

$SPLUNK_HOME/etc/bundles/

$SPLUNK_HOME/etc/bundles/local/

Bundles may use wide range of possible configuration options and the *.conf files themselves are very tolerant of syntax errors. This means splunkd cannot differentiate between your desired configuration and the many possible variations that, while technically correct, are not what you intended. Test your bundle configurations carefully to ensure the correct results.

Add & Override Properties

These parameters add or reconfigure processing properties inside the Splunk Server. Properties are exposed by C++ code within individual processors.

Filename

Format

[<spec>]
attribute1 = val1
attribute2 = val2
...

Spec

<sourcetype>
the sourcetype of an event
host::<host>
where <host> is the host for an event
reportinghost::<host>
where <host> is the reporting host for an event
source::<source>
where <source> is the source for an event

Attributes

If the same <spec> is found in two bundle directories, the following precedence rules apply.

Overriding is performed attribute by attribute.

Linemerging

SHOULD_LINEMERGE = <bool> (True)
When set to true splunk can combine several input lines into a single event, based on the following configuration attributes.

These are used only when SHOULD_LINEMERGE = True

AUTO_LINEMERGE = <bool> (True)
Directs splunk to use automatic learning methods to determine where to break a stream of lines into events.
MAX_EVENTS = <integer> (256)
Specifies the maximum number of input lines that will be added to any event. Splunk will break after the specified number of lines are read.

The following two conditions are checked in the order below. If the first matches, the second is not tested.

MUST_BREAK_AFTER = <regular expression> (Empty)
When set, and the regular expression matches the current line, splunk is guaranteed to create a new event for the next input line. Splunk may still break before the current line if another rule matches.
MUST_NOT_BREAK_AFTER = <regular expression> (Empty)
When set and the current line matches the regular expression, splunk will not break on any subsequent lines until the MUST_BREAK_AFTER expression matches.

The following four conditions are checked in the order below. If one matches, then the remainder are not tested.

BREAK_ONLY_BEFORE = <regular expression> (Empty)
When set, splunk will create a new event if and only if it encounters a new line that matches the regular expression.
BREAK_ONLY_BEFORE_DATE = <bool> False
When set, splunk will create a new event if and only if it encounters a new line with a date.
BREAK_BEFORE_DATE = <bool> (True)
When set to true, splunk will create a new event when it encounters a new line with a timestamp. It will also emit an event containing all lines since the last new event was created. Note that splunk will still create a new event unless there is a rule that says not to break or the automatic learning method determines that splunk should not break.
MUST_BREAK_BEFORE = <regular expression> (^\s*$)
Note Deprecated. When set, splunk will create a new event when it encounters a new line that matches the regular expression. This is similar to BREAK_BEFORE_DATE.

Timestamp extraction configuration

DATETIME_CONFIG = <filename relative to SPLUNK_HOME> (/etc/datetime.xml)
Specifies the file to configure the timestamp extractor. This configuration may also be set to "NONE" to prevent the timestamp extractor from running or "CURRENT" to assign the current system time to each event.
MAX_TIMESTAMP_LOOKAHEAD = <integer> (150)
Specifies how far into an event splunk should look for a timestamp.

Typing configuration

TYPING_CONFIG = <filename relative to SPLUNK_HOME>
(/etc/event-types/current/default.xml)

Specifies the file to configure the event typer. This configuration may also be set to "NONE" to prevent the event typer from running.

AUTO_TAG = <bool> (False)
Specifies whether to automatically tag new event types with important keywords from events of that event type.

Regex configuration

REGEXES<class> = <"regex name","regex name",...>
{see regexes.conf.spec}

Regular expressions can be defined in regexes.conf files. Splunk configures classes of regular expressions for each event.

For each class, splunk takes the configuration from the highest precedence configuration block. This means that if a particular class is specified for a source, it will override the same class if it is specified for a sourcetype. Similarly, if a particular class is specified in the local bundle for a sourcetype, it will override that class for the default bundle for that sourcetype.

Class names ("-annotation") must be unique within a Splunk instance, so the new Regex configuration does not override an existing configuration. Always add a class name rather than use "REGEXES" alone to avoid replacing Splunk default behavior.

The following is an example REGEXES class in the default bundle for all sourcetypes:

REGEXES-annotation = filetype,loglevel,os,browser,language,ip,email,url

This configuration uses the "test-pipeline" regex (defined in regexes.conf) for sourcetype "access_log":

[access_log]
REGEXES-test = test-pipeline

Source Type configuration

sourcetype = <string> (Empty)
If set for a [source::...] block, it will cause that source to be assigned the specified sourcetype.

Examples

If host matches nyc* then set Eastern Time Zone. See man tzet for format help.

[host::nyc.*]
TZ = EST-5EDT01:00:00,M4.1.0/02:00:00,M10.5.0/02:00:00

If the sourcetype is apache_error, linemerge source data into multi-line events.

[apache_error]
SHOULD_LINEMERGE = True

If the source matches, turn on auto-tagging. The tags will be based on useful-looking words in events during typing process.

[source::.*datatotag.*]
AUTO_TAG = 1

Use the shipped syslog-regex in regex-props.conf to extract meta events.

[source::.*sysloglikesource.*]
REGEXES = syslog-regex

Use the regex access-ip in overlay-regex-props.conf to extract meta-events.

[source::.*access.*]
REGEXES = access-ip

Create Regular Expressions

These parameters let you create regular expressions that can be invoked by properties to match patterns in data and meta data.

Splunk uses the PCRE library for Perl Compatible Regular Expressions. For additional information on building your own regular expressions, see the perlre documentation as well as the PCRE man pages for differences between PCRE and Perl 5. (Search for "PCRECOMPAT(3)".)

Filename

Format

[<regex name>]
attribute1 = val1
attribute2 = val2
...

Attributes

If the same <spec> is found in two bundle directories, the following precedence rules apply:

Overriding is performed attribute by attribute.

SOURCE_KEY = <string> (_raw)
Specifies which key in the event should be the source of the string on which to perform the regex.
LOOKAHEAD = <integer> (256)
Specifies how far into the string to match.
REGEX = <regular expression> (Empty)
If set, this is the regular expression that will operate on the data.
FORMAT = <string> ($1)
A string with arguments specified by $n, where n represents the output of the n-th parenthesized grop of the REGEX. If the regex does not have n groups, the matching fails. The special identifier $0 represents what was in the DEST_KEY before this regex was performed.
DEFAULT_VALUE = <string> (Empty)
If set, and the regex fails, this value will be written to the DEST_KEY.
DEST_KEY = <string> (<regex name>)
Specifies where the formatted result of running the regex should be stored. Any key prefixed by '_' will not be stored when it reaches the index.
REPEAT_MATCH = <bool> (False)
Specifies whether the regular expression should be run several times on the SOURCE_KEY. If set, it will start wherever the last match stopped, and will continue until no more matches are found.
WRITE_META = <string> (False)
Specifies that whatever was written to DEST_KEY should also be appended to the "meta data" for the event.

Splunk Reserved Keys

As data is streamed through Splunk pipelines, it lives in a set of keys. Some keys are indexed and/or stored and others are neither.

You can create new keys, any you create without _ prefixes are indexed in entirety (no major/minor segmentation) but not stored. Any key prefixed by '_' will not be stored when it reaches the index.

_raw
The raw text of the event. It is indexed with major/minor segmentation. It is intended to preserve the integrity of original content. You can change _raw with regexes, but you will destroy data integrity.
_done
If set to any string this is the last event in a stream.
_meta
The metadata for the event. It is indexed with space delimited segmentation.
_time
The timestamp of the event, in seconds since 1/1/1970 UTC. It is in the primary index but not stored.
queue
The queue to be used by the next queue output processor. It is neither indexed nor stored.
cluster
Used to create meta-events. It is indexed in entirety but not stored.
MetaData:FinalType
The event type hash. It is indexed in entirety but not stored.
MetaData:Host
The host associated with the event. The value must be prefixed by "host::" It is indexed in entirety but not stored.
_MetaData:Index
The index where the event should be stored. It is neither indexed nor stored.
_MetaData:IndexTerms
Used by Splunk to capture terms to be indexed but not stored. It is indexed with major/minor segmentation.
MetaData:Source
The source associated with the event. The value must be prefixed by "source::" It is indexed in entirety but not stored.
MetaData:Sourcetype
The sourcetype of the event. The value must be prefixed by "sourcetype::" It is indexed in entirety but not stored.

Examples

This creates a meta event for each IP address via regex.

[access-ip]
DEST_KEY      = cluster
REGEX         = (\d+\.\d+\.\d+\.\d+)
DEFAULT_VALUE = 
PREPEND       = ip::
WRITE_META    = False

To make use of the meta events, you can Splunk for anything and add the modifer index::metaevents. Or, if you search for two values that are in separate events but in the same meta event (because both events contain the same IP value), the meta event will appear in your results as if it were a real log event.

Add Input Configurations

These parameters add or reconfigure input sources from which the Splunk Server will access and index data.

Filename

Format

[<inputtype>:<path>]
attribute1 = val1
attribute2 = val2
...

Any attributes not specified will be set to the values for the [default] entry.

Input Types

Tail File

This directs splunk to use the tail file input module to watch all files in the <path>, or just <path> if it represents a single file.

Batch File

Same as tail file, except Splunk uses the batch file input module.

FIFO

This directs splunk to read from the fifo at the specified path.

TCP

This configures splunk to listen on the specified port. If a connection is made from <remote server>, this stanza is used to configure the input. If <remote server> is blank, this stanza matches all connections on the specified port.

UDP

Similar to TCP, except that it listens on a UDP port.

Splunk-2-Splunk

This is the same as TCP, except the remote server is assumed to be a splunk server. For splunktcp, the host or connection_host will be used if the remote splunk server does not set a host, or if the host is set to host::localhost.

ODBC

This will run the SQL command "SELECT * FROM <table>" on a regular basis. For each returned row, a new event will be created with the raw text being a comma separated list of column name=column value.

Attributes

Every input type has these possible options:

This is a shortcut for MetaData:Host = <string>. It sets the host of events from this input to be the specified string. "host::" is automatically prepended to the value when this shortcut is used.

This is a shortcut for _MetaData:Index = <string>. It sets the index where events from this input will be stored.

This is a shortcut for MetaData:Source = <string>. It sets the source name of events from this input to be the specified string. "source::" is automatically prepended to the value when this shortcut is used.

This is a shortcut for MetaData:Sourcetype = <string>. It sets the source type name of events from this input to be the specified string. "sourcetype::" is automatically prepended to the value when this shortcut is used.

This is a shortcut for Reportinghost = <string>. It sets the reporting host name of events from this input to be the specified string. "reportinghost::" is automatically prepended to the value when this shortcut is used.

This generically associates <string> with <KEY> for all events from this input.

Additional Attributes

Each input type has additional attributes specific to the type:

Tail

If specified, the tail file input module will use the specified regualar expression to extract the host from the filename of each input. Specifically the first group of the regex is used as the host. If the regex fails to match, the "host =" attribute is used as the host.

If specified, the tail file input module will use the specified '/' separated segment of the path as the host of each input. If the value is not an integer, or is less than 1, the "host =" attribute is used as the host.

If specified, files from this path will be tailed only if they match the specified regular expression.

If specified, files from this path will not be tailed if they match the specified regular expression. If a file is inadvertently specified for both _whitelist and _blacklist, _blacklist prevails and the file will not be tailed.

Batch

This specifies the policy to be used to handle the files. The "sinkhole" policy will delete the files as they are read, while the other two methods will link or copy the files into a separate directory.

Batch input ignores these parameters.

UDP

The UDP input processor will rewrite the host with the ip address of the remote server if "ip" is set, the DNS name of the remote server if "dns" is set, and will do nothing to the host for any other string.

Splunk-2-Splunk

This specifies where the TCP input processor should deposit the events that it reads.

ODBC

This specifies the DSN in which to find the table. It is required.

Examples

The following are example UDP input configurations. To use one of these configurations, copy the configuration block into inputs.conf in $SPLUNK_HOME/etc/bundles/local/inputs.conf

This configuration directs the server to listen on UDP port 514 for raw data from ANY remote server. The "host" of the data will be set as the IP address of the remote server.

[udp::514]

This configuration directs the server to listen on UDP port 9995 for raw data from ANY remote server. The "host" of the data will be set as the host name of the remote server. All data will also be assigned the sourcetype "log4j" and the source "UDP:9995"

[udp::9995]
connection_host = dns
sourcetype = log4j
source = udp:9995

This configuration directs the server to listen on UDP port 9994 for raw data from 10.1.1.10. All data will be assigned the
host "webhead-1", the sourcetype "access_common" and the the source "10.1.1.10/var/log/apache/access.log"

[udp:10.1.1.10:9994]
host = webhead-1
sourcetype = access_common
source = 10.1.1.10/var/log/apache/access.log

Recognize Source Types

Create new source types to describe your data input sources.

Filename

Format

[source::<path>]
sourcetype=mySourceType
...

Create a new configuration with the path to your file (or files) and the name you wish to assign as a sourcetype.

Examples

[source::/var/log/netinfo.log]
sourcetype=netinfo_log
[source::/var/log/httpd/mywebsite_*_log]
sourcetype=mywebsite_log

Authentication Configuration

These parameters configure your authentication scheme, either LDAP or the default Splunk Authentication.

Filename

Format

[<spec>]
attribute1 = val1
attribute2 = val2
...

Selecting an Authentication Scheme

[auth]
authType = Splunk

This sets the authentication to use, either "Splunk" or "LDAP". For Splunk Authentication, no further configuration is needed.

For more on LDAP, see the comments in the auth.conf.spec file for details. LDAP should be configured by an experienced administrator, as an incorrect configuration is likely to leave Splunk inaccessible.

Some important considerations:

The failsafe login is used when LDAP is unavailable. When you enter the plaintext failsafe password in auth.conf, the next time Splunk starts it will replace it with the encrypted password. Also, distributed live splunks uses the failsafe password because it does not have access to user credentials from LDAP.

You can configure and enable LDAP from the web interface as well. Note that when you switch authentication systems from the GUI, any users currently logged in will be logged out.

Error messages go to $SPLUNK_HOME/var/log/splunk/splunkd.log.

Live Splunks and Saved Splunks Configuration

These parameters configure Live Splunks and Saved Splunks

Filenames

Format

[<spec>]
attribute1 = val1
attribute2 = val2
...

For more information, see the comments in the livesplunks.conf.spec and savedsplunks.conf.spec files.

Live Splunk Alert via Email

If you configured your Live Splunk to send email when its alert threshold has been met, the parameters of the email message are specified in $SPLUNK_HOME/bin/liveSplunkMailer.sh. To change the format of the message (subject, message body, etc.) simply edit this file. If you checked "Include results in email", the default method is to attach the search results in uuencoded form. This can also be modified by changing the way the searchresults.txt file is created in liveSplunkMailer.sh.

Modules

How Modules Work

Modules add new pipelines and processors to splunkd. A module is a small directory of files that contains a configuration file, config.xml . It may contain other files, usually a processor executable. As with bundles, the intent of modules is to have discrete units that can be moved in or out as needed, to simplify customization and configuration.

Modules must be placed into a Splunk installation as subdirectories of the $SPLUNK_HOME/etc/modules/ directory.

Startup

When splunkd starts, it walks the modules directory (in chronological order, from oldest to newest) and reads the config.xml file in each subdirectory. If the file contains a valid pipeline configuration, splunkd launches the pipeline. Modules can share processors and other runnable code - a pipeline defined in one subdirectory can include a processor that lives in another module.

The default Splunk Server filesystem contains these subdirectories under $SPLUNK_HOME/etc/modules:

distributedSearch
pipeline for distributed search, maintained through the GUI or CLI
input
for accessing data outside Splunk and reading it into the Splunk Server
internal
internal Splunk functions (as opposed to add-on input methods or other functionality)
output
for sending data out by various methods

Pipelines will be loaded from any subdirectory of $SPLUNK_HOME/etc/modules. You should choose meaningful locations for your modules for easier maintenance. For example, you could put new input type modules under input, or create a new directory in etc/modules for all your custom modules.

There's only one required, canonical filename within each module subdirectory.

config.xml
defines a new pipeline or modifies Splunk's universal pipeline.

Your custom module can change the configuration of processors in the universal pipeline, such as inserting new processors ahead of, behind, or in place of those defined in the universal pipeline. You can also replace the universal pipeline with your own custom processing order, using a combination of built-in Splunk processors and your own.

Typical Uses

These are just a few of the possible uses for modules.

Custom input modules
For example, to read from a proprietary database source.
Extended processing
To look up data in a local database, or perform transformations that can't be done through properties.
Universal pipeline customization
To modify the default pipeline through which all data is routed, or to create a slightly modified version of the pipeline with a different configuration of its processors.

Customizing the Universal Pipeline

The universal pipeline, or the parsing pipeline, is where events are input, processed and output to the indexing pipeline. The processors that make up this pipeline can be reconfigured or replaced by creating a new module with your custom processing pipeline.

The Splunk processor loading architecture supports the ability to insert a processor before, after or instead of another processor. If you are extending splunk by providing your own processor or wish to change the processing pipelines defined in splunkd.xml it is recommended that you define your processor in a new module and use the "insertBefore", "insertAfter", or "replace" attribute.

For example, lets suppose you wanted to add a processor "replaceProcessor" just before the "indexer" processor:

  1. Create a new module directory in $SPLUNK_HOME/etc/modules/replaceProcessor
  2. Create a config.xml file in the new module directory
  3. Define a module configuration with a list of processors - note that for this case you do not use the <pipeline> ... </pipeline> tags because you are not defining a new pipeline but changing an existing one.
  4. Define your processor(s) in the config.xml file and specify the action and target and target pipeline.
    • action is a value : "insertBefore", "insertAfter" or "replace"
      • target is the name of processor to insert before/after or replace
        • pipelineTarget is the pipeline where the inserted or replaced processor exists.

Here is the config.xml for an example replaceProcessor (uses an example urlencodeprocessor.) It will insert the "replaceProcessor" before the "indexer" processor in the "indexerPipe" pipeline:

<module>
   <processor name="replaceProcessor" plugin="urlencodeProcessor" action="insertBefore" target="indexer" pipelineTarget="indexerPipe" >
       <config></config>
    </processor>
</module>

Below is a list of the processors, in order executed, that make up the default universal pipeline. You can see this in the pipeline "parsingPipeline" in $SPLUNK_HOME/etc/myinstall/splunkd.xml file.

readerIn
Queue Input processor, data comes in here
utf8
UTF8 processor
linebreaker
Line Breaking
aggregator
Line Merging/Date Extraction
regexreplacement
Regex Extraction
typing
Event Typing
clusterer
Meta Event Creation
sendOut
Queue Output, data sent to next pipeline

Custom Input Modules

Contact support for assistance with custom inputs. We are moving away from the need to make custom processors and modules for new data inputs.

Writing C/C++ Processors

Processors allow for custom actions that cannot be done by configuration alone. But note that many things that required a processor in previous versions are now supported without custom code.

Developing C/C++ processors for the Splunk server follows a simple development model where a user supplies a function/method that gets called by splunkd for every event processed. A data structure is passed to this function that contains the event data and meta data. Helper functions are provided to manipulate portions of the data structure such as setting host, adding meta data, etc.

There are 3 things that need to be done to build and hook up your own processor:

  1. Write a callback function
  2. Write a program with a main to register your function
  3. Add a config stanza that specifies your external processor

Below is a brief overview of each of the three steps. More complete examples can be found on on the Processor C Example and Processor C++ Example pages.

Writing Your Callback Function

The first thing to do is to implement a function with the following signature, this function will be called once for each event during processing. The actual name of the function need not be processEvent, it can be called anything you like.

     static int processEvent( struct SplunkTransaction *trans);

You will need to #include the header file splunk-extplugin.h located in SPLUNK_HOME/include.
An online version is on the splunk-extplugin.h page.

This header file defines helper functions that can be used to easily and safely manipulate the event data. There are several types of helper functions:

Get/Set Functions

These are used for manipulating the event and event meta data such as Host, Source, Sourcetype, the raw data, etc. Below are some of these functions, more can be found in the header file.

extern const char *SplunkGetSource(struct SplunkTransaction *trans);
extern void SplunkSetSource(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetSourceType(struct SplunkTransaction *trans);
extern void SplunkSetSourceType(struct SplunkTransaction *trans, const char *nval);

Logging Functions

The following functions provide logging facilities to the Splunk logs located in SPLUNK_HOME/var/log/splunk/splunkd.log.

extern void SplunkDebug(const char *fmt, ...)
extern void SplunkInfo(const char *fmt, ...)
extern void SplunkWarn(const char *fmt, ...)
extern void SplunkError(const char *fmt, ...)
extern void SplunkFatal(const char *fmt, ...)

Start splunk with the --debug option to see messages generated with the SplunkDebug function.

Configuration Functions

The following function provides key/value access to the tags in the external processes xml config (see below.)

extern const char *SplunkInstanceConfig(const char *key);

Error and Process Management

The following function provides a clean way to exit your custom processor.

extern void SplunkExit(int rval, const char *fmt, ...)

Write Main

The external processor is a separate application that is invoked during the Splunk startup process. The main() function is called, allowing for initialization and the hooking up of the event processing function described above.

The following example is the main function for the processor that just hooks up the event processing function. The function SplunkProcess_SingleThreaded registers your function so splunkd knows how to call it.

int main(int argn, char * const argv[])
{
    SplunkProcess_SingleThreaded(processEvent);
}

Note If you run your program from the command line, you will see the message "I/O Error -- error writing: Socket operation on non-socket." This is normal.

Add a Processor Config Stanza to Pipeline

To get your processor called, you must add it to a pipeline. Create a new module and pipeline for your processer and insert the XML for your processor in your module's config.xml.

The following XML stanza defines the external processor:

NOTE Replace the path in <command> ... </command> with the actual path to your executable, it can be anywhere but we recommend putting it in your custom module's directory. If your program requires command-line arguments, you can include them here although a better way to handle this is through config.xml. (See the complete examples for more on this.) extcmdprocessor is the built-in processor that handles custom processors.

<processor name="extcmdprocessor" plugin="extcmdprocessor" action="insertAfter" target="indexIn" pipelineTarget="indexerPipe" >
      <config>
       <command>$$SPLUNK_HOME]]/etc/modules/myModule/sampleProcessor</command>  <!--  $$SPLUNK_HOME]] references your splunk installation directory -->
      </config>
     </processor>

Restart splunk to load your module and register your processor with splunkd. Errors, warnings and your log messages can be found in the splunkd.log file. You will also see messages like this indicating your module has been loaded:

02-20-2007 09:27:41.846 INFO loader - Processing Module ----> /opt/splunk/etc/modules/myModule
02-20-2007 09:27:41.851 INFO loader - Overlaying processors
02-20-2007 09:27:41.851 INFO loader - Overlaying processor extcmdprocessor into pipeline. Target=indexIn Action=insertAfter
02-20-2007 09:27:41.851 INFO loader - processor extcmdprocessor overlay completed
02-20-2007 09:27:41.854 INFO splunklogger - Module /opt/splunk/etc/modules/myModule loaded

Complete C Example

Before building, you should first set up your build environment. An easy way is to source the script $SPLUNK_HOME/bin/setSplunkEnv. This is the same environment for running the Splunk server, so the same shared libraries can be found at runtime.

> source /opt/splunk/bin/setSplunkEnv

The following is sample processor written in C and just adds some text to the end of the event.
It can easily be extended using the same pattern to processor other attributes such as source, host, sourctype, etc.

1) compile it using

gcc -o sampleProcessor -I$SPLUNK_HOME/include -L$SPLUNK_HOME/lib sampleProcessor.c $SPLUNK_HOME/lib/libextcmdapi

2) copy the compiled binary to a convenient location, such as your module's directory under etc/modules
3) add the XML config section and restart

Configure the text string to append by adding this XML after your <command>:

<addToRaw>All your log are belong to Splunk</addToRaw>

Complete C Code

//  Sample Processor

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>

#include "splunk-extplugin.h"

// The following are used to hold config values picked up dring initialization
char *addToRaw;
char defaultAddToRaw[] = "No addToRaw specified in config file";

//-----------------------------------------------
// Function is called to mark up the raw data
//------------------------------------------------
void fixRaw(struct SplunkTransaction *trans)
{
  char *raw;
  char *newraw;

  /* get the raw data */
  /* SplunkGetRaw returns const char *, remember to play nice with it */
  raw = (char *)SplunkGetRaw(trans);
  SplunkDebug( "RAW = \"%s\"\n", raw );
 
 
  /* allocate memory for the data plus additional text */
  newraw = malloc(strlen(raw) + strlen(addToRaw));
  if (newraw == NULL)
  {
    SplunkExit(1, "out of memory");
  }
  
  /* Append text to all of the log lines */
  strcpy(newraw, raw);
  strcat(newraw, addToRaw);
  
  // Put the data back
  SplunkSetRaw(trans, newraw);
  SplunkDebug( "NEW RAW = \"%s\"\n", newraw );

  free(newraw);
}

//---------------------------------------------------------
// This function is called for each event during processing 
//---------------------------------------------------------
static int processEvent(struct SplunkTransaction *trans)
{
    // Call our helper function to mark up the raw data
    fixRaw( trans );

    // return true to continue sending the event through pipeline
    return 1;  
}

//---------------------------------------------------------
//  Main is called once during initialization
//  We pull out config values declared in the xml config
//---------------------------------------------------------
int main(int argn, char * const argv[])
{  
  (void) argn;
  
  // arguments can be passed in on the <command> config line
  // it is more tidy to supply additonal config through their own xml tags ( see below )
  if (argv[1] != NULL)
  {
    // if we want pull out the args
  }

  // additional key values can be passed in through the config files.
  // These values will be used during processing each event. 
  addToRaw = (char *) SplunkInstanceConfig("addToRaw");
  if ( addToRaw == NULL )
  {

    addToRaw = malloc(strlen(defaultAddToRaw));
    strcpy(addToRaw, defaultAddToRaw);
    
    // If in Warn mode then log that no value was passed
    SplunkWarn( "No value for addToRaw specified in sample processors config" );
    
    // if we wanted we could exit
    //    SplunkExit(1, "No value for addToRaw in config");
  }

  // spit out a debug message with text to append
  SplunkDebug("Using \"%s\" to add to raw", addToRaw);
  
  // This will hook up the per event processing function
  SplunkProcess_SingleThreaded(processEvent);
}

Complete C++ Example

Before building, you should first set up your build environment. An easy way is to source the script $SPLUNK_HOME/bin/setSplunkEnv. This is the same environment for running the Splunk server, so the same shared libraries can be found at runtime.

> source /opt/splunk/bin/setSplunkEnv

The following is sample processor written in C++ and just adds some text to the end of the event.
It can easily be extended using the same pattern to processor other attributes such as source, host, sourctype, etc.

1) compile it using

g++ -o sampleProcessor -I$SPLUNK_HOME/include -L$SPLUNK_HOME/lib sampleProcessor.cpp -lextcmdapi

2) copy the compiled binary to a convenient location, such as your module's directory under etc/modules
3) add the XML config section and restart

Configure the text string to append by adding this XML after your <command>:

<addToRaw>All your log are belong to Splunk</addToRaw>

Complete C++ Code

#include "splunk-extplugin.h"
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <errno.h>

//----------------------------------------------------------------
// Sample Processor class
//----------------------------------------------------------------
class SampleProcessor : public Splunk::SingleThreadedProcessor {
    std::string  addToRaw;   

    //----------------------------------------------------------------
    // Method to append data to raw
    // Modifying host, source, sourcetype, etc follow the same pattern 
    //----------------------------------------------------------------
    bool fixRaw(Splunk::Transaction *trans)
    {

        // get the raw data
        // SplunkGetRaw returns const char *, remember to play nice with it    
        const char *r = trans->getRaw();

        // If in debug mode dump out the data
        SplunkDebug( "RAW = \"%s\"\n", r );

        std::string rs(r);

        // append the string to raw
        rs.append(addToRaw);

        // put the modified raw data back 
        trans->setRaw(rs);
        
        // return true to continue sending the event through pipeline
        return true;        
    }

    //----------------------------------------------------------------
    // This method is called once per event in the pipeline
    //----------------------------------------------------------------
    bool handler(Splunk::Transaction *trans)
    {
        // call method to fix up raw
        return fixRaw(trans);
    }

    public:
    

    //----------------------------------------------------------------
    // Constructor pulls config values
    //----------------------------------------------------------------
    SampleProcessor(char * const args[])
    {
        // during initialization we try and read config data and cache result
        addToRaw.append(" {");

        // additional key values can be passed in through the config files.
        // These values will be used during processing each event. 
        const char *r = Splunk::InstanceConfig["addToRaw"];
        addToRaw.append((r == NULL) ? "UNKNOWN" : r);
        addToRaw.append("}");

        // spit out a debug message with text to append
        SplunkDebug("Using \"%s\" to add to raw", addToRaw.c_str() );

    }
};

//---------------------------------------------------------
//  Main is called once during initialization
//  We construct our processor class and call run
//---------------------------------------------------------
int main(int argn, char * const argv[])
{
    (void) argn;
    // if we had passed args in via the command tag we could pull them here.
    if (argv[1] == NULL);

    // construct our class and go
    SampleProcessor sp(argv);
    sp.run();

    return 1;
}

Using a Bundle with a Module

You can create a bundle to configure your module, just as you can use bundles with Splunk standard modules. Below is an example using a bundle to only send certain events, 404 messages from an apache log file, to the custom module.

Create the Processor

This example uses the sample C++ module described on the previous page, here referred to as "sampleProcessor".

Create the Module

Make a new directory under $SPLUNK_HOME/etc/modules. For this example, it is called "testModule" and the sampleProcessor executable file goes there.

Next, create a new config.xml with a new pipeline and the desired processors. This pipeline calls "readerin", "sampleProcessor" and "sendOut". readerin is configured to get data from the queue "testQueue", sampleProcessor appends the text "{xyzzy}" to the raw event and sendOut passes it to indexQueue to continue to the indexer.

Note You can also specify the full path instead of using "$$SPLUNK_HOME]]"

<module>
  <pipeline name="test-pipeline" type="startup">

    <processor name="readerIn" plugin="queueinputprocessor">
      <config>
        <queueName>testQueue</queueName>
      </config>
    </processor>

    <processor name="myProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>xyzzy</addToRaw>
      </config>
    </processor>

    <processor name="sendOut" plugin="queueoutputprocessor">
      <config>
        <queueName>indexQueue</queueName>
      </config>
    </processor>

  </pipeline>
</module>

Create the Bundle

Make a new directory under $SPLUNK_HOME/etc/bundles, here also called testModule. In it, create three new files:

Note Replace testbox.splunk.com with your hostname and the tail path with your desired path.

inputs.conf specifies the files we want to tail and assigns a sourcetype.

host = testbox.splunk.com

[tail:///var/log/httpd]
disabled = false
host = testbox.splunk.com
sourcetype = access_log

props.conf specifies that events with sourcetype access_log should use the regular expression configured in regexes.conf for test-pipeline. "REGEXES-test" specifies the class ("-test") to identify a particular configuration stanza. Class names must be unique for each Splunk instance to avoid overriding existing behavior.

[access_log]
REGEXES-test = test-pipeline

regexes.conf defines test-pipeline to look for the specified regular expression and send those events to a queue named "testQueue".

[test-pipeline]
REGEX = \s404\s
DEST_KEY = queue
FORMAT = testQueue

The complete bundle looks like this:

testbox [root]:/opt/splunk/etc/bundles$ ls testModule
inputs.conf     props.conf      regexes.conf

Restart splunk to load the new pipeline. The 404 events should now show the specified text appended:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy}

Add an Additional Processor

You can configure several processors in the same pipeline. For this example, to add a second instance of sampleProcessor, add the XML for a second processor with a different name. It can reference the same executable as myProcessor.

    <processor name="myOtherProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>You are in a maze of twisty little passages, all alike</addToRaw>
      </config>
    </processor>

Then, the text of the event would look like this:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy} {You are in a maze of twisty little passages, all alike}

splunk-extplugin.h

This is the header file for the external api library. It can be found in $SPLUNK_HOME/include.


/* Copyright (C) 2005-2007 Splunk Inc.  All Rights Reserved.  Version 2.2 */

#ifndef SPLUNK_EXTPLUGIN_H
#define SPLUNK_EXTPLUGIN_H

#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */

#include <stdarg.h>
#include <time.h>

enum SplunkLogLevel {
    SPLUNKLOG_DEBUG,
    SPLUNKLOG_INFO,
    SPLUNKLOG_WARN,
    SPLUNKLOG_ERROR,
    SPLUNKLOG_FATAL,
};

extern void SplunkVLog(enum SplunkLogLevel level, const char *fmt, va_list va)
#ifdef __GNUC__
    __attribute__ ((format (printf, 2, 0)))
#endif /* __GNUC__ */
    ;
extern void SplunkLog(enum SplunkLogLevel level, const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 2, 3)))
#endif /* __GNUC__ */
    ;

extern void SplunkVExit(int rval, const char *fmt, va_list va)
#ifdef __GNUC__
    __attribute__ ((format (printf, 2, 0)))
    __attribute__ ((noreturn))
#endif /* __GNUC__ */
    ;
extern void SplunkExit(int rval, const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 2, 3)))
    __attribute__ ((noreturn))
#endif /* __GNUC__ */
    ;

/* Convinience functions so you don't have to specify log level */
extern void SplunkDebug(const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif /* __GNUC__ */
    ;
extern void SplunkInfo(const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif /* __GNUC__ */
    ;
extern void SplunkWarn(const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif /* __GNUC__ */
    ;
extern void SplunkError(const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif /* __GNUC__ */
    ;
extern void SplunkFatal(const char *fmt, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif /* __GNUC__ */
    ;

struct SplunkTransaction;    /* Opaque type */

extern const char *SplunkGetRaw(struct SplunkTransaction *trans);
extern void SplunkSetRaw(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetConf(struct SplunkTransaction *trans);
extern void SplunkSetConf(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetDone(struct SplunkTransaction *trans);
extern void SplunkSetDone(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetHost(struct SplunkTransaction *trans);
extern void SplunkSetHost(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetMeta(struct SplunkTransaction *trans);
extern void SplunkSetMeta(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetPath(struct SplunkTransaction *trans);
extern void SplunkSetPath(struct SplunkTransaction *trans, const char *nval);

extern time_t SplunkGetTime(struct SplunkTransaction *trans);
extern void SplunkSetTime(struct SplunkTransaction *trans, time_t nval);

extern const char *SplunkGetType(struct SplunkTransaction *trans);
extern void SplunkSetType(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetIndex(struct SplunkTransaction *trans);
extern void SplunkSetIndex(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetInput(struct SplunkTransaction *trans);
extern void SplunkSetInput(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetSource(struct SplunkTransaction *trans);
extern void SplunkSetSource(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetSourceType(struct SplunkTransaction *trans);
extern void SplunkSetSourceType(struct SplunkTransaction *trans,
                const char *nval);

extern const char *SplunkGetIndexTerms(struct SplunkTransaction *trans);
extern void SplunkSetIndexTerms(struct SplunkTransaction *trans,
                const char *nval);

extern const char *SplunkGetReportingHost(struct SplunkTransaction *trans);
extern void SplunkSetReportingHost(struct SplunkTransaction *trans,
                   const char *nval);

/* Asks splunkd for a setting from the property map */
extern const char *SplunkConfig(struct SplunkTransaction *trans,
                const char *key);
/* And from the XML config: */
extern const char *SplunkInstanceConfig(const char *key);

extern void SplunkProcess_SingleThreaded(int (*f)(struct SplunkTransaction *))
#ifdef __GNUC__
    __attribute__ ((noreturn))
#endif /* __GNUC__ */
    ;

extern void SplunkProcess_SetEofHandler(void (*handler)(void));
extern void SplunkProcess_SetOomHandler(void (*handler)(void));

#ifdef __cplusplus

extern int splunk_transaction_is_supposed_to_be_opaque;

} // extern "C"

#include <string>

// This is a little evil -- even though we're an opaque type we allow
// C++-style method accesses
struct SplunkTransaction {
    SplunkTransaction()
    {
        // This just generates a link-time error
        splunk_transaction_is_supposed_to_be_opaque++;
    }
#define make_methods(cname)                        const char *get##cname(void)                    {                                    return SplunkGet##cname(this);                }                                void set##cname(const char *nval)                {                                    SplunkSet##cname(this, nval);                }                                void set##cname(const std::string &nval)            {                                    set##cname(nval.c_str());                }
    make_methods(Raw)
    make_methods(Conf);
    make_methods(Done);
    make_methods(Host);
    make_methods(Meta);
    make_methods(Path);

    time_t getTime(void)
    {
        return SplunkGetTime(this);
    }
    void setTime(time_t nval)
    {
        SplunkSetTime(this, nval);
    }

    make_methods(Type);
    make_methods(Index);
    make_methods(Input);
    make_methods(Source);
    make_methods(SourceType);
    make_methods(IndexTerms);
    make_methods(ReportingHost);
#undef make_methods

    const char *config(const char *key)
    {
        return SplunkConfig(this, key);
    }
    const char *config(const std::string &key)
    {
        return SplunkConfig(this, key.c_str());
    }
};

// We also provide a "Splunk::" namespace which allows clients to be written
// with a completely C++ binding
namespace Splunk {
    typedef struct SplunkTransaction Transaction;

    class InstanceConfig_IMPL {
        public:
        InstanceConfig_IMPL()
        {
        }
        const char *operator[] (const char *confval) const
        {
            return SplunkInstanceConfig(confval);
        }
        const char *operator[] (const std::string &confval) const
        {
            return SplunkInstanceConfig(confval.c_str());
        }
    };
    extern const InstanceConfig_IMPL InstanceConfig;

    typedef enum SplunkLogLevel LogLevel;
    void VLog(LogLevel level, const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 2, 0)))
#endif // __GNUC__
        ;
    void Log(LogLevel level, const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 2, 3)))
#endif // __GNUC__
        ;
    inline void Log(LogLevel level, const std::string &str)
    {
        Log(level, "%s", str.c_str());
    }
    void VExit(int rval, const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 2, 0)))
        __attribute__ ((noreturn))
#endif // __GNUC__
        ;
    void Exit(int rval, const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 2, 3)))
        __attribute__ ((noreturn))
#endif // __GNUC__
        ;
    void Exit(int rval, const std::string &str)
#ifdef __GNUC__
        __attribute__ ((noreturn))
#endif // __GNUC__
        ;
    void Exit(int rval)
#ifdef __GNUC__
        __attribute__ ((noreturn))
#endif // __GNUC__
        ;
    void VDebug(const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
        ;
    void Debug(const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
        ;
    inline void Debug(const std::string &str)
    {
        Debug("%s", str.c_str());
    }
    void VInfo(const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
        ;
    void Info(const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
        ;
    inline void Info(const std::string &str)
    {
        Info("%s", str.c_str());
    }
    void VWarn(const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
        ;
    void Warn(const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
        ;
    inline void Warn(const std::string &str)
    {
        Warn("%s", str.c_str());
    }
    void VError(const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
        ;
    void Error(const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
        ;
    inline void Error(const std::string &str)
    {
        Error("%s", str.c_str());
    }
    void VFatal(const char *fmt, va_list va)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
        ;
    void Fatal(const char *fmt, ...)
#ifdef __GNUC__
        __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
        ;
    inline void Fatal(const std::string &str)
    {
        Fatal("%s", str.c_str());
    }

    class SingleThreadedProcessor {
        virtual bool handler(Transaction *trans) = 0;
        static int _shim_handler(Transaction *trans);
        virtual void eof(void);
        static void _shim_eof_handler(void);
        virtual void oom(void);
        static void _shim_oom_handler(void);

        // Allow logging functions to be called directly as methods
        // inside of ->handler()
        protected:
        static void vlog(LogLevel level, const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 2, 0)))
#endif // __GNUC__
            ;
        static void log(LogLevel level, const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 2, 3)))
#endif // __GNUC__
            ;
        static inline void log(LogLevel level, const std::string &str)
        {
            log(level, "%s", str.c_str());
        }
        static void vexit(int rval, const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 2, 0)))
            __attribute__ ((noreturn))
#endif // __GNUC__
            ;
        static void exit(int rval, const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 2, 3)))
            __attribute__ ((noreturn))
#endif // __GNUC__
            ;
        static void exit(int rval, const std::string &str)
#ifdef __GNUC__
            __attribute__ ((noreturn))
#endif // __GNUC__
            ;
        static void exit(int rval)
#ifdef __GNUC__
            __attribute__ ((noreturn))
#endif // __GNUC__
            ;
        static void vdebug(const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
            ;
        static void debug(const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
            ;
        static inline void debug(const std::string &str)
        {
            debug("%s", str.c_str());
        }
        static void vinfo(const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
            ;
        static void info(const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
            ;
        static inline void info(const std::string &str)
        {
            info("%s", str.c_str());
        }
        static void vwarn(const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
            ;
        static void warn(const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
            ;
        static inline void warn(const std::string &str)
        {
            warn("%s", str.c_str());
        }
        static void verror(const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
            ;
        static void error(const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
            ;
        static inline void error(const std::string &str)
        {
            error("%s", str.c_str());
        }
        static void vfatal(const char *fmt, va_list va)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 0)))
#endif // __GNUC__
            ;
        static void fatal(const char *fmt, ...)
#ifdef __GNUC__
            __attribute__ ((format (printf, 1, 2)))
#endif // __GNUC__
            ;
        static inline void fatal(const std::string &str)
        {
            fatal("%s", str.c_str());
        }

        public:
        SingleThreadedProcessor()
        {
        }
        virtual ~SingleThreadedProcessor()
        {
        }
        void run(void)
#ifdef __GNUC__
            __attribute__ ((noreturn))
#endif // __GNUC__
            ;
    };
};

#endif /* __cplusplus */

#endif /* !SPLUNK_EXTPLUGIN_H */

Skins

How Splunk Uses Skins

The default interface is defined by the HTML, CSS, JavaScript and XSL found under the $SPLUNK_HOME/share/splunk/search directory. Each skin has a CSS file that overrides the default styles, particularly for color properties and images.

Skins css Files

$SPLUNK_HOME/share/splunk/search/static/css/skins

The .css files in this directory are where most customizations are done. Files placed here will appear in the Themes menu in the splunkSearch interface. Since any file is assumed to be a skin, don't forget to remove temporary files created by your text editor. The standard skins provided are basic.css, the default theme, and black.css, the all-black "inverted" or "night mode" theme. You can use either of these files as a base to design your own themes.

images/skins directory

$SPLUNK_HOME/share/splunk/search/images/skins

This is the standard location for image files, one directory for each skin. You can use the default images in your new skins by specifying their path from $SPLUNK_HOME/share/splunk/search (like "background-image:url(/images/skins/basic/menu_arrow_bg.gif".) Any new images should go in a new directory to avoid confusing them with the default ones.

static/css directory

This directory contains the standard styles that control the appearance of the interface. You can override default properties by adding individual elements to your custom skin CSS file with the new styles.

Add or Remove Themes

Copy your new theme's .css file to the $SPLUNK_HOME/share/splunk/search/static/css/skins directory (or create a new one) to add a new theme. To remove an existing theme, delete its .css file. Restart the server to see your changes.

How To Create a New Skin

The easiest way to create a new skin is to copy an existing one to a new .css file and edit the items you wish to change. The standard skin files contain comments to help you identify which items you are interested in, but expect to make many changes scattered throughout the file due to to the complexity of interface elements.

Edit the new file, restart the server, and select your theme from the Preferences menu.

Note: You can also restart the web interface only with this command:

# splunk restart splunkweb

Splunk does not validate the skins files. In general, an invalid (or empty) css file will still leave the interface usable, but may not appear as expected.

Examples

This example shows how to change the standard popup menus from white to blue.

The menus have two basic colors, one for the background and one for the highlight. In reality, there are multiple elements that need to have the correct style applied to make it all look like a single color menu. And there are sometimes side-effects to watch out for.

For this example, we will use a medium blue (#6699ff) for the background and a light blue (#99ccff) for the highlight.

Copy basic.css to a new file, blue.css.

First, change the menu background color:

/* POPUP MENUS */
/* basic styles for submenu arrow-icons and highlight/opened states */
.popupMenu ul{
        background-color:#6699ff;  /* was #fff */
}
.popupMenu li.secondary label {
        background-color:#6699ff;  /* was #fff */
        background-image:url(/images/skins/basic/menu_arrow_bg.gif);
        background-position:right;
        background-repeat:no-repeat;
}

Next, change the menu item border to match:

/*---------------------*/
/* border color styles */
/*---------------------*/
.popupMenu ul label {
        border:1px solid #6699ff;  /* was #fff */
}

The check mark is an image, and the one that basic.css (menu_checkbox_bg.gif) uses has a white border. If you prefer something else, you can create a new image or use a different one such as menu_checkbox_boxed_bg.gif:

/* checkboxes and radio buttons in menu options */
.softWrap #softWrap,
.helpActive #helpActive,
.exploded #exploded,
#liteMode {
        background-image:url(/images/skins/basic/menu_checkbox_boxed_bg.gif); /* was menu_checkbox_bg.gif */
        background-repeat:no-repeat;
}

Now the highlight is where it gets complicated.

In basic.css, there are two elements using the same style:

.popupMenu li.secondary label.explicitMouseOver,
.popupMenu li.open label {
        background-color:#fffad1;
}

An open menu item is .popupMenu li.open label, but we don't want to also change .popupMenu li.secondary label.explicitMouseOver. So change these lines so each element has its own style. Then we can apply the new style only the one we are interested in:

.popupMenu li.secondary label.explicitMouseOver,
{
        background-color:#fffad1;
}
.popupMenu li.open label {
        background-color:#99ccff; /* was #fffad1 */
}

And again for the divider between sections of the menu:

.popupMenu ul, 
/* .popupMenu ul li,  handle this below */
.typeAhead,
div.histogramAndTabs,
div#tabs,
div.subsections,
div#topBar .selected,
input.disabled, select.disabled {
    border-color:#ccc;
}

.popupMenu ul li {
        border-color:#99ccff; /* was #ccc */
}

Similarly, the highlight of the selected item shares a style with another element:

label.explicitMouseOver,
.typeAhead label.explicitMouseOver {
        background-color:#fffad1 !important;
}

We can separate the two as we did above, but there is a side-effect to consider. label.explicitMouseOver is also used for other lists, such as the Source Types and Sources lists on the Search page, as well as highlighing segments in a list of events. You can change it, but be aware that it will impact more than just the menu.

If you still want light blue, change only label.explicitMouseOver:

label.explicitMouseOver  {
        background-color:#99ccff !important; /* was #fffad1 */
}

.typeAhead label.explicitMouseOver {
        background-color:#fffad1 !important;
}

Re-brand Splunk

Use a custom skin and your own images to re-brand the Splunk interface for your organization. Create a new directory for your theme's images under $SPLUNK_HOME/share/splunk/search/images/skins and change the path and filenames in your css file.

Many images are constrained in size by other elements that are part of the default interface, outside the skin css file. For best results, use the same size image as the one you are replacing.

Examples

Put a new logo next the Splunk box on the home page by specifying another image. Change a#logo for the free version and .proVersion a#logo for Splunk Professional.

a#logo {
        background-image:url(/images/skins/blue/logo_136x52.jpg); /* was basic/logo_splunk.png */
        background-repeat:no-repeat;
}
.proVersion a#logo {
        background-image:url(/images/skins/blue/logo_202x65.jpg); /* was basic/logo_splunk_pro.png */
        background-repeat:no-repeat;
        width:202px;
        height:65px;
}

Change the logo on the Splunk Professional login page:

.proVersion a#bigLogo {
        background-image:url(/images/skins/blue/logo_blue.gif); /* was basic/logo_big_splunk_pro.png */
        background-repeat:no-repeat;
}

2_1skinsHideResults

(This wiki page has not been created yet. Click the 'edit' button to create the first version.)

Learn more about creating wiki pages at WikiTutorial.

Hide Admin Controls

You can disable access to the Admin menu by changing the display property of its element to "none". To override the default behavior in your stylesheet, add it (normally found only in default.css) to your skin file and then set the new value.

#adminLink {
        display:none;
}

You can add display: none; to other elements to disable them in your theme.

Warning If you disable the Preferences menu or the Themes item, you will not be able to switch back to another theme. You can revert to the default Basic skin by removing the cookie that determines your skin preferences. Look in your browser's cookie list for one from your hostname called "setSkin".

Change Interface Text

The text strings presented to the user in the web interface are found in three files, TextContentAndErrorMessages.py, FrontEndConfiguration.py. in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/search and $SPLUNK_HOME/share/splunk/search/dynamic/html/main_ui.html.

Warning These files are overwritten during upgrade. Save a copy of your changes in a safe location for future reference. Also be aware the structure and content of these files will change substantially in future versions.

Save the files as UTF-8 and restart the web interface ("splunk restart splunkweb") for any changes to take effect.

TextContentAndErrorMessages.py contains most of the text used in the web interface as a list of strings.

pythonStrings["NUMBER_OF_RESULTS_PULLDOWN_TOOLTIP"] = "Wählen Sie die Ergebnisse die sie darstellen wollen."

FrontEndConfiguration.py contains the text for tabs, user types, Admin page labels and distributed status messages. When editing this file, take care to maintain the format of the existing items.

adminSectionRowLabels["user"] = {"name" : "Username", \
"realName" : "Nom et prénoms", \
"userType" : "Rôle", \
}

distributedStatusMessages = {
"notauthenticated" : "problema di autorizzazione",
"invaliduserorpsw" : "problema di accesso",
"down" : "non raggiungibile",
"versionMismatch" : "versione errata",
"productMismatch" : "nessuna licenza",
"missingServerName" : "nessun nome del server"
}

main_ui.html contains a few others such as Start Time and End Time popup menu labels.

Workflow Integration

Live Splunk Integrations to other Apps

Live Splunks can be configured to notify other applications in via script, RSS or email, and to optionally include the search results. Report Splunks can include a full report table.

RSS

RSS feeds use the format shown below. The link value is a permalink URL to run the Live Splunk now on the server.

<?xml version="1.0"?>
<rss version="2.0">
    <channel>
        <title>test</title>
        <link>http:qa-fc4:8000/?events?q=meta%3a%3aall%20</link>
        <description>Live,Splunk Feed for live splunk test</description>
        <item>
            <title>Query run from 0 To 1134079839</title>
            <link>http:qa-fc4:8000/?events?q=meta%3a%3aall%20%20starttimeu%3a%3a0%20endtimeu%3a%3a1134079839</link>
            <description> The Number of Events (1000) was Greater Than 1.</description>
            <pubDate>1134079840</pubDate>
        </item>
    </channel>
</rss>

Shell Script

The splunkd process will run a configured alert script and pass it these parameters.

Email

Alert messages use the format below. Outbound SMTP must be enabled on the Splunk Server's host.

From: livesplunk@splunk.enet.interfoo.net (Joe Admin)
Date: November 16, 2005 3:19:23 PM PST
To: admin-list@interfoo.net
Subject: alert if fewer than 10.

Live Splunk http:splunk.enet.interop.net:8000/?events?q=7%318989%32500%20m%69%6e%75t%65sago
%3a%3a%310%20doma%69%6e%3a%3ad%65fa%75%6ct%20 triggered with the result :

The Number of Events (0) was Less Than 10.

Splunk Name : calls home last 10 minutes
Query Terms : 7189892500 minutesago::10 index::default

Auto-generated by Splunk Professional

Launching Splunk with Permalinks

Permalinks are URLs with embedded search parameters. They open the splunkweb UI and run a specific search to display the results.

Parameters

The number of search results to display per page.

The Splunk search terms. The syntax is the same used in the Splunk box. Use "%20" for spaces. URL encoding - "%3A" instead of ":" - is optional but not required.