This document last updated: 08/11/08 11:08am

Print Developer Manual

Overview

Overview

Splunk provides timeline indexing for event based logging activities, such as those contained in system logfiles. The Splunk product platform is powerful and complex, and provides methods for multiple levels of customization including administration settings, configuration files, a REST API, a SOAP API, CSS configuration and more.

Use the information in the following sections to extend Splunk or embed Splunk into applications.
You can:

and more.

Proceed through the links at left to learn more.

User Interface

User Interface

The SplunkWeb user interface can be modified to accomplish rebranding and color changes.

Skinning SplunkWeb

The default interface is defined by the HTML, CSS, JavaScript and XSL found under the $SPLUNK_HOME/share/splunk/search_oxiclean directory. Each skin has a CSS file that overrides many default style settings, particularly for color properties and images.

Skins css Files

$SPLUNK_HOME/share/splunk/search_oxiclean/static/css/skins

The .css files in this directory are where most customizations are done. Files placed here will appear in the Themes drop-down of the Preferences menu in SplunkWeb. Since any file is assumed to be a skin, don't forget to remove temporary files created by your text editor. The standard skins provided are basic.css, the default theme, and black.css, the all-black "inverted" or "night mode" theme. You can use either of these files as a base to design your own themes.

Images directory

$SPLUNK_HOME/share/splunk/search_oxiclean/images/skins

This is the standard location for image files, one directory for each skin. You can use the default images in your new skins by specifying their path from $SPLUNK_HOME/share/splunk/search_oxiclean (like "background-image:url(/images/skins/basic/menu_arrow_bg.gif".) Any new images should go in a new directory to avoid confusing them with the default ones.

static/css directory

This directory contains the standard styles that control the appearance of the interface. You can override default properties by adding individual elements to your custom skin CSS file with the new styles.

Splunk toolbar for Firefox

The Splunk Firefox toolbar is a standard browser extension that can be customized for your specific environment. It is located in the $SPLUNK_HOME/share/splunk/extras/splunkbar directory. To install it, copy the splunktoolbar.xpi file to your local client machine and drag it into an open Firefox window. You will be prompted to install and then restart Firefox to enable the toolbar.

For more information on working with Firefox extensions, see http://developer.mozilla.org/en/docs/Extensions

Permalinks

Permalinks are URLs with embedded search parameters. They open SplunkWeb and run a specific search to display the results. If you do not currently have a valid session in your browser, you will be prompted to login.

Parameters

The Splunk search terms. The syntax is the same used in the Splunk box. Use "%20" for spaces. URL encoding (using "%3A" instead of ":") is supported but not required.

Unsupported Parameters

URLs that appear in your browser when using SplunkWeb may contain other parameters, such as "eventspage," but these are not yet supported in permalinks. They will be ignored.

Examples

http://localhost:8000/?q=meta%3A%3Aall

http://localhost:8000/?q=savedsearch::%22Splunk%20errors%20last%2024%20hours%22

Configuring SplunkWeb

Skins

How Splunk Uses Skins

The default interface is defined by the HTML, CSS, JavaScript and XSL found under the $SPLUNK_HOME/share/splunk/search_oxiclean directory. Each skin has a CSS file that overrides the default styles, particularly for color properties and images.

Note The directory $SPLUNK_HOME/share/splunk/search_pinesol contains old UI files. They no longer work with the 3.0 UI, do not make changes in this directory.

Skins css Files

$SPLUNK_HOME/share/splunk/search_oxiclean/static/css/skins

The .css files in this directory are where most customizations are done. Files placed here will appear in the Themes menu in SplunkWeb. Since any file is assumed to be a skin, don't forget to remove temporary files created by your text editor. The standard skins provided are basic.css, the default theme, and black.css, the all-black "inverted" or "night mode" theme. You can use either of these files as a base to design your own themes.

images/skins directory

$SPLUNK_HOME/share/splunk/search_oxiclean/images/skins

This is the standard location for image files, one directory for each skin. You can use the default images in your new skins by specifying their path from $SPLUNK_HOME/share/splunk/search (like "background-image:url(/images/skins/basic/menu_arrow_bg.gif".) Any new images should go in a new directory to avoid confusing them with the default ones.

static/css directory

This directory contains the standard styles that control the appearance of the interface. You can override default properties by adding individual elements to your custom skin CSS file with the new styles.

Localization

The text strings presented to the user in SplunkWeb are defined in literals.conf The full list can be found in $SPLUNK_HOME/etc/bundles/default/literals.conf. Strings are organized in stanzas, like other Splunk conf files, and can be put in a custom bundle. Be sure to use the correct stanza name for the strings you wish to change, some stanzas have a large number of items.

Warning Do not alter or remove any format strings (like "%s") that are used to display a variable. Splunk will still attempt to pass the argument, if it is missing, you will see a Python stack trace with the message "exceptions.TypeError: not all arguments converted during string formatting". Also, incorrect HTML (like mismatched or missing tags) may generate an exception.

If you make any changes including non-ascii characters, use an editor that can save the files as UTF-8. Restart the web interface ("splunk restart splunkweb") for changes to take effect.

[ui-admin-section-labels-/admin/users]
name = Username
realName = Nom et prénoms
userType = Rôle

[ui]
FILE_ALREADY_UPLOADED_MESSAGE = Splunk is already loading a file named %s. Please wait for it to finish, then try uploading this file again.

Scripting the Splunk search command

You can create custom scripts to handle your Splunk search results and function as a new search command. To build a search script, put a Python script in $SPLUNK_HOME/etc/searchscripts. Python scripts in the searchscripts directory are available in the search language and can be used in a search.

Some things to know about passing results to and from a search command:

If your Python script is called myNewCommand.py, it can be used in a search as follows:

access denied | myNewCommandSearch

Please note:

The splunk.Intersplunk module directs events from Splunk to your Python search scripts.

The output of your script can then be fed back into Splunk as events. In the simplest case, your script does nothing and just returns what it received. To accomplish this, you would write the following script:

import sys,splunk.Intersplunk

# this call populates the results variable with all the events passed into the search script:
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

# hand the results right back to Splunk
splunk.Intersplunk.outputResults(results)

Although this code snippet does not do much, it shows you how you get the events and how you pass the data back to Splunk. If you want to change some of the events, you would add a loop to iterate over all the events. Each event is comprised of a set of key-value pairs for every extracted field.

import sys,splunk.Intersplunk

# this call populates the results variable with all the events passed into the search script:
results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults()

# Iterate over all the events:
for result in results:
  # for all the events, you want to iterate over all the extracted fields:
  for key,value in result.items():
    # change the result items. This example makes all the values lowercase.
    value = value.lowercase()
    # add the changed values to a new array that is later passed back to Splunk.
    newresults.append( {key:value} )

# hand the results right back to Splunk
splunk.Intersplunk.outputResults(newresults)

The above is probably the most common use-case for what you are trying to do; changing events with your own command. You do not necessarily have to return the entire original set of events. You can return any key-value pairs back to Splunk. The following is absolutely legitimate:

# This prepares the return value for the script
newresults = [ { "afterglowFilename" : "afterglow.html" } ]

splunk.Intersplunk.outputResults(newresults)

This example returns only one key/value pair. This could then be combined with a field action to execute some action on this field, for example displaying the html file indicated in the value part.

REST API

The Splunk REST API is new for 3.0. It allows access to Splunk features from your custom application through a HTTP request. It returns an XML document that can be parsed for the desired information such as configuration values or search results. Splunk 3 uses a REST endpoint to communicate with its browser client. External developers can, and are encouraged to, use this service to integrate Splunk with existing infrastructure. REST is based on the standard HTTP protocol, making it accessible from any programming environment that supports HTTP requests.

The SOAP API from previous versions will continue to be supported for existing functionality until Splunk 3.1, but for new applications we encourage use of REST instead. Note that several features, such as user management, are not complete. For these functions, continue to use SOAP until they are available in the REST API. While calls to the SOAP and REST interfaces will not be directly compatible, for the remaining features there will be some commonality that is intended to ease the transition to REST. Also, an XML API will be available in a forthcoming release.

To see the endpoint reference, from a supported browser login to a 3.x Splunk server and then go to http://yourhost:8000/v3. This document contains a list of all valid endpoints along with documentation on required arguments and test forms to try various options. You can use the test forms to create sample requests to use as a model for accessing the API from your own application.

Note that v3/controlapi, direct access to all the methods in control_api and the Python Control Layer, is not intended for production use. The controlapi interface is expected to change frequently. You can enable or disable access to controlapi with enableRestControlApi in web.conf.

Messages

The REST API will return any messages (info, warnings, errors) in an 'messages' node. If there are no messages, then no node will appear. Your application should check for the presence of this node and handle accordingly. For example:

<envelope>
   <messages>
      <msg type="ERROR">The apikey is invalid</msg>
      <msg type="PERSISTENT">Your license has expired</msg>
      <msg type="INFO">Today is your lucky day</msg>
   </messages>
   <somethingElse>
   </somethingElse>
</envelope>

Possible message types are:

Authentication

Most requests require authentication. The standard authentication method is through the /auth/getToken method. Provide the username and password in the URL and then parse the returned XML document for the authToken element:

http://localhost:8000/v3/auth/getToken?usr=admin&pwd=changeme
<envelope>
  <authToken>f0913e8bf5c1c077df0cc18df21b1586</authToken>
</envelope>

The authToken string is then specified as an apikey argument in subsequent calls, like this:

http://localhost:8000/v3/alerts/get?id=34&apikey=f0913e8bf5c1c077df0cc18df21b1586

You can also authenticate from your browser with a valid login (a cookie containing your authToken is created with a port number to identify different Splunk instances, like "SPLUNK_SESSION_8000". When you login to Splunk from your browser and then go to v3, you are using the same session and do not need to specify an apikey argument.

Search

Construct a search request with a search string, output format and your authToken. Note that when you explicitly construct a search this way you have to provide terms that are normally handled for you in the Splunk UI, like "search". Also, whitespace and punctuation must be correctly encoded for use in a URI.

To get results from your search, pipe it to a command that formats it in the desired format, either outputxml outputraw, or outputcsv.

The search

http://localhost:8000/v3/splunk/search?q=search%20sudo%20guest%20%7C%20outputxml&apikey=2603471463dd923c4278607226070485

returns an XML document containing the results. For no results, it looks like this:

<envelope>
    <searchResults timestamp="07/03/2007:13:22:14">
<command>search sudo guest | outputxml</command>
<results type="empty"/>
</searchResults>
</envelope>

Enabling/disabling REST configuration options

Various options for the REST API can be controlled by the web.conf configuration file.

twistedLoginTimeout
timeout value for Twisted

restApiPostMode
Certain REST endpoint should be accessed via HTTP POST, instead of HTTP GET. By default, the REST endpoint will log a warning in the web_service.log whenever an endpoint that recommends a POST operation is accessed via GET. You can change this setting with restApiPostMode.

none = does not check; GET and POST are allowed
permissive = warns if REST method should be accessed via POST method
strict = throws error if POST-only REST method is accessed via GET

enableRestControlApi
true to enable the controlapi methods, default is false

Examples

The following Javascript excerpts are from an application that requests an auth token and then retrieves a list of the user's saved searches. Note that it is using XMLHttpRequest and parsing the returned document with standard XML DOM (Document Object Model) methods.

First, get a valid auth token:

var feedAuth = {url:"http://localhost:8000/v2/auth/getToken?usr=admin&pwd=changeme"};    

    xml_auth_request = new XMLHttpRequest();
    xml_auth_request.overrideMimeType("text/xml");
    xml_auth_request.open("GET", feedAuth.url, true);
    xml_auth_request.onreadystatechange=processRequest;    
    xml_auth_request.setRequestHeader("Cache-Control", "no-cache");
    xml_auth_request.send(null);
function processRequest()
{
    if(xml_auth_request.readyState==4)
    {
        if(xml_auth_request.status==200)
        {
            // 200 means we got a good response
            // extract auth token
            
            var doc = xml_auth_request.responseXML;
            var authTokenElements = doc.getElementsByTagName("authToken");

            // extract the auth token
            if (authTokenElements && authTokenElements.length) 
            {
                var authTokenItem = authTokenEls.item(0);
                if (authTokenItem.firstChild) 
                {
                    authToken = authTokenItem.firstChild.data;
                }
            }

With the auth token, you can use it to request the list of saved searches:

var feedSaved = {url:"http://localhost:8000/v2/saved/all"};        

            xml_request.open("GET", feedSaved.url + "?apikey=" + authToken);

Python example

Note Splunk requires a number of environment variables to be set, such as SPLUNK_HOME, and the correct PATH to be able to locate splunk files. Source the file setSplunkEnv in the splunk bin directory (usually /opt/bin/splunk) to set up the necessary environment. As long as the environment is correct, you can run your Python code from anywhere on your filesystem.

source /opt/splunk/bin/setSplunkEnv

import urllib as net
import xml.dom.minidom as xml

#
# set variables
#
endpoint = 'http://localhost:8000'
username = 'admin'
password = 'changeme'

mysearch = 'error'
numberOfResults = 5
outputAsCsv = False

# get authToken, via POST
# this makes a request to: http://localhost:8000/v3/auth/getToken?usr=admin&pwd=changeme
#
authMethod = endpoint + '/v3/auth/getToken'
authData = {'usr': username, 'pwd': password}
authResponse = net.urlopen(authMethod, net.urlencode(authData))
xmlDoc = xml.parse(authResponse)
tokenElements = xmlDoc.getElementsByTagName('authToken')
if not tokenElements:
    print 'No auth token found!'

authToken = tokenElements[0].firstChild.nodeValue
print 'Got authToken=%s' % authToken

# make search request, via GET
# this makes a request to: http://localhost:8000/v3/splunk/search?q=<mysearch>&apikey=<authToken>
#
searchMethod = endpoint + '/v3/splunk/search'
searchArgs = {
    'q': 'page 0-%d 50000 [search %s] | outputxml' % (numberOfResults, mysearch),
    'apikey': authToken
}
if outputAsCsv:
    searchArgs['clientformat'] = 'csv'

searchResponse = net.urlopen(searchMethod + '?' + net.urlencode(searchArgs))
print ''.join(searchResponse.readlines())

SOAP API

The Splunk SOAP API is being replaced with an XML API starting with 3.1. This documentation is being maintained to support existing SOAP integrations. For future use, please see either the REST API (new for 3.0) or look for the forthcoming XML API.

How Splunk uses SOAP

Splunk's SOAP API is how the splunkweb process communicates with splunkd process. Anything you can do through Splunk's Web interface, you can do through SOAP. You can make calls directly by building complete SOAP messages via the front-end of your choice, or use the Python Control Layer interface.

Python Control Layer

The Python Control Layer (PCL) allows you to send SOAP requests from your Python code without having to generate the full call yourself. You can still use SOAP directly, but many operations are available in the PCL and more will be added in the future. All of the data input methods in the PCL take a (more or less) standardized Python dictionary object and, in most cases, return a dictionary as well.

As for the methods themselves, they follow the convention:

def <module>_<operation>(args, fromCLI)

where <module> is the input module (e.g. "tail" or "fifo") and <operation> is the operation that is to be performed with the module (e.g. "addFile" or "list"). args is a Python dictionary object. fromCLI is a boolean variable that is set to True if the method is called from the command line tool, but is otherwise defaulted to False for calls from other consumers, such as the search interface.

The PCL Data Input Methods can be found in the control_api.py module, located in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib. To use this module, import the following into your Python code:

control_api.py
cli_common.py
control_exceptions.py

control_api.py defines most of the interface, cli_common.py contains authorization functions and control_exceptions.py contains the exceptions used by the PCL. This directory also contains many additional source files that actually implement the interface. All the Python code that Splunk uses is in these files for you to examine. As more functionality is folded into the PCL, methods will be appropriately exposed through control_api.py.

You can find the list of supported commands in cli.py (search for "cList" around line 250.) This file defines the interface exactly as Splunk uses it, so this is always the most accurate source for information. The list looks like this:

cList.addCmd(SplunkCmd("add",    "user",             user_add, "username" , proReq = True))
cList.addCmd(SplunkCmd("add",    "fifo",             fifo_add, "source"))

The argument "proReq = True" indicates that this function is only available with a Splunk enterprise license. Any request to a Splunk server with an enterprise license must be authenticated by passing a valid authentication token with the request. Additional details can be found by looking at the function definitions elsewhere in the file or the implementations in the *.py files.

All methods take a dictionary (set of key-value pairs). Methods usually have one or more required key-value pairs and zero or more optional key-value pairs. All keys are required to be in lower-case. There is no case restriction on the values. If a module is unable to find a required key-value pair in the incoming dictionary, an exception will be raised. If the method finds a key-value pair that is not in its set of required or optional key-values, an exception will be raised.

On error the methods will raise exceptions, so calls to the PCL should be wrapped in try/except blocks for error handling purposes.
Most methods will return a dictionary on completion. In most cases, an empty dictionary on return signifies a successful call. In certain cases, the dictionary will contain a "serverRestartReq" key-value signifying that splunkd must be restarted for changes to take effect.

Environment

Splunk requires a number of environment variables to be set, such as SPLUNK_HOME, and the correct PATH to be able to locate splunk files. Source the file setSplunkEnv in the splunk bin directory (usually /opt/bin/splunk) to set up the necessary environment. As long as the environment is correct, you can run your Python code from anywhere on your filesystem.

source /opt/splunk/bin/setSplunkEnv

Authentication

Every call requires an authentication token be passed in the dictionary, even for the free server which does not support users or authentication. In this situation, use a dictionary with an empty string for the authstr.

All requests to a Splunk server with an enterprise license must be authenticated, otherwise they will fail. To do this, first call getAuthInfo with a valid username and password. The server responds with an authentication token to be used for subsequent requests. This token is good until you submit a logout request or the next time splunkd is restarted.

Example

This is a minimal example of using the PCL to generate a SOAP call. It has no authentication or error handling but shows the basic structure of a PCL call (in this case to a free server.) It generates a SOAP request for function tail_list and prints the returned dictionary containing a list of all the tails currently configured in the server. Note that it calls the version of Python supplied with Splunk and not any that may be installed on the system. Use Splunk's version of Python to avoid problems with incompatible versions.

#!/opt/splunk/bin/python -u

import splunk.clilib.control_api as ca

argsDict = {
  "authstr" : ""
}

results = ca.tail_list(argsDict)
print results

The output looks like this:

{'dirs': ['/var/log/httpd'], 'dynamic': ['/var/log/httpd/error_log', '/var/log/httpd/access_log'], 'static': ['/var/log/system.log']}

For this minimal configuration, you can see there is one directory (/var/log/httpd) that contains two files (error_log and access_log) and one additional file specified by name (system.log.)

See the following page for more complete examples of making PCL calls from your Python code.

Using the PCL to Make SOAP Requests

Authentication Example

Below is a more complete example of the list_tails function. It authenticates against a Splunk server with an enterprise license, which could be elsewhere on the network, passes the authentication token with the list_tails request, and prints the contents of the returned dictionary.

#!/opt/splunk/bin/python -u

# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm

# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions

# for exit()
import sys

# optionally connect to another server at hostname:port
# 8089 is the default Splunk SOAP port
# comm.setURI("https://testserver:8089")

try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")

except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)

# make a dictionary with the auth token
argsDict = {
  "authstr" : authString
}

# PCL call, returns list of tails
results = ca.tail_list(argsDict)

# output the results
for key, values in results.items():
  print "%s: %s" % (key, values)

Run against a Splunk server with an enterprise license, the output looks like this:

dirs: ['/var/log/httpd']
dynamic: ['/var/log/httpd/error_log', '/var/log/httpd/access_log']
static: ['/var/log/system.log']

If this example is run against a free server, an exception is raised because it can't handle the call to getAuthInfo.

Traceback (most recent call last):
  File "./listTails.py", line 18, in ?
    authString = comm.getAuthInfo("admin", "changeme")
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 135, in getAuthInfo
    retStr = callAPI(block)
  File "/opt/splunk/lib/python2.4/site-packages/splunk/clilib/cli_common.py", line 180, in callAPI
    raise InvokeAPI, str(root.documentElement.firstChild.data)
splunk.clilib.control_exceptions.InvokeAPI: 'badparams'

The message "badparams" is a generic error. In this case it indicates that the requested operation, authentication against a free server, does not exist.

Search Example

Here is an example of a search. The key "terms" is the same as what you would type in the Splunk Box to do a search. For more on what you can specify here, see the Language Reference. 

#!/opt/splunk/bin/python -u

# splunk api
import splunk.clilib.control_api as ca
import splunk.clilib.cli_common  as comm

# splunk exception handling
# use "from ... import ..." to avoid namespace problems
from splunk.clilib import control_exceptions

# for exit()
import sys

try:
# send the username/password, get back the auth token
  authString = comm.getAuthInfo("admin", "changeme")

except control_exceptions.AuthError:
  print "Authentication failed."
  sys.exit(1)

# make a dictionary with the auth token and the search string
argsDict = {
  "authstr" : authString,
  "terms"   : "404"
}

results = ca.search(argsDict)
print results

The output when run against the same minimal Splunk server with an enterprise license is this:

127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302

To get the full event, including metadata, specify you want XML by adding "format" to the dictionary:

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "format"  : "xml"
}

Now the result is this:

<queryResult><ids>
</ids>
<results type="events"> <result cd="0:2891"><segtext xml:space="preserve">127.0.0.1 - - [07/Feb/2007:10:28:33 -0800] "GET /favicon.ico HTTP/1.1" 404 302<meta></meta></segtext>               <timestamp>1170872913</timestamp>               <source cd="2">/var/log/httpd/access_log</source>
                <host cd="1" name="test.splunk.com" >                 <tags></tags>
</host>         <sourcetype cd="2" base="too_small">too_small</sourcetype>
                <type cd="9" wob=" v:226e b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

For more about the search function, you can look at the Python code in the file searchizzle.py. The function definition begins like this:

def search(args, fromCLI):
  paramsReq = ("terms", "authstr")
  paramsOpt = ("output", "format", "get")

which shows the required and optional parameters. "output" is the output type, the default is "splunkui" (and this is almost always the correct choice.) "format" was mentioned above, and "get" is the type of data to return.

The default for get is "events", to return matching events. You could also specify "hosts" to return a list of hosts with matching events and so on. The list of possible values is defined in validGetTypes a few lines below:

  validGetTypes     = ("events", "hosts", "sources", "sourcetypes", "types")

Additional PCL Search Examples

The search function has a number of optional parameters for determining what results are returned. In addition to the standard event results discussed in the previous example, here are some other ways you can use the PCL search function.

output

For a short report of selected matching event statistics, use "output":

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "output"  : "scheduler"
}

If a regular search returns these events:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /test.html HTTP/1.1" 404 305
127.0.0.1 - - [27/Feb/2007:14:00:50 -0800] "GET /favicon.ico HTTP/1.1" 404 307
127.0.0.1 - - [26/Feb/2007:16:16:35 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:40:27 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:37:04 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:29:29 -0800] "GET /test.html HTTP/1.1" 404 300
127.0.0.1 - - [26/Feb/2007:15:27:07 -0800] "GET /favicon.ico HTTP/1.1" 404 302

Then specifying an output type of "scheduler" returns this:

eventCount:          7
hostCount:           1
sourceCount:         1
typeCount:           2
sourceTypeCount:     1
eventTagCount:       0
hostTagCount:        0
starttime:           12/31/1969:16:00:00
endtime:             03/06/2009:12:46:17

get

Use "get" to return a list of hosts, sources, sourcetypes or types that contain matching events. The dictionary shown below returns a list of the event types of the matching events.

argsDict = {
  "authstr" : authString,
  "terms"   : "404",
  "get"     : "types"
}

returns

eventtype::?9
eventtype::?11

With any of these, you can also specify a format of "xml" to get the output in XML with additional information:

<queryResult><ids>
</ids>
<results type="types">  <result c="5" cd="9"><segtext xml:space="preserve">5 results from eventtype::?9<meta></meta></segtext>          <type cd="9" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:0  ">
                        <tags></tags>
</type>
</result>       <result c="2" cd="11"><segtext xml:space="preserve">2 results from eventtype::?11<meta></meta></segtext>                <type cd="11" wob=" v:cc9b b1:110 a1:49 j1:4111519 k2:529728502 h2:2550214376 g1:2550123436  ">
                        <tags></tags>
</type>
</result></results></queryResult>

Creating SOAP Requests

Observing PCL SOAP Calls

You can see exactly how Splunk creates SOAP calls with a small change to the existing PCL code to output the data sent and received. Use these examples to generate your own SOAP messages from the front-end of your choice and parse the data returned from splunkd for your application.

1. Open $SPLUNK_HOME/lib/python2.4/site-packages/splunk/clilib/cli_common.py
2. Find the following block in the callAPI function:

  try:
    if use_rpc == True:
      retStr = rpc(data)
    else:
      retStr = server.invokeAPI(apiData=data)
  except socket.error:
    raise SOAPConnectionException, "Could not connect to splunk server.  Please ensure that splunkd is running."

3. After the else: line, add (with 6 leading spaces):

      print "BEGIN SOAP CALL:\n%s\nEND SOAP CALL" % data

4. After the SOAPConnectionException line, add (with 2 leading spaces):

  print "BEGIN RETURNED XML:\n%s\nEND RETURNED XML" % retStr

5. Save the file
6. Run a few commands from the GUI to see what is being sent. The results can be found in $SPLUNK_HOME/var/log/splunk/web_access.log. If you prefer to send commands from the CLI, the output goes to stdout.

Remember to remove the added code later to avoid filling this file wtih SOAP calls.

Creating a SOAP Call

Here is an example of search via SOAP:

1. Make a request to https://localhost:8089/ (replace with your hostname:port) to generate an authentication token (valid for as long as you leave splunkd running, or until you switch authentication mechanisms):

<call name="userLogin"><params><login><username>admin</username><password>changeme</password></login></params></call>

This returns an authentication token, which you must include in your search call:

<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>

2. Now run the search call, with a few important notes:

<call name="executeQuery"><params><query>SEARCH  meta::all GET events::0-2 
OUTPUT splunkui::2.1 
format::raw</query><user>livesplunkuser</user><queryId>37627661733490256838</queryId>
<auth>
  <userId>1</userId>
  <username>admin</username>
  <authToken>3105802749</authToken>
</auth>
</params></call>

This results in the following xml (3 results, one in each <result> block),
which you can parse as you wish:

<queryResult>
<ids>
</ids>
<results type="events"><result cd="0:873197"><segtext xml:space="preserve">Feb 
19 14:25:49 VeeAte last message repeated 1964 times<meta><sg 
c="4003720337">format::l1_::_</sg> </meta></segtext>                
<timestamp>1171923949</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873110"><segtext xml:space="preserve">Feb 19 14:24:49 VeeAte 
last message repeated 1956 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923889</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
<result cd="0:873023"><segtext xml:space="preserve">Feb 19 14:23:49 VeeAte 
last message repeated 1951 times<meta><sg c="4003720337">format::l1_::_</sg> 
</meta></segtext>               <timestamp>1171923829</timestamp>
                <source cd="1">/var/log/messages</source>
                <host cd="1" name="veeate" ><tags></tags></host>
                <sourcetype cd="1" base="syslog">syslog</sourcetype>
                <type cd="178" wob=" v:2af8 b1:97 a1:49 j1:1122 k2:5772 
h2:2550729191 g1:54764362 l2:0  ">
<tags></tags></type>
</result>
</results><performance>Queryid=3263159498132591711719240037627661733490256838 
user=livesplunkuser result="success" results_returned=9774 
submitted=02/19/2007:14:28:38 time_between_submission_and_execution=0.000 
execution_time=0.020 total_time=0.020</performance>
<query>SEARCH meta::all GET events::0-2 OUTPUT splunkui::2.1 
format::raw</query>
</queryResult>

executeQuery.pl

Here is a perl script that executes SOAP calls:
Note This requires XML::XPath and SOAP::Lite

#!/usr/bin/perl

use XML::XPath;
use SOAP::Lite;

$| = 1;

# URL to Splunk Server Management Port
$splunkd = "https://defiant.splunk.com:8089";

# Login to splunk instance supplied by argument and return auth token.
sub Splunk_SOAP_login {
    my ($url) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);

    $apiData = "<call name=\"userLogin\"><params><login><username>admin</username><password>changeme</password></login></params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    $authToken = $s->invokeAPI($call) -> result;
    return $authToken;
}

## Execute the SOAP invoke api call and return the result
sub Splunk_SOAP_invokeAPI {
    my ($url, $authToken, $callname, $data) = @_;
    $s = SOAP::Lite
        -> uri('urn:m2c-ManagementService')
        -> proxy($url);
    $apiData = "<call name=\"" . $callname . "\"><params>" . $authToken . $data . "</params></call>";
    $call = SOAP::Data->name('apiData' => $apiData);
    return $s->invokeAPI($call) -> result;
}

# Helper function to build the data portion of a query call.
sub buildQuery {
    my ($searchString) = @_;
    $qdata = "<query>SEARCH $searchString GET events::0-9 OUTPUT maxlines::14 splunkui::2.1 summary::2.1 format::all "timeformat::%m/%d/%Y, %H:%M:%S"</query><user>admin</user><queryId>1234567890</queryId>"; 

    return $qdata;
}

# Obtain an auth token.
$authToken = Splunk_SOAP_login($splunkd);
print("authToken: $authToken\n");

# Build the API call.

# change this to send a different call
# some possible choices are:
#     getLicenseInfo
#     getUserInfo
#     getHostname
#     deleteUser
$callname = "executeQuery";

# for queries, this is the search term
$term = "error";

$query = buildQuery($term);

# Make the call and print results.
$results = Splunk_SOAP_invokeAPI($splunkd, $authToken, $callname, $query);
print $results;

SOAP Message Examples

Here are examples of a full SOAP message, including the XML, SOAP body and envelope.

Login

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 642
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <invokeAPI xmlns="urn:m2c-ManagementService">
            <apiData xsi:type="xsd:string"><call
            name="userLogin"><params><login><username>admin<
            /username><password>changeme</password></login><
        /    params></call></apiData>
        </invokeAPI>
    </soap:Body>
</soap:Envelope>

Execute Query

POST https://defiant.splunk.com:18089 HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 925
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:m2c-ManagementService#invokeAPI"

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <invokeAPI xmlns="urn:m2c-ManagementService">
            <apiData xsi:type="xsd:string"><call name="executeQuery"><params>
            <auth><userId>1</userId><username>admin</username>
            <authToken>3135052749</authToken></auth>
            <query>SEARCH error GET events::0-9 OUTPUT maxlines::14
            splunkui::2.1 summary::2.1 format::all "timeformat::%m/%d/%Y,
            %H:%M:%S"</query><user>admin</user><
            queryId>1234567890</queryId></params></call>
            </apiData>
        </invokeAPI>
    </soap:Body>
</soap:Envelope>

SOAP error messages

A SOAP error message can be returned for any SOAP call. There are 3 error messages.

1.

Request Denied. You must provide correct credentials to perform this action.

You have a valid login but the action you just tried is beyond your permission level.

2. 

User Session is not valid.

You provided a session token but it didn't match.

3.

No authentication provided for call.

You provided no auth on the call at all.

Data Inputs

Data Inputs

This page is still under development.

Input config

Scripted inputs

By configuring inputs.conf, Splunk can also accept events from scripts.

Scripted input is useful for and command-line tools, such as vmstat, iostat, netstat, top, etc.

Please note: currently, scripted inputs do not get bundled in the deployment server. In the future, Splunk will support this behavior. For now, please use your preferred configuration automation tool to push your script directory to your server classes.

Configuration

Please note: your script must be in the bin/ directory underneath your scripts/ directory.

[script://$SCRIPT] 
interval = X 
index = {main, $YOUR_INDEX}
sourcetype = {iostat, vmstat, etc}  OPTIONAL
source = {iostat, vmstat, etc} OPTIONAL
disabled = false

Variables:

Example

This example shows the use of the UNIX top command as a data input source.

$ mkdir $SPLUNK_HOME/etc/bundles/scripts
$ #!/bin/sh
 top -bn 1  # linux only - different OSes have different paramaters
chmod +x $SPLUNK_HOME/etc/bundles/scripts/bin/top.sh
$SPLUNK_HOME/etc/bundles/scripts/bin/top.sh
[script:///opt/splunk/etc/bundles/scripts/bin/top.sh]
interval = 5                # run every 5 seconds
sourcetype = top        # set sourcetype to top
source = script://./bin/top.sh   # set source to name of script

Please note:

[top]
BREAK_ONLY_BEFORE = GobblyGook
DATETIME_CONFIG = CURRENT

Understanding modules and processors

Default processors and pipelines

The universal pipeline, or the parsing pipeline, is where events are input, processed and output to the indexing pipeline.

Modules, pipelines and queues

Below is a list of the processors, in order executed, that make up the default universal pipeline. You can see this in the pipeline "parsingPipeline" in $SPLUNK_HOME/etc/myinstall/splunkd.xml file.

readerIn
Queue Input processor, data comes in here
utf8
UTF8 processor
linebreaker
Line Breaking
aggregator
Line Merging/Date Extraction
regexreplacement
Regex Extraction
typing
Event Typing
clusterer
Meta Event Creation
sendOut
Queue Output, data sent to next pipeline

Pipeline data keys

Changing the default parsing and indexing sequence

The processors that make up what is called the Universal pipeline can be reconfigured or replaced by creating a new module with your custom processing pipeline.

The Splunk processor loading architecture supports the ability to insert a processor before, after or instead of another processor. If you are extending splunk by providing your own processor or wish to change the processing pipelines defined in splunkd.xml it is recommended that you define your processor in a new module and use the "insertBefore", "insertAfter", or "replace" attribute.

For example, lets suppose you wanted to add a processor "replaceProcessor" just before the "indexer" processor:

  1. Create a new module directory in $SPLUNK_HOME/etc/modules/replaceProcessor
  2. Create a config.xml file in the new module directory
  3. Define a module configuration with a list of processors - note that for this case you do not use the <pipeline> ... </pipeline> tags because you are not defining a new pipeline but changing an existing one.
  4. Define your processor(s) in the config.xml file and specify the action and target and target pipeline.
    • action is a value : "insertBefore", "insertAfter" or "replace"
      • target is the name of processor to insert before/after or replace
        • pipelineTarget is the pipeline where the inserted or replaced processor exists.

Here is the config.xml for an example replaceProcessor (uses an example urlencodeprocessor.) It will insert the "replaceProcessor" before the "indexer" processor in the "indexerPipe" pipeline:

<module>
   <processor name="replaceProcessor" plugin="urlencodeProcessor" action="insertBefore" target="indexer" pipelineTarget="indexerPipe" >
       <config></config>
    </processor>
</module>

Adding custom processors

Custom processors can provide data input and handling not available by other methods. You can add to the existing default processing or replace it by changing the configuration of Splunk pipelines.

Coding C/C++ processors

Processors allow for custom actions that cannot be done by configuration alone. But note that many things that required a processor in previous versions are now supported without custom code.

Developing C/C++ processors for the Splunk server follows a simple development model where a user supplies a function/method that gets called by splunkd for every event processed. A data structure is passed to this function that contains the event data and meta data. Helper functions are provided to manipulate portions of the data structure such as setting host, adding meta data, etc.

There are 3 things that need to be done to build and hook up your own processor:

  1. Write a callback function
  2. Write a program with a main to register your function
  3. Add a config stanza that specifies your external processor

Below is a brief overview of each of the three steps. More complete examples can be found on on the Processor C Example and Processor C++ Example pages.

Writing Your Callback Function

The first thing to do is to implement a function with the following signature, this function will be called once for each event during processing. The actual name of the function need not be processEvent, it can be called anything you like.

     static int processEvent( struct SplunkTransaction *trans);

You will need to #include the header file splunk-extplugin.h located in SPLUNK_HOME/include.
An online version is on the splunk-extplugin.h page.

This header file defines helper functions that can be used to easily and safely manipulate the event data. There are several types of helper functions:

Get/Set Functions

These are used for manipulating the event and event meta data such as Host, Source, Sourcetype, the raw data, etc. Below are some of these functions, more can be found in the header file.

extern const char *SplunkGetSource(struct SplunkTransaction *trans);
extern void SplunkSetSource(struct SplunkTransaction *trans, const char *nval);

extern const char *SplunkGetSourceType(struct SplunkTransaction *trans);
extern void SplunkSetSourceType(struct SplunkTransaction *trans, const char *nval);

Logging Functions

The following functions provide logging facilities to the Splunk logs located in SPLUNK_HOME/var/log/splunk/splunkd.log.

extern void SplunkDebug(const char *fmt, ...)
extern void SplunkInfo(const char *fmt, ...)
extern void SplunkWarn(const char *fmt, ...)
extern void SplunkError(const char *fmt, ...)
extern void SplunkFatal(const char *fmt, ...)

Start splunk with the --debug option to see messages generated with the SplunkDebug function.

Configuration Functions

The following function provides key/value access to the tags in the external processes xml config (see below.)

extern const char *SplunkInstanceConfig(const char *key);

Error and Process Management

The following function provides a clean way to exit your custom processor.

extern void SplunkExit(int rval, const char *fmt, ...)

Write Main

The external processor is a separate application that is invoked during the Splunk startup process. The main() function is called, allowing for initialization and the hooking up of the event processing function described above.

The following example is the main function for the processor that just hooks up the event processing function. The function SplunkProcess_SingleThreaded registers your function so splunkd knows how to call it.

int main(int argn, char * const argv[])
{
    SplunkProcess_SingleThreaded(processEvent);
}

Note If you run your program from the command line, you will see the message "I/O Error -- error writing: Socket operation on non-socket." This is normal.

Add a Processor Config Stanza to Pipeline

To get your processor called, you must add it to a pipeline. Create a new module and pipeline for your processer and insert the XML for your processor in your module's config.xml.

The following XML stanza defines the external processor:

NOTE Replace the path in <command> ... </command> with the actual path to your executable, it can be anywhere but we recommend putting it in your custom module's directory. If your program requires command-line arguments, you can include them here although a better way to handle this is through config.xml. (See the complete examples for more on this.) extcmdprocessor is the built-in processor that handles custom processors.

This example inserts sampleProcessor into the main indexing pipeline. This means it will execute for every event Splunk indexes, including internal ones. If you want to only process some events, create a new pipeline as shown in the Using a Bundle with a Module example.

<processor name="extcmdprocessor" plugin="extcmdprocessor" action="insertAfter" target="indexIn" pipelineTarget="indexerPipe" >
      <config>
       <command>$$SPLUNK_HOME]]/etc/modules/myModule/sampleProcessor</command>  <!--  $$SPLUNK_HOME]] references your splunk installation directory -->
      </config>
     </processor>

Restart splunk to load your module and register your processor with splunkd. Errors, warnings and your log messages can be found in the splunkd.log file. Note that if you have output any messages using SplunkDebug(), you must run Splunk with "splunk start --debug" to have them generated. In the splunkd.log file you will also see messages like this indicating your module has been loaded:

02-20-2007 09:27:41.846 INFO loader - Processing Module ----> /opt/splunk/etc/modules/myModule
02-20-2007 09:27:41.851 INFO loader - Overlaying processors
02-20-2007 09:27:41.851 INFO loader - Overlaying processor extcmdprocessor into pipeline. Target=indexIn Action=insertAfter
02-20-2007 09:27:41.851 INFO loader - processor extcmdprocessor overlay completed
02-20-2007 09:27:41.854 INFO splunklogger - Module /opt/splunk/etc/modules/myModule loaded

Complete C Example

Before building, you should first set up your build environment. An easy way is to source the script $SPLUNK_HOME/bin/setSplunkEnv. This is the same environment for running the Splunk server, so the same shared libraries can be found at runtime.

> source /opt/splunk/bin/setSplunkEnv

The following is sample processor written in C and just adds some text to the end of the event.
It can easily be extended using the same pattern to process other attributes such as source, host, sourctype, etc.

1) compile it using

gcc -o sampleProcessor -I$SPLUNK_HOME/include -L$SPLUNK_HOME/lib sampleProcessor.c $SPLUNK_HOME/lib/libextcmdapi

2) copy the compiled binary to a convenient location, such as your module's directory under etc/modules
3) add the XML config section and restart

Configure the text string to append by adding this XML after your <command>:

<addToRaw>All your log are belong to Splunk</addToRaw>

Complete C Code

//  Sample Processor

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>

#include "splunk-extplugin.h"

// The following are used to hold config values picked up dring initialization
char *addToRaw;
char defaultAddToRaw[] = "No addToRaw specified in config file";

//-----------------------------------------------
// Function is called to mark up the raw data
//------------------------------------------------
void fixRaw(struct SplunkTransaction *trans)
{
  char *raw;
  char *newraw;

  /* get the raw data */
  /* SplunkGetRaw returns const char *, remember to play nice with it */
  raw = (char *)SplunkGetRaw(trans);
  SplunkDebug( "RAW = \"%s\"\n", raw );
 
 
  /* allocate memory for the data plus additional text */
  newraw = malloc(strlen(raw) + strlen(addToRaw));
  if (newraw == NULL)
  {
    SplunkExit(1, "out of memory");
  }
  
  /* Append text to all of the log lines */
  strcpy(newraw, raw);
  strcat(newraw, addToRaw);
  
  // Put the data back
  SplunkSetRaw(trans, newraw);
  SplunkDebug( "NEW RAW = \"%s\"\n", newraw );

  free(newraw);
}

//---------------------------------------------------------
// This function is called for each event during processing 
//---------------------------------------------------------
static int processEvent(struct SplunkTransaction *trans)
{
    // Call our helper function to mark up the raw data
    fixRaw( trans );

    // return true to continue sending the event through pipeline
    return 1;  
}

//---------------------------------------------------------
//  Main is called once during initialization
//  We pull out config values declared in the xml config
//---------------------------------------------------------
int main(int argn, char * const argv[])
{  
  (void) argn;
  
  // arguments can be passed in on the <command> config line
  // it is more tidy to supply additonal config through their own xml tags ( see below )
  if (argv[1] != NULL)
  {
    // if we want pull out the args
  }

  // additional key values can be passed in through the config files.
  // These values will be used during processing each event. 
  addToRaw = (char *) SplunkInstanceConfig("addToRaw");
  if ( addToRaw == NULL )
  {

    addToRaw = malloc(strlen(defaultAddToRaw));
    strcpy(addToRaw, defaultAddToRaw);
    
    // If in Warn mode then log that no value was passed
    SplunkWarn( "No value for addToRaw specified in sample processors config" );
    
    // if we wanted we could exit
    //    SplunkExit(1, "No value for addToRaw in config");
  }

  // spit out a debug message with text to append
  SplunkDebug("Using \"%s\" to add to raw", addToRaw);
  
  // This will hook up the per event processing function
  SplunkProcess_SingleThreaded(processEvent);
}

Complete C++ Example

Before building, you should first set up your build environment. An easy way is to source the script $SPLUNK_HOME/bin/setSplunkEnv. This is the same environment for running the Splunk server, so the same shared libraries can be found at runtime.

> source /opt/splunk/bin/setSplunkEnv

The following is sample processor written in C++, it adds some text to the end of the event and creates two fields that can later be searched.
It can easily be extended using the same pattern to processor other attributes such as source, host, sourctype, etc.

1) compile it using

g++ -o sampleProcessor -I$SPLUNK_HOME/include -L$SPLUNK_HOME/lib sampleProcessor.cpp -lextcmdapi

2) copy the compiled binary to a convenient location, such as your module's directory under etc/modules
3) add the XML config section to your pipeline and restart

Configure the text string to append by adding this XML after your <command>:

<addToRaw>All your log are belong to Splunk</addToRaw>

Complete C++ Code

#include "splunk-extplugin.h"
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <errno.h>

//----------------------------------------------------------------
// Sample Processor class
//----------------------------------------------------------------
class SampleProcessor : public Splunk::SingleThreadedProcessor {
    std::string  addToRaw;   

    //----------------------------------------------------------------
    // Method to process the event
    // Modifying host, source, sourcetype, etc follow the same pattern 
    //----------------------------------------------------------------
    bool fixEvent(Splunk::Transaction *trans)
    {

        // get the raw data
        // SplunkGetRaw returns const char *, remember to play nice with it    
        const char *r = trans->getRaw();

        // get the metadata
        const char *m = trans->getMeta();

        // If in debug mode dump out the data
        SplunkDebug( "RAW = \"%s\"\n", r );

        std::string rs(r);
        std::string ms(m);

        // append the string to raw
        rs.append(addToRaw);

        // put the modified raw data back 
        trans->setRaw(rs);
        
       // append the field info to meta, space between items
       ms.append(" myfield1::red myfield2::purple");

       // put the modified metadata back
       trans->setMeta(ms);

        // return true to continue sending the event through pipeline
        return true;        
    }

    //----------------------------------------------------------------
    // This method is called once per event in the pipeline
    // it implements the virtual method from the base class
    //----------------------------------------------------------------
    bool handler(Splunk::Transaction *trans)
    {
        // call method to fix up the event
        return fixEvent(trans);
    }

    public:
    

    //----------------------------------------------------------------
    // Constructor pulls config values
    //----------------------------------------------------------------
    SampleProcessor(char * const args[])
    {
        // during initialization we try and read config data and cache result
        addToRaw.append(" {");

        // additional key values can be passed in through the config files.
        // These values will be used during processing each event. 
        const char *r = Splunk::InstanceConfig["addToRaw"];
        addToRaw.append((r == NULL) ? "UNKNOWN" : r);
        addToRaw.append("}");

        // spit out a debug message with text to append
        // must run with "splunk start --debug" to enable debug messages
        SplunkDebug("Using \"%s\" to add to raw", addToRaw.c_str() );

    }
};

//---------------------------------------------------------
//  Main is called once during initialization
//  We construct our processor class and call run
//---------------------------------------------------------
int main(int argn, char * const argv[])
{
    (void) argn;
    // if we had passed args in via the command tag we could pull them here.
    if (argv[1] == NULL);

    // construct our class and go
    SampleProcessor sp(argv);
    sp.run();

    return 1;
}

Selecting events to process with a bundle

You can create a bundle to configure your module, just as you can use bundles with Splunk standard modules. Below is an example using a bundle to only send certain events, 404 messages from an apache log file, to the custom module.

Create the Processor

This example uses the sample C++ module described on the previous page, here referred to as "sampleProcessor".

Create the Module

Make a new directory under $SPLUNK_HOME/etc/modules. For this example, it is called "testModule" and the sampleProcessor executable file goes there.

Next, create a new config.xml with a new pipeline and the desired processors. This pipeline calls "readerin", "sampleProcessor" and "sendOut". readerin is configured to get data from the queue "testQueue", sampleProcessor appends the text "{xyzzy}" to the raw event and sendOut passes it to indexQueue to continue to the indexer.

Note You can also specify the full path instead of using "$$SPLUNK_HOME]]"

<module>
  <pipeline name="test-pipeline" type="startup">

    <processor name="readerIn" plugin="queueinputprocessor">
      <config>
        <queueName>testQueue</queueName>
      </config>
    </processor>

    <processor name="myProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>xyzzy</addToRaw>
      </config>
    </processor>

    <processor name="sendOut" plugin="queueoutputprocessor">
      <config>
        <queueName>indexQueue</queueName>
      </config>
    </processor>

  </pipeline>
</module>

Create the Bundle

Make a new directory under $SPLUNK_HOME/etc/bundles, here also called testModule. In it, create three new files:

Note Replace testbox.splunk.com with your hostname and the tail path with your desired path.

inputs.conf specifies the files we want to tail and assigns a sourcetype.

host = testbox.splunk.com

[tail:///var/log/httpd]
disabled = false
host = testbox.splunk.com
sourcetype = access_log

props.conf specifies that events with sourcetype access_log should use the regular expression configured in regexes.conf for test-pipeline. "TRANSFORMS-test" specifies the class ("-test") to identify a particular configuration stanza. Class names must be unique for each Splunk instance to avoid overriding existing behavior.

[access_log]
TRANSFORMS-test = test-pipeline

transforms.conf defines test-pipeline to look for the specified regular expression and send those events to a queue named "testQueue".

[test-pipeline]
REGEX = \s404\s
DEST_KEY = queue
FORMAT = testQueue

Note that the new 3.x format is transforms.conf instead of regexes.conf and TRANSFORMS-classname for REGEXES-classname.The complete bundle looks like this:

testbox [root]:/opt/splunk/etc/bundles$ ls testModule
inputs.conf     props.conf      transforms.conf

Restart splunk to load the new pipeline. The 404 events should now show the specified text appended:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy}

Add an Additional Processor

You can configure several processors in the same pipeline. For this example, to add a second instance of sampleProcessor, add the XML for a second processor with a different name. It can reference the same executable as myProcessor.

    <processor name="myOtherProcessor" plugin="extcmdprocessor">
      <config>
        <command>$$SPLUNK_HOME]]/etc/modules/testModule/sampleProcessor</command>
        <addToRaw>You are in a maze of twisty little passages, all alike</addToRaw>
      </config>
    </processor>

Then, the text of the event would look like this:

127.0.0.1 - - [27/Feb/2007:16:02:31 -0800] "GET /foo.html HTTP/1.1" 404 305 {xyzzy} {You are in a maze of twisty little passages, all alike}

Data Outputs

Data Outputs

This page is still under development.

Customize alert options

Email alerts

Limited customization can be done via alert_actions.conf to specify the message subject and From: address used for alert emails.
Before making modifications to any configuration file, please see about bundles. See this page for details of the alert_actions.conf file.

[email]
# from email address
from=splunk@localhost

# subject of the email
subject=Splunk Results

If you need additional customization, you can edit sendemail.py in $SPLUNK_HOME/etc/searchscripts. This is called for each alert, must be named sendemail.py and will be overwritten on upgrade. Save a copy of your changes in a safe location for future reference.

Warning Keep a backup copy of the original version so you can revert in the event of a problem. Test your changes carefully. For more information on the Python language, see http://www.python.org.

The mail() function, called by splunkd for each alert email, formats and sends the message using arguments configured in alert_actions.conf and internal defaults. You can modify the location of the mail server or the contents of the message header by changing the value of the variables defined at the top of the function:

    serverURL = argvals.get("server", "localhost")
    sender    = argvals.get("from", "splunk@" + socket.gethostname())
    to        = argvals.get("to", None)
    cc        = argvals.get("cc", None)
    bcc       = argvals.get("bcc", None)
    subject   = argvals.get("subject", "Splunk Results")
    format    = argvals.get("format", "html")
    importance= argvals.get("priority", None)

If you are interested in changing the format of the email, note that the variable bodytext contains the text of the message body, including search results.

CLI for search

Note: this page has not been fully updated for 3.0.

The command-line search API supports the exact same syntax as the Splunk box, with additonal parameters.

Actions

Default Argument

Parameters

where range is n items returned from the full results. Example:

splunk search 404 -get sources::0-9

returns the first 10 sources from the specified search.

Example

splunk search -get hosts "smtp NOT success hoursago::1"

By default only 100 events are returned when a search is done from the CLI. This can be changed by adding maxresults:: to your search. For large searches, we recommend you use the "raw" output type to reduce memory usage.

splunk search -output rawevents "meta::all minutesago::120 maxresults::100000"

TCP Output

XML API for data output

An XML API will be available in an upcoming feature release. Please check back later for more details.

Management

Management

This page is still under development.

Authentication

Splunk Professional requires all commands to be authenticated. The API supports both per-command and per-session authentication.

Command Authentication

Individual commands can be authenticated by including an authentication username and password. One parameter can specify both user and password, but note that the --auth command must come last on the command line. If either the username or password aren't specified, the command line prompts the user.

Parameters

Examples

splunk search meta::all -auth admin:changeme
splunk search meta::all -auth admin

Session Authentication

Alternatively, users can use the "login" command to cache their auth credentials on the local filesystem securely. Auth tokens persist until a logout command is issued or the Splunk server is restarted.

Examples

splunk login
splunk logout

CLI for management

User Accounts

Splunk user accounts can be created and maintained through the CLI. An Enterprise license is required for user accounts.

Actions

Default Argument

Parameters

Examples

splunk add user -username algore -full-name "Al Gore" -password pa55word
splunk edit user algore -role user

Controls

Commands for sysadmins to start and stop Splunk Server processes, and to check the syntax of configuration files for errors.

Actions

Contexts

Default Argument

Parameters

--nodaemon only starts splunkd, start the GUI manually with {start splunk splunkweb} in another shell.

Examples

splunk start

splunk start --nodaemon



splunk restart splunkd
ipaq sell | SP-CAAAAJQ

splunk stop splunkweb

Configuring splunkd

Creating default logins on first startup

Splunk can create specific administrative accounts on first startup through the user-seed.conf file. You can create a user-seed.conf file in any bundle, but it will only be used the first time you start Splunk and ignored thereafter.

Specify the usernames and passwords to create in this form:

[user_info]
USERNAME = staff
PASSWORD = passw0rd

If users are created via user-seed.conf, the usual default admin account will not be created, only the ones specified. Any accounts created this way will have the Admin role.

Note You should remove this file after first startup, as the passwords are saved in plaintext.

Changing the SSL configuration

The SSL configuration is controlled by the file server.conf. To change the default settings, create a new server.conf file in a configuration bundle. You should create a new bundle rather than using the README directory for easier maintenance later.

An SSL configuration stanza begins with [sslConfig] and can contain the following options:

enableSplunkdSSL = <true|false>
Enables/Disables SSL on the splunkd management port.
enableSplunkSearchSSL = <true|false>
Enables/Disables SSL on the frontend/GUI.
keyfile = <string>
Server certificate file. One of these will be generated by splunkd on startup, you may also replace it with your own cert, PEM format, file.
keyfilePassword = <string>
Server certificate password.
caCertFile = cacert.pem
Public key of the signing authority.
caPath = $$SPLUNK_HOME/etc/auth
path where all these certs are stored.
certCreateScript = genSignedServerCert.sh
Creation script for generating certs on startup of splunk.

For additional information, see server.conf.spec and server.conf.example in $SPLUNK_HOME/etc/bundles/README.

If you enable SSL for the GUI with enableSplunkSearchSSL = true, then you will not be able to access Splunk without specifying https in your browser. Splunkweb will continue to use the same port number, by default 8000.

To disable the web interface entirely, go to the Server > Settings tab in the GUI and select No for "Run Splunk's web interface?" You can also change the port numbers from this page. Save and restart Splunk for your settings to take effect.

XML API for management

An XML API will be available in an upcoming feature release. Please check back later for more details.