• Overview
  • Basic Server Operations
    • Add Splunk to Your Shell Path
    • Start the Server
    • Check if Splunk is running
    • Stop the Server
    • Restart the Server
    • Index Your Data
    • Delete Data from the Index
    • Change Network Ports
    • Uninstall the Release
  • Users, Authentication & Access Control
  • What's Installed Where
    • Users & Groups
    • Files & Directories
    • Network Ports
    • Unix Processes
  • Automatic Update Notification
    • Disabling the "phone home" update check
  • Troubleshooting
    • composite.xml and compositelibmap.xml
    • Test Configuration Changes
    • Core Files
    • Debug mode
    • Built-In Tools
    • Alerts (Splunk Professional)
    • Indexing Performance
    • splunkd Monitor
• Setting Up Data Inputs
  • Web-based Configuration
    • Admin Interface
    • Guided Setup
    • Restarting the Server
    • Built-in Help
    • Configuration Files
  • Command Line Configuration
    • Data Input Configuration Commands
    • Tail Inputs
    • Watch Inputs
    • FIFO Inputs
    • Network Port (TCP, UDP) Inputs
    • Database Table (ODBC) Inputs
  • Configuration Files
    • Format
    • Input Types
    • Attributes
    • Additional Attributes
    • Examples
• Authentication
  • LDAP Authentication
    • Configure your Authentication Strategy
    • Configure Splunk to use your LDAP server
    • Determining your User and Group Base DN
    • Mapping existing LDAP groups to Splunk Roles
    • Testing your LDAP configuration
• Changing the Way Splunk Processes Your Data
  • Correct handling of multi-line events
  • Extract host from an event
  • Configure time zone offsets
  • Correct timestamp recognition
    • Training Splunk to Recognize Timestamps
    • Positional timestamp extraction
  • Extract additional meta data (e.g. user, severity) from events
  • Modify or delete segments within events
  • Summarize related events as a meta event
    • Transitive Meta Events
  • Auto-tag event types
  • Pre-process binary formats
  • Pre-process files with a specific extension
  • Long Lines and Large Events
  • Filter unwanted events before indexing
  • Index Specific Events
  • Send specific events to an alternate index
  • Strip syslog headers before processing
  • Turn off event typing
  • Turn off segment indexing
  • Turn off timestamp parsing
  • Change the characters that separate segments within events
  • Reduce indexing density
  • Setting dates in timestamps
  • How to include a date column in reports
  • Anonymize values
  • File whitelisting / blacklisting
    • Configuration
    • Example
    • Verification tool
  • Log file rotation
    • How log rotation works
  • Define Search-Time Report Fields
• Creating and Using Configuration Bundles
  • Configuration Files
    • How Bundles Work
    • Typical Uses
    • Example files
  • inputs.conf - Configure data inputs
  • props.conf - Add or override processing properties
  • regexes.conf - Create regular expressions for use in properties
• Configuring Your Sources
  • ftp, rcp, scp, sftp
  • log4j
  • How to enable Exchange message tracking
  • IIS logs
  • MonitorWare
  • NFS
  • SMB/CIFS
  • Snare
  • SNMP
  • syslog
  • syslog-ng
  • Data Base Access
  • Custom Inputs
• Splunk-2-Splunk Management
  • Forwarding and Receiving
    • How to configure receiving
    • How to configure forwarding
    • SSH tunnel forwarding / receiving
    • Redundant / High Availability configurations
  • Light Weight Splunk-2-Splunk forwarder Configuration
    • Configurations for light weight forwarder
  • Distributed Search
  • Blacklisting & Whitelisting Servers
  • Advanced Configurations
    • Store and Forward
    • Redundant Forking
    • Conditional Routing
  • Conditional Routing Example
    • Requirement
    • Splunk Configuration Tasks
    • Module Configuration
    • Bundle Configuration
• Integration
  • CA Unicenter NSM
  • Nagios
    • Splunk and Nagios on the same host
    • Splunk and Nagios on separate hosts
  • Splunk-2-Netcool
    • Integrated IT Data Search with IBM Netcool and Splunk
    • System Requirements
    • Installing Splunk-2-Netcool
    • Using Splunk-2-Netcool
    • Documentation, Help and Support
  • Splunk-2-Smarts
    • Integrated IT Data Search with EMC Smarts and Splunk
    • System Requirements
    • Receive Live Splunk Alerts in Smarts SAM console
    • Launch Splunk from Smarts SAM Console
    • Documentation, Help and Support
  • Secure Configuration
  • Chain of Evidence Management
  • Send an SNMP Trap from a Live Splunk
  • How to forward Splunk data to non-Splunk systems
• Managing Your Filesystem
  • Backups in 3 Easy Steps
  • Users and Splunks
    • Manage user accounts
    • Manage Saved Splunks
  • Disk Usage
    • Set a minimum free disk space
    • Remove files beyond a certain size
    • Remove files beyond a certain age
    • Recover free space
    • How to archive old data
• How-To's
  • Change the Default Time Range
  • Splunkd appears to be down
  • Splunk Data Anonymizer
  • HTTP GET login (versions 2.1.x and 2.2.x)
  • Public server setup
  • SSL setup for splunkd
  • Determing your daily volume
  • Disable Webserver Startup
  • Placing Splunk 2.2 behind a web proxy
    • Scenario:
    • Solution:

Basic Server Operations

Add Splunk to Your Shell Path

Setting a SPLUNK_HOME environment variable and adding $SPLUNK_HOME/bin to your shell's path will save a lot of typing in the future. The example below works for bash users who accepted the default installation location. Use the correct syntax and path for your own installation.

# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

Start the Server

From a Unix shell prompt on the sever host, run this command.

# splunk start

Additional flags can be passed when starting Splunk.
--answer-yes Answers yes to any y/n question
--no-prompt Quits when user would be prompted
--accept-license Will accept the license agreement on first time startup
--debug Turns on verbose logging

Check if Splunk is running

From a Unix shell prompt on the sever host, run this command.

# splunk status

You should see this output on stdout .

splunkd is running
splunkSearch is running

Or you can use ps to check for running Splunk processes:

# ps aux | grep splunk | grep -v grep

Solaris users, type -ef instead of aux:

# ps -ef | grep splunk | grep -v grep

You should see two of them, similar to the example entries below.

The splunkd process is the search and index engine.

root 5057 31.1 1.3 426212 21472 pts/11 Sl 14:05 0:02 /opt/splunk/bin/splunkd

The twistd.py process is the Web server itself. It's a lightweight, Python-powered Web server that communicates with your browser via http, and communicates with splunkd via a SOAP application program interface (API).

root 5092 4.1 0.8 19500 12736 pts/11 S 14:05 0:00 python /opt/splunk/lib/python2.4/site-packages/splunk/core/twistd.py -noy /opt/splunk/lib/python2.4/site-packages/splunk/search/Search.tac

Finally, check for output logs from the server.

# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log

You should see a steady stream of messages arriving every few seconds.

Stop the Server

To shut down the Splunk Server, run this command.

# splunk stop

If you get the return status [FAILED] instead of [OK], or if the script seems to hang, try again. If the command still fails, grep for splunk processes. Use kill or kill -9 to shut them down, but be aware this may corrupt your index if the splunkd process is in the middle of indexing data. It's best to give it a minute to see if it exits on its own.

Restart the Server

To restart the server after making configuration file changes, click Admin on the upper left of the Splunk Server interface. Go to the Server -> Control panel to find the Restart button.

Or, from the server host's shell prompt, run this command.

# splunk restart

Index Your Data

If you have never set up a Splunk Server, click on Admin at the upper left of the interface and then click the Guided Setup option for step-by-step instructions.

To set up additional inputs, click the Admin -> Data Inputs tab and use the separate configuration panels for Files & Directories, FIFOs, Network Ports and Database Tables.

For more detailed instructions, see the section on Data Inputs.

Delete Data from the Index

Delete everything

Run this command on the Splunk Server host to delete all indexed data, users, and saved/live splunks:

# splunk clean all

Run this command to delete event data:
# splunk clean eventdata

Delete a subset of events

Use the delete:: modifier either in the search box of the Splunk Server's web interface, or from the command line on the Splunk Server host:

delete::<search-terms>

The syntax for delete:: is tricky for now:

• You can only use one delete:: modifier per command.
• You can additionally add one deleterestrict:: operator to further filter the command.
• You can only restrict deletion by source, by host and by time range.
• Additional search terms must come first.

For example, this Splunk will delete every event with host::10.1.1.72 from source::/var/log/anaconda.log whose timestamp is within the past day.

daysago::1 delete::host::10.1.1.72 deleterestrict::source::/var/log/anaconda.log

If you type delete:: into the web interface, you'll see a typeahead list of all allowable completions for a delete:: command on your data. The same is true for deleterestrict::.

The best way to use delete:: is like this:

• First, create a search that returns only the events you want to delete.
• Then, add delete:: in front of one of its parameters to delete the same set of events. Move the delete:: term to the rightmost end.

NOTE: The delete:: modifier does not free disk space on the index. The data still resides in the index, it simply cannot be returned in a search. If you need to recover the disk space you will need to use the command # splunk clean eventdata

Change Network Ports

Click Admin in the upper left of the Splunk Server interface. This will open the Admin page to the Server -> Settings tab, where you can change the port settings. If you set either your HTTP or HTTPS port to 0 listening on that service will be disabled.

Uninstall the Release

Your installation may have a different pathname to your Splunk software and files, depending on how it was installed.

To erase all traces of the Splunk Server installation, first stop the server.

# splunk stop

Then run the uninstaller in your installation's top-level directory.

# $SPLUNK_HOME/uninstall

Alternately, you can use these Unix commands to remove all traces of Splunk from your host.

# splunk clean
# rm -rf $SPLUNK_HOME
# userdel splunk
# groupdel splunk

By default your index was under the /opt/splunk directory and was removed by the commands above. But if you had relocated your index ($SPLUNK_DB) outside of /opt/splunk , erase that directory, too.

# rm -rf /foo/bar/splunkdb/