This documentation applies to the following versions of Splunk: 4.0 , 4.0.1 , 4.0.2 , 4.0.3 , 4.0.4 , 4.0.5 , 4.0.6 , 4.0.7 , 4.0.8 , 4.0.9 , 4.0.10
For contact information, see the main Support contact page.
If you are a Splunk user at a large organization, it's possible that you may have to go through an authorized contact to reach Splunk support. Check with your Splunk administrator to find out who that is.
If you are an authorized contact, you can specify other users in your organization as authorized contacts. To do so, visit this page. If you do not have access to this page, check with your Splunk Sales representative for access.
Here is some information on tools and techniques Splunk Support uses to diagnose problems. Many of these you can try yourself.
Note: Before you send any files or information to Splunk Support, verify that you are comfortable with sending it to us. We try to ensure that no sensitive information is included in any output from the commands below, but we cannot guarantee compliance with your particular security policy.
The diag command collects basic info about your Splunk server, including Splunk's configuration details (such as the contents of $SPLUNK_HOME/etc and general details about your index such as host and source names). It does not include any event data or private information.
You must run this command from the /bin directory, not from any subdirectories within $SPLUNK_HOME.
From $SPLUNK_HOME/bin run
UNIX:
./splunk diag
Windows:
splunk diag
If you have difficultly running diag in your environment, you can also run the python script directly from the bin directory using cmd
./splunk cmd python ../lib/python2.6/site-packages/splunk/clilib/info_gather.py
Or in Splunk 3.x:
./splunk cmd python ../lib/python2.5/site-packages/splunk/clilib/info_gather.py
This produces splunk-diag.tar.gz (or .zip) that you can send to Splunk Support for troubleshooting.
Upload your diag output to your Support case here -
Splunk logging levels can be changed to provide more detail for different features either from within Splunk Manager, or by editing $SPLUNK_HOME/etc/log-local.cfg.
Splunk's logging levels are DEBUG INFO NOTICE WARN ERROR CRIT ALERT FATAL EMERG (most to least verbose). If a default level is not specified for a category the logging level defaults to your rootCategory setting. The easiest way to increase the verbosity of the log is to enable all messages with the --debug option. This does impact performance and should not be used routinely.
splunkd.log file by moving it to a new filename, like splunkd.log.old.
splunk start --debug.
splunkd.log file elsewhere and restore your old one.
Specific areas can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file $SPLUNK_HOME/etc/log.cfg to set specific log levels without enabling a large number of categories as with --debug. Restart Splunk after changing this file.
Note: Not all messages marked WARN or ERROR indicate actual problems with Splunk; some indicate that a feature is not being used.
To enable debugging, execute the following search:
3.4.x:
| oldsearch !++cmd++::logchange !++param1++::root !++param2++::DEBUG
To set a particular category of messages, replace "root" with the desired category. To return to the default log level, execute the following search:
| oldsearch !++cmd++::logchange !++param1++::root !++param2++::WARN
4.x:
logchange searches now update individual categories, one at a time.
| debug cmd=logchange param1=FileInputTracker param2=DEBUG
| debug cmd=logchange param1=FileInputTracker param2=WARN
These searches do not change any settings in log.cfg. On restart, the log level reverts to what is defined in log.cfg.
Note This search will return a "Search Execute failed because Setting priority of ... " message. This is normal.
For investigating problems monitoring files, use the FileInputTracker and selectProcessor categories. These are not enabled with the normal "--debug" option because they are very verbose.
Enable additional Splunk Web debugging in web.conf:
[settings] appLoggingLevel = DEBUG
Restart the splunkweb process with the command ./splunk restart splunkweb. The additional messages are output in $SPLUNK_HOME/var/log/splunk/web_service.log file.
To collect a core file, use ulimit to remove any maximum file size setting before starting Splunk.
# ulimit -c unlimited
# splunk restart
This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option (splunk start --nodaemon). In another shell, start the web interface manually with splunk start splunkweb.
Depending on your system, the core may be named something like core.1234, where the number indicates the process id and be the same location as the splunkd executable.
If you are having trouble setting up LDAP, Support will typically need the following information:
authentication.conf file from $SPLUNK_HOME/etc/system/local/.
In instances where you have the configuration partially working, a debug splunkd.log and web_service.log are helpful.
Important: You must contact Splunk support for direction before using this command.
The recover-metadata command recovers missing or corrupt metadata associated with any Splunk index directory, sometimes also referred to as a 'bucket'. If your Splunk instance will not start up, one possible diagnosis is that one or more of your index buckets is corrupt in some way. Contact support; they will help you determine if this is indeed the case and if so, which bucket(s) are affected. Then, run this command:
$SPLUNK_HOME/bin/splunk cmd recover-metadata <full path to the exact bucket>
Splunk will return a success or failure message.