Contact Support

For contact information, see the main Support contact page.

Here is some information on tools and techniques Splunk Support uses to diagnose problems. Many of these you can try yourself.

infoGather

The infoGather script collects basic info about your Splunk server, including Splunk's configuration details (such as the contents of $SPLUNK_HOME/etc and general details about your index such as host and source names). It does not include any event data or private information.

From $SPLUNK_HOME/bin run

./splunk diag

This produces splunk-diag.tar.gz (or .zip) that you can send to Splunk Support for troubleshooting.

Log levels and starting in debug mode

Splunk logging levels can be changed to provide more detail for different features in the $SPLUNK_HOME/var/log/splunk/splunkd.log. The easiest way is to enable all messages with the --debug option. This does impact performance and should not be used routinely.

Stop Splunk, if it is running.
Save your existing splunkd.log file by moving it to a new filename, like splunkd.log.old.
Restart Splunk in debug mode with splunk start --debug.
When you notice the problem, stop Splunk.
Move the new splunkd.log file elsewhere and restore your old one.
Restart Splunk normally.

Specific areas can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file $SPLUNK_HOME/etc/log.cfg to set specific log levels. Note that not all messages marked WARN or ERROR indicate actual problems with Splunk; some indicate that a feature is not being used.

Debug Splunk Web

Enable additional Splunk Web debugging in the file $SPLUNK_HOME/etc/SplunkWeb.tac.

Change this line:

# set global logging level
appLoggingLevel = logging.INFO

To this:

# set global logging level
appLoggingLevel = logging.DEBUG

The additional messages are output in $SPLUNK_HOME/var/log/splunk/web_service.log file.

For 3.2+, debug messages in splunkd.log can also be enabled dynamically with a search:

To enable debugging search for

| oldsearch !++cmd++::logchange !++param1++::root !++param2++::DEBUG

To return to the default log level search for

| oldsearch !++cmd++::logchange !++param1++::root !++param2++::WARN

This does not change any settings in log.cfg. On restart, the log level reverts to what is defined in log.cfg.

Note This search will return a "Search Execute failed because Setting priority of ... " message. This is normal.

Core Files

To collect a core file, use ulimit to remove any maximum file size setting before starting Splunk.

# ulimit -c unlimited
# splunk restart

This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option (splunk start --nodaemon). In another shell, start the web interface manually with splunk start splunkweb.

Depending on your system, the core may be named something like core.1234, where the number indicates the process id and be the same location as the splunkd executable.

LDAP configurations

If you are having trouble setting up LDAP, Support will typically need the following information:

The authentication.conf file from $SPLUNK_HOME/etc/system/local/.
An ldif for a group you are trying to map roles for.
An ldif for a user you are trying to authenticate as.

In some instances, a debug splunkd.log or web_service.log are helpful.

Previous: wmi.conf | Next: Splunkd is down

Comments

No comments have been submitted.