The venerable old-skool Splunk forums are now closed. Feel free to search for old content here, but new posts are no longer supported.

Instead, please visit the thriving community at answers.splunk.com to ask and answer questions about your Splunk deployment and how to get the most out of it.

Forums: SplunkAdministration: Splunkd service does not start

Previous Topic: Delopying as agent causing 100% CPU usage  |   Next Topic: The need to search for a hosts defined "hosttag"


Posts 1–5 of 5

After adding a search in the savedsearches.conf file, i restarted splunk, but the splunkd service did not start.
I removed the supposedly offending search (!?) to see if i had provoked the crash, but the service did not start either.
Rebooted the server :-) , but this did not do any magic.
On a web client i receive:

The splunkd daemon is no longer responding and may have crashed. The UI is fine, so check splunkd.log or look for any crash dumps.

I checked the splunkd.log with no luck. Nothing has been logged in there.
I checked the web_service.log as it is the last modified .log file and found these:

2009-11-20 15:51:25,079 INFO init:176 - Setting logger=splunk level=INFO
2009-11-20 15:51:25,079 INFO init:176 - Setting logger=splunk.appserver level=INFO
2009-11-20 15:51:25,079 INFO init:176 - Setting logger=splunk.appserver.controllers level=INFO
2009-11-20 15:51:25,079 INFO init:176 - Setting logger=splunk.appserver.lib level=WARN
2009-11-20 15:51:33,438 INFO lists:59 - List controller loaded: EntitiesListGenerator
2009-11-20 15:51:33,438 INFO lists:65 - Setting lists/entities
2009-11-20 15:51:33,438 INFO lists:59 - List controller loaded: JobsListGenerator
2009-11-20 15:51:33,438 INFO lists:65 - Setting lists/jobs
2009-11-20 15:51:33,438 INFO lists:59 - List controller loaded: SavedListGenerator
2009-11-20 15:51:33,438 INFO lists:65 - Setting lists/saved
2009-11-20 15:51:33,438 INFO lists:59 - List controller loaded: SavedSearchesListGenerator
2009-11-20 15:51:40,032 ERROR startup:40 - Unable to read in product version information; Splunkd daemon is
not responding: ('[Errno 10061] No connection could be made because the target machine actively refused it',)
2009-11-20 15:51:40,127 INFO root:102 - ENGINE: Bus STARTING
2009-11-20 15:51:40,127 INFO custompidfile:17 - Writing pidfile at C:\Program Files\Splunk\var\run\splunk\splu
nkweb.pid (PID: 1360)"
2009-11-20 15:51:40,174 ERROR root:102 - ENGINE: Set handler for console events.
2009-11-20 15:51:40,174 INFO root:102 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2009-11-20 15:51:41,063 INFO root:102 - ENGINE: Serving on 0.0.0.0:8000
2009-11-20 15:51:41,063 INFO root:102 - ENGINE: Bus STARTED
2009-11-20 16:37:29,848 INFO root:102 - ENGINE: Started monitor thread 'Monitor'.
2009-11-20 16:37:30,911 ERROR startup:40 - Unable to read in product version information; Splunkd daemon is
not responding: ('[Errno 10061] No connection could be made because the target machine actively refused it',)
2009-11-20 16:37:30,911 INFO decorators:115 - No sessionKey session variable set
2009-11-20 16:37:32,005 ERROR startup:40 - Unable to read in product version information; Splunkd daemon is
not responding: ('[Errno 10061] No connection could be made because the target machine actively refused it',)
2009-11-20 16:37:44,911 ERROR startup:40 - Unable to read in product version information; Splunkd daemon is
not responding: ('[Errno 10061] No connection could be made because the target machine actively refused it',)
2009-11-20 16:37:46,006 ERROR decorators:161 - Splunkd daemon is not responding: ('[Errno 10061] No
connection could be made because the target machine actively refused it',)
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py",
line 147, in handle_exceptions
return fn(self, *a, **kw)
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\controllers\account.py
", line 71, in login
sessionKey = splunk.auth.getSessionKey(username, password, hostPath=self.splunkd_urlhost)
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\splunk\auth.py", line 21, in getSessionKey
serverResponse, serverContent = rest.simpleRequest(uri, postargs=args)
File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\splunk\rest\init.py", line 359, in
simpleRequest
raise splunk.SplunkdConnectionException, str(e)
SplunkdConnectionException: Splunkd daemon is not responding: ('[Errno 10061] No connection could be made
because the target machine actively refused it',)

Splunk Version: 4.0.4 (build 67724)

Thoughts?
Thanks,
Marcelo

The error on this file seems to be:

ERROR startup:40 - Unable to read in product version information; Splunkd daemon is
not responding: ('[Errno 10061] No connection could be made because the target machine actively refused it',)

... but that could just be telling me that the splunkd service is down...

maybe i should be looking on another log (which one?) to see why the splunkd could not start. ?

TIA
Marcelo

These are the last lines in my splunkd.log. They seem to have been logged during last shutdown.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'Pipeline:exec' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'Pipeline:exec' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'Pipeline:multiFilePipe' to shutdown, awaiting
response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'Pipeline:multiFilePipe' completed shutdown
immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'Pipeline:parsing' to shutdown, awaiting
response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'Pipeline:parsing' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'Pipeline:merging' to shutdown, awaiting
response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'Pipeline:merging' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'Pipeline:typing' to shutdown, awaiting
response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'Pipeline:typing' completed shutdown immediately
11-20-2009 14:44:09.723 INFO ShutdownHandler - shutting down level "ShutdownLevel_Queue"
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'parsingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'parsingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'aggQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'aggQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'typingQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'typingQueue' completed shutdown immediately
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'indexQueue' to shutdown, awaiting response.
11-20-2009 14:44:09.723 INFO ShutdownHandler - Component 'indexQueue' completed shutdown immediately
11-20-2009 14:44:09.723 INFO ShutdownHandler - shutting down level "ShutdownLevel_Exec"
11-20-2009 14:44:09.723 DEBUG ShutdownHandler - Signal sent to 'ExecProcessor' to shutdown, awaiting response.
11-20-2009 14:44:09.723 WARN pipeline - Exiting pipeline indexerPipe gracefully: got eExit from processor
indexer
11-20-2009 14:44:09.832 WARN pipeline - Exiting pipeline dev-null forcefully
11-20-2009 14:44:10.286 DEBUG ShutdownHandler - Received notification of successful shutdown from
'ExecProcessor', time taken: 562.6 milliseconds
11-20-2009 14:44:10.286 DEBUG ShutdownHandler - shutdown level "ShutdownLevel_Exec" finished in 562.6

You should check the user of the service and the permissions on the log directories (and other directories). The splunkd.log shows nothing, possibly because it can't write to the file.

Hi gkanapathy,

Thanks for your suggestion!
I believe i solved it. : )

For the record (in case anyone else ever needs it) . . .

On the C:\ drive, for some unrelated reason, there was a file named Program..

I deleted the offending c:\program. file and rebooted the server. Now things seem to be back OK.

Again thanks for your help.