Forums: SplunkGeneral: splunkd running with way more than two processes

When I type ps aux | grep splunk, I get 12 processes, not two . . . is that normal?

Also, the server I'm running on is a 2.8 GHz P4, but the Splunk processes still manage to run at a very high CPU usage level; anywhere from 10-50%. Sometimes more than one of the 12 processes is hogging CPU simulaneously too.

Is there any way to throttle this thing down?

Let me guess. You're using the tailing processor?
How many log files are you watching?

The splunkd server should only be two processes. What is the server currently doing, eating files ? by what mechanism are you doing that? (batch loader, tailing processor, etc.).

Can you run a ps auxfft and paste the result of that in here ? what os and kernel version are you running ?

cheers,
Rory

Kernel 2.4.26, Slackware 10.0.

Yeah, I'm using the tailing processor. Is that a bad idea?

One problem I noticed was that I forgot to move/delete my /varl/log/scripts and /var/log/packages directories, so it was trying to tail all those install scripts. 700+ sources. That explains the CPU usage, I guess.

So, I went through and removed /var/log as a source, and re-added only the files/dirs that I'm interested in tailing. Here's the list:

1. ./splunk tail list

Files/directories currently being tailed:
/opt/splunk/var/log/splunk/searchhistory.log
/opt/splunk/var/log/splunk/splunkd.log
/opt/splunk/var/log/splunk/splunkSearch.log
/opt/splunk/var/log/splunk/splunklogger.log
/usr/local/mysql/data/choochoo.err
/usr/local/apache/logs/error_log
/usr/local/apache/logs/access_log
/var/log/messages
/var/log/cron
/var/log/wtmp
/var/log/kdm.log
/var/log/messages
/var/log/debug
/var/log/dmesg
/var/log/lastlog
/var/log/faillog
/var/log/maillog
/var/log/Xorg.0.log
/var/log/spooler
/var/log/scrollkeeper.log
/var/log/samba.smbmount
/var/log/Xorg.0.log.old
/var/log/secure
/var/log/syslog
None
/var/log/gdm
/var/log/cups
/var/log/samba
/var/log/iptraf
None

CPU usage, much better. But:

1. ps auxfft | grep splunk

root 31486 7.9 12.0 427300 108600 ? S 10:19 0:02 /opt/splunk/bin/splunkd -p 8089 start
root 31487 0.0 0.3 8920 2964 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31488 0.0 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31489 0.0 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31490 0.0 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31495 0.0 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31496 0.4 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31497 6.4 12.0 427300 108600 ? S 10:19 0:01 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31499 6.5 12.0 427300 108600 ? S 10:19 0:01 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31500 0.6 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31501 4.7 12.0 427300 108600 ? S 10:19 0:01 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31502 3.3 12.0 427300 108600 ? S 10:19 0:00 \_ /opt/splunk/bin/splunkd -p 8089 start
root 31507 1.2 1.2 14424 11548 ? S 10:19 0:00 python /opt/splunk/bin/twistd --pidfile=/opt/splunk/var/run/splunk/splunkSearch.pid --python=/opt/splunk/etc/Search.tac --logfile=/opt/splunk/var/log/splunk/splunkSearch.log

1. date

Thu Jun 22 10:20:22 EDT 2006

Think it matters?

There used to be a problem with the tailing processor in previous versions. It would use a thread for every source, which caused alot of resource problems. Since the 2.x builds it has been using just one thread for all sources so adding alot of sources to tail would bring up your CPU load (as it needs to read alot of files) but would not affect your thread count.

I notice your running a 2.4 kernel, I seem to remember that 2.4 will show you the threads in the process list. I think your seeing just two processes there and the rest of the entries are just the threads.
If your running the new version of splunk, 2 processes and 10 threads sounds about right for that.

Are you see performance problems ?

cheers,
Rory

Not anymore. I cleaned out the 700 or so sources that it was looking at, and now it barely registers as using any CPU.

Thanks for the info, I never knew that about ps and threads.