The venerable old-skool Splunk forums are now closed. Feel free to search for old content here, but new posts are no longer supported.

Instead, please visit the thriving community at answers.splunk.com to ask and answer questions about your Splunk deployment and how to get the most out of it.

Forums: SplunkGeneral: Optimizing IO

Previous Topic: Linux install doesn't  |   Next Topic: How to potentially double or more the performance of indexing in 2.0.3


Posts 1–4 of 4
1
posted by: robsuh  |  posts
date: May 26, 2006
permalink

Has anyone optimized their splunk instance for IO on Linux? Any hints (perhaps kernel parameter to tweaks?)

I have two instances of Splunk 2.0.2 on Linux. One running on a SATA drive and the other running on a RAID1 (HP Smart Array 6i with battery backed write cache.)

The RAID1 system does > 2x the IO/sec than the SATA, but that is still not enough. With 'iostat', I'm still seeing the system spend 45% time in iowait - starved for data.

I'm tempted to try and put it on our SAN (EMC Clariion) with a RAID10 LUN.

Our index size has so far grown to around 20GB with 7M events. The system gets very unreponsive.

2
posted by: Joseph Reeves  |  posts
date: May 26, 2006
permalink

Hmmm... with only 7 million events your splunk instance should not be getting unresponsive. I've got over 13 million in mine right now on more modest hardware and the performance is well within acceptable. Have you tweaked any of your Splunk index settings yet? First edit your properties.xml config like so. Then go to /opt/splunk/etc/myinstall/pluginConfs [assuming splunk's installed in /opt] and edit your multiIndexer.xml according to the recommendations here. Limiting the number of files that Splunk keeps open at one time should improve your I/O performance dramatically. I prefer to keep all of my databases that are being read "warm" and no colddbs. Let me know if you're like to know the specifics on how I accomplished that.

3
posted by: stechert  |  posts
date: May 26, 2006
permalink

Erik has a blog entry about this coming in the next day or so. We do have an answer for this.

4
posted by: erik  |  posts
date: May 27, 2006
permalink

Just curions if the following fixes your problem:
http://blogs.splunk.com/erik/2006/05/26/how-to-potentially-double-in-some-cases-nearly-triple-indexing-performance-in-the-203-build/

Please do let me konw if this does not reduce your io pressure.