Forums: SplunkGeneral: Control disk space used.

Hi all,

If I want to limit the space used by the index to 5 G and keep data in splunk for 6 weeks, can you please confirm that the following settings are ok :

Use frozenTimePeriodInSecs to delete cold db older than 6 weeks :

frozenTimePeriodInSecs3628800frozenTimePeriodInSecs

Set maxWarmDBCount to
(5 * 1024)/(500 + (0.5*300) -1 = 7

Can you please explain the parameters :

maxColdDBCount2maxColdDBCount

and

rotatePeriodInSecs300rotatePeriodInSecs

Thanks a lot for your help.
Pierre.

Hi Pierre,

A quick explaination of how Splunk indexing works. Basically it rotates files depending on your settings. Here's the progression. It starts out with "hot", which is the file that splunk writes to. After that, it rolls like this: hot->warm->cold->frozen. Unless you have a frozen script set up, it doesn't actually do anything in that step. The purpose of the frozen step is to set up archival of your indexes if you wish to. Hot, warm and cold are all readable by splunk when you do a search. So the maxWarmDBCount and maxColdDBCount control how many files of each of those types that splunk keeps around. The rotatePeriodInSecs parameter controls how long the file stays in each state before it's rolled to the next one. If you can get a rough estimate of your logging rate, you can use those settings to determine how much space splunk is using.

Hope this helps!
Joe

Hi Joe,
Thanks for your answer but it's not clear for me.

Why is the rotatePeriodInSecs set to 300 sec by default ? Why is it so short ??

I have around 100M of syslog per day, I want to keep 6 weeks of logs and I want to limit the space used by splunk to 5 G.

If I apply the formula found in splunk administrator's guide, I have (5 * 1024)/(500 + (0.5*300)-1 = 7.

Is it ok ?

Thanks for your help.
Pierre.

Pierre,

I would say that controlling disk space usage using the index settings in Splunk is not quite an exact science. I've actually been tweaking my own a good bit lately to try to get splunk to stop using so much disk space. Since your log volume is a lot lower than mine [and has less variance] though, it should be easier to get things under control. I would say that 512Kb a second is probably far more than your server is logging if you only log 100mb total per day. However, a warmDB count of 7 sounds fairly reasonable to me. I'd say try it with 7 and if that's keeping too much data on disk, adjust accordingly. Keep an eye on the space that splunk is using by doing a du -sh on the directory like so:
du -sh /opt/splunk/var/lib/splunk

Also, as I've recently discovered, you have to keep an eye on how many colddbs your system is keeping. I just disabled mine entirely since it didn't seem to be working properly. So this is the setting I changed to do that:
<frozenTimePeriodInSecs>0</frozenTimePeriodInSecs>

As you can see, it set it to 0 since I was told that would prevent Splunk from creating colddbs. I plan to control disk usage by limiting the number of warmDBs. Maybe this isn't the "right way" to do things persay, but I'll bet it's effective. I apologize if this confuses you more. It's not exactly a straightforward process and I actually suggested a while back that they include a section in the installer that walks you through setting up index size settings with a few simple questions. I know that a "config tool" is planned, so perhaps in the future this will be a lot easier to set up!

I'll keep you posted on how things are going with mine, let me know how it works out for you and we'll compare notes!

Thanks!
Joe

Joe,

Thanks for your good answer. I will let you know how it will evolve.

Thanks for your support.
Regards.
Pierre.