Max What?

Is it possible to configure anything beyond the basic time-based or size-based policies for the content in the Splunk datastore? Yes, it is. In fact, there are so many knobs and dials, it may not be easy to distinguish which parameters do what. The parameters live in indexes.conf and have sadistically similar names–maxDataSize, maxTotalDataSizeMB, maxMemMB.

First, when setting up retention/archival policy, it’s helpful to understand the basics of how Splunk manages data. Then below is a visual to help match the exact configuration parameter you need to adjust for controlling the flow of data from one stage to the next.

For most environments, the parameters you will likely need to adjust are few:

• maxTotalDataSizeMB – to apply a size-based retention policy
• frozenTimePeriodInSecs – to apply a time-based retention policy
• maxWarmDBCount – to split the hot/warm and cold directories among separate partitions

Currently, it is possible to set maxTotalDataSizeMB (size-based policy) and maxDataSize (bucket size) via SplunkWeb under Manager > Indexes > MyIndex. All other parameters are set using indexes.conf. Here is a simple example:

[myLovelyIndex]
# set directory location of hot/warm, cold and thawed
homePath = $SPLUNK_DB/lovelydb/db
coldPath = /path/to/another/partition/lovelydb/colddb
thawedPath = /my/computer/lovelydb/thaweddb
# set time-based policy of 1 year
frozenTimePeriodInSecs = 31556926
# set size-based policy of 1 TB
maxTotalDataSizeMB = 1048576
# do not delete cold dbs, instead move them to slower storage
coldToFrozenScript = moveToSlow.sh

For more advanced configurations and backup purposes, the full set of parameters will empower you to take advantage of Splunk’s flexible data management.

----------------------------------------------------
Thanks!
Vi Ly