Set a retirement and archiving policy

Set a retirement and archiving policy

Configure data retirement and archiving policy by controlling the size of indexes or the age of data in indexes.

For a discussion of the best practices for for backing up your Splunk data, see "Best practices for backing up" on the Community Wiki. For a related discussion of "buckets", and how Splunk uses them, see "Understanding buckets" on the Deployment Wiki.

Caution: Whenever you change your data retirement and archiving policy settings, Splunk deletes old data without prompting you.

Note: All index locations must be writable to configure data

Splunk indexes go through four stages of retirement. When an index reaches a frozen state, Splunk deletes ALL frozen data by default. You must specify a valid coldToFrozenScript in $SPLUNK_HOME/etc/system/local/indexes.conf (or your own custom app directory in $SPLUNK_HOME/etc/apps/ to avoid losing your data.

Retirement stage	Description	Searchable?
Hot	Open for writing. Can be one or more for each index.	Yes.
Warm	Data rolled from hot. There are many warm indexes.	Yes.
Cold	Data rolled from warm. There are many cold indexes.	Yes.
Frozen	Data rolled from cold. Eligible for deletion.	Splunk deletes frozen data by default.

Splunk defines the sizes, locations, and ages of indexes in indexes.conf.

Note: Edit indexes.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work. Do not edit the copy in default.

Remove files beyond a certain size

If an index grows bigger than a specified maximum size, the oldest data is archived into frozen. To set this maximum size, add the following line to your custom indexes.conf.

maxTotalDataSizeMB = <non-negative number> (500000)

Example:

[main]
maxTotalDataSizeMB = 2500000

Restart Splunk for the new setting to take effect. It may take up to 40 minutes for Splunk to move events out of the index to conform to the new policy. You may see high CPU usage during this time.

Note: Make sure that the data size you specify for maxTotalDataSizeMB = is expressed in Megabytes.

Remove files beyond a certain age

Splunk ages out data by buckets. Specifically, when the most recent data in a particular bucket reaches the configured age, the entire bucket is rolled. If you are indexing a large volume of events, bucket size is less a concern for retirement policy because they fill quickly. You can adjust the bucket size by setting maxDataSize in indexes.conf smaller so they roll faster. But more, smaller buckets take more time to search than fewer, larger buckets. To get the results you are after, you will have to experiment a bit for the right size. Due to the structure of the index, there isn't a direct relationship between time and data size.

Set the variable frozenTimePeriodinSecs in indexes.conf to the number of seconds after which indexed data should be erased. The example below configures Splunk to cull old events from its index when they become more than 180 days old. The default value is approximately 6 years.

[main]
frozenTimePeriodInSecs = 15552000

Restart Splunk for the new setting to take effect.

Note: Make sure that the time you specify for frozenTimePeriodInSecs = is expressed in seconds.

Example

To define a policy where events are deleted after 100 days, multiply how many seconds there are in a day (86400) times how long you want to keep logs around (100) and set that as the frozenTimePeriodsInSecs at the top of $SPLUNK_HOME/etc/system/local/indexes.conf. Hot buckets will automatically roll if they haven't been written to in a given amount of time.

How time and age policies combine

If you configure a policy based on both time and age, whichever triggers an archival/removal action first takes effect. Both sets of rules remain in place and do not conflict.