Set a retirement and archiving policy

Configure data retirement and archiving policy by controlling the size of indexes or the age of data in indexes.

Caution: Whenever you change your data retirement and archiving policy settings, Splunk deletes old data without prompting you.

Note: All index locations must be writable to configure data

Splunk indexes go through four stages of retirement. When an index reaches a frozen state, Splunk deletes ALL frozen data by default. You must specify a valid coldToFrozenScript in $SPLUNK_HOME/etc/system/local/indexes.conf (or your own custom app directory in $SPLUNK_HOME/etc/apps/ to avoid losing your data.

Retirement stages

Hot Open for writing. Only one of these for each index. Searchable.

Warm Data rolled from hot. There are many warm indexes. Searchable.

Cold Data rolled from warm. There are many cold indexes. Searchable only if a search's time range is applicable to data in the Cold stage.

Frozen Data rolled from cold. Eligible for deletion. Splunk deletes frozen data by default.

Splunk defines the sizes, locations, and ages of indexes in indexes.conf.

Note: Edit indexes.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work. Do not edit the copy in default.

Remove files beyond a certain size

Add the following to your custom indexes.conf.

maxTotalDataSizeMB = <non-negative number> (500000)
   * The maximum size of an index. If an index grows bigger than this the oldest data is frozen out.

Example:

[main]
maxTotalDataSizeMB = 2500000

Restart Splunk for the new setting to take effect. It may take up to 40 minutes for Splunk to move events out of the index to conform to the new policy. You may see high CPU usage during this time.

Note: Make sure that the data size you specify for maxTotalDataSizeMB = is expressed in Megabytes.

Remove files beyond a certain age

Splunk ages out data by buckets. Specifically, when the most recent data in a particular bucket reaches the configured age, the entire bucket is rolled. If you are indexing a large volume of events, bucket size is less a concern for retirement policy because they fill quickly. You can adjust the bucket size by setting maxDataSize in indexes.conf smaller so they roll faster. But more, smaller buckets take more time to search than fewer, larger buckets. To get the results you are after, you will have to experiment a bit for the right size. Due to the structure of the index, there isn't a direct relationship between time and data size.

Set the variable frozenTimePeriodinSecs in indexes.conf to the number of seconds after which indexed data should be erased. The example below configures Splunk to cull old events from its index when they become more than 180 days old. The default value is approximately 6 years.

[main]
frozenTimePeriodInSecs = 15552000

Restart the server for the new setting to take effect.

Note: Make sure that the time you specify for frozenTimePeriodInSecs = is expressed in seconds.

Previous: Move an index | Next: Automate archiving

Comments

No comments have been submitted.

Hot	Open for writing. Only one of these for each index. Searchable.
Warm	Data rolled from hot. There are many warm indexes. Searchable.
Cold	Data rolled from warm. There are many cold indexes. Searchable only if a search's time range is applicable to data in the Cold stage.
Frozen	Data rolled from cold. Eligible for deletion. Splunk deletes frozen data by default.