Documentation: 3.3
|  Print Version Contents
This page last updated: 09/19/08 09:09pm

Automate archiving

Set up Splunk to archive your data automatically as it ages. To do this, configure indexes.conf to call archiving scripts located in $SPLUNK_HOME/bin. Edit this file in $SPLUNK_HOME/etc/system/local/, or in your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work. Do not edit the copy in default.

Note: By default, Splunk deletes ALL frozen data. To avoid losing your data, you must specify a valid coldToFrozenScript in $SPLUNK_HOME/etc/system/local/indexes.conf (or your own custom app directory in $SPLUNK_HOME/etc/apps/).

Use Splunk's index aging policy to archive

Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which correspond to file directory locations. Data starts out in the hot database $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db_hot. Then, data moves through the warm database $SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database $SPLUNK_HOME/var/lib/splunk/defaultdb/colddb.

Finally, data reaches the frozen state. Splunk erases frozen index data once it is older than frozenTimePeriodinSecs in indexes.conf. The coldToFrozenScript (also specified in indexes.conf) runs just before the frozen data is erased. The default script simply writes the name of the directory being retired, e.g. /opt/splunk/var/lib/splunk/defaultdb/colddb, to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log.

Add the following to $SPLUNK_HOME/etc/system/local/indexes.conf:

[<index>]
coldToFrozenScript = <script>

  • [<index>]
    • Specify which index to archive.
  • coldToFrozenScript = <script>
    • Specify the archiving script to use by changing <script>.
    • Define <$script> paths relative to splunk/bin.
    • Splunk ships with two default archiving scripts that you can use.
      • Note: Modify these scripts to set the archive location for your installation. By default, the location is set to opt/tmp/myarchive.
      • compressedExport.sh: Export with tsidx files compressed as gz.
      • flatfileExport.sh: Export as a flat text file.
  • Windows users use this notation: coldToFrozenScript = <script> "$DIR"
    • <script> can be either:
      • compressedExport.bat (download the script here).
      • flatfileExport.bat (download the script here).
Previous: Set a retirement and archiving policy    |    Next: Restore archived data

Comments

  1. update: windows scripts are now available.
    Posted by emma on Sep 22 2008, 12:11pm

  2. @djz:

    yes these scripts are not currently available on Windows. they should be available in the next maintenance release. currently, this is being tracked as issue SPL-15940, so check for that in the release notes.
    Posted by emma on Sep 12 2008, 5:30pm

  3. This page says that Splunk ships with two archiving scripts, but I couldn't find the referenced scripts in the $SPLUNK_HOME/bin directory. Installed on Windows, so maybe that is the difference?
    Posted by djz on Aug 12 2008, 11:40am

Log in to comment.