Documentation: 3.2.1
|  Print Version Contents
This page last updated: 04/23/08 05:04pm

Automate archiving

Set up Splunk to archive your data automatically. To do this, configure indexes.conf to call archiving scripts located in $SPLUNK_HOME/bin.

Use Splunk's index aging policy to archive

Splunk rotates old data out of the index based on your data retirement policy. Data moves through several stages, which also correspond to file directory locations. Data starts out in the hot database $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db_hot. Then, data moves through the warm database -- a new directory in $SPLUNK_HOME/var/lib/splunk/defaultdb/db. Eventually, data is aged into the cold database $SPLUNK_HOME/var/lib/splunk/defaultdb/colddb.

Finally, data reaches the frozen state. Splunk erases frozen index data once frozenTimePeriodinSecs in indexes.conf is reached. The coldToFrozenScript value specified in indexes.conf runs just before the frozen data is erased. The default script simply writes the name of the directory being retired, e.g. /opt/splunk/var/lib/splunk/defaultdb/colddb, to the log file $SPLUNK_HOME/var/log/splunk/splunkd_stdout.log.

From $SPLUNK_HOME/etc/bundles/local/indexes.conf:

coldToFrozenScript = echo $DIR

Note: coldToFrozenScript must take two arguments -- the script or command and then $DIR.

Archiving scripts

If you want to keep retired data, change coldToFrozenScript to specify a script to archive your data. Splunk ships with two archiving scripts in $SPLUNK_HOME/bin. Use one of these, or create your own and place it in $SPLUNK_HOME/bin.

Note: Modify these scripts to set the archive location for your installation. By default, the location is set to opt/tmp/myarchive.

Splunk's two archiving scripts are:

compressedExport.sh

This script exports files with the tsidx files compressed as gz.

#!/bin/sh
gzip $1/*.tsidx
cp -r $1 /opt/tmp/myarchive  #replace this with your archive directory

flatfileExport.sh

This script exports files as a flat text file.

#!/bin/sh
exporttool $1 ${1}/index.export meta::all
rm -rf ${1}/*.data
rm -rf ${1}/rawdata
rm -rf ${1}/*.tsidx
cp -r $1 /opt/tmp/myarchive  #replace this with your archive directory
Previous: Set retirement policy    |    Next: Restore archived data

Comments

No comments have been submitted.

Log in to comment.