Documentation: 3.3
|  Print Version Contents
This page last updated: 05/17/08 09:05pm

Use separate partitions for Splunk's datastore

Splunk can use separate disks and partitions for its datastore. Theoretically, you can use any sort of arrangement of disks/partitions so long as you mount them correctly and configure the DB rolling.

Splunk indexes roll through four DB stages:

  • Hot - open for writing. Only one of these for each index. Searchable.
  • Warm - data rolled from hot. There are many warm DBs. Searchable.
  • Cold - data rolled from warm. There are many cold DBs. Searched only when the search specifies a time range included in these files.
  • Frozen - data rolled from cold. Eligible for deletion.

The most common way to arrange Splunk's datastore on separate partitions is to keep the hot and warm databases on the local machine, and to keep the cold database on a separate array or disks (for longer term storage). You want to run your hot and warm databases on a machine with partitions that read and write fast (since you'll be doing a majority of your search operations on hot and warm). Cold should be on a reliable array of disks.

You may experience pauses in indexing and searching when you use separate partitions for the datastore.

Set up separate partitions

Set up partitions just as you'd normally set them up in any operating system. Mount the disks/partitions, and make sure Splunk points to the correct path in indexes.conf.

First, add the correct paths in $SPLUNK_HOME/etc/system/local/indexes.conf. Set paths on a per-index basis -- under an [$INDEX] entry.

homePath = <path on server>

  • The path that contains the hot and warm databases and fields for the index.
  • Databases that are warm have a handle open to them at all times in splunkd.
  • CAUTION: Path MUST be writable.

coldPath = <path on server>

  • The path that contains the cold databases for the index.
  • Cold databases are opened as needed when searching.
  • CAUTION: Path MUST be writable.

thawedPath = <path on server>

  • The path that contains the thawed (resurrected) databases for the index.

If you put your cold DB on a separate partition, you should set a warmToColdScript in indexes.conf. Set up a script to move your warm DBs from one partition to the partition where you store your cold DBs.

warmToColdScript = <$script>

  • Specify a script to run when moving data from warm to cold.
  • The script must accept two variables:
    • first, the warm directory to be rolled to cold.
    • second, the destination in the cold path.
  • You only need to set this if you store warm and cold dbs on separate partitions.
  • Contact Splunk Support if you need help configuring this setting.
  • Defaults to empty.
Previous: Disk usage    |    Next: Use WORM (Write Once Read Many) volumes for Splunk's datastore

Comments

No comments have been submitted.

Log in to comment.