Use separate partitions for index data

Use separate partitions for index data

Splunk can use separate disks and partitions for its index data. It's possible to configure Splunk to use many disks/partitions/filesystems on the basis of indexes and warm/cold, so long as you mount them correctly and configure the DB rolling. However, we recommend that you use a single high performance file system to hold your Splunk index data for the best experience.

Splunk indexes roll through four stages:

• Hot - open for writing. There are multiple hot buckets. Searchable.
• Warm - data rolled from hot. There are many warm buckets. Searchable.
• Cold - data rolled from warm. There are many cold buckets. Searched only when the search specifies a time range included in these files.
• Frozen - buckets entering the frozen state are immediately deleted.

If you do use separate partitions, the most common way to arrange Splunk's index data is to keep the hot and warm buckets on the local machine, and to keep the cold bucket on a separate array or disks (for longer term storage). You want to run your hot and warm buckets on a machine with partitions that read and write fast (since you'll be doing a majority of your search operations on hot and warm). Cold should be on a reliable array of disks.

Bucket flow:

• The single hot bucket rolls to warm when it reaches the specified size (maxDataSize)
• Buckets roll from warm to cold when the number of warm buckets exceeds the configured maximum count (maxWarmDBCount)
• Buckets stay in cold (or warm) until they are selected for archiving

Set up separate partitions

Set up partitions just as you'd normally set them up in any operating system. Mount the disks/partitions, and make sure Splunk points to the correct path in indexes.conf.

First, add the correct paths in $SPLUNK_HOME/etc/system/local/indexes.conf. Set paths on a per-index basis -- under an [$INDEX] entry.

homePath = <path on server>

• The path that contains the hot and warm databases and fields for the index.
• Databases that are warm have a handle open to them at all times in splunkd.
• CAUTION: Path MUST be writable.

coldPath = <path on server>

• The path that contains the cold databases for the index.
• Cold databases are opened as needed when searching.
• CAUTION: Path MUST be writable.

thawedPath = <path on server>

• The path that contains the thawed (resurrected) databases for the index.