Documentation: 3.2
|  Print Version Contents
This page last updated: 04/14/08 03:04pm

Disk usage

There are several methods for controlling disk space used by Splunk. Most disk space will be used by Splunk's indexes and compressed log files (collectively called the database). If you run out of disk space, Splunk will stop indexing. You can set a minimum free space limit to control how low you will let free disk space fall before indexing stops. Indexing will resume once you space exceeds the minimum.

Set minimum free disk space

Use settings in Splunk Web to set a minimum amount of disk space to keep free on on the disk where indexed data is stored. If the limit is reached, the server stops indexing data until more space is available.

Note:

  • The Splunk server will not clear any of its own disk space under this method. It will simply wait for more space to become available.
  • Some events may be lost if they are not written to a file during the paused period.

In Splunk Web

  • Click Admin in the upper right corner of the web interface.
  • Click the Server tab.
  • Click on Settings heading.
  • Under the Datastore section, find Pause indexing if free disk space falls below ___ MB:

http://www.splunk.com/assets/doc-images/30_admin_DiskUsage/datastore1.jpg

  • Enter your desired minimum free disk space in megabytes.
  • Click Save at the bottom of the page.

You will need to restart the server for the new setting to take effect.

From the Command line interface (CLI)

You can set the minimum free megabytes via Splunk's CLI. To use the CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.

Simply type:

# splunk set minfreemb 200000
# splunk restart

Set database size

Controls for indexes are in indexes.conf. You can control disk storage usage by controlling total index size, age of data in the database, and aging policy. When one of these limits is reached, data will be removed. You can archive the data using one of Splunk's predefined archive scripts or create your own. As with all configuration changes, you should make changes to this file in $SPLUNK_HOME/etc/bundles/local/ or create a new bundle.

Find this entry in indexes.conf

maxTotalDataSizeMB = (500000)
* The maximum size of an index. If an index grows bigger than this the oldest data is frozen out.
and set it to it new value (in megabytes)

Example:
[main]
maxTotalDataSizeMB = 2500000

You will need to restart the server for the new setting to take effect. It will take some time, up to 30 or 40 minutes, for Splunk to move events out of the index to conform to the new policy, during which you may see high CPU usage.

Previous: Export event data    |    Next: Use separate partitions for Splunk's datastore

Comments

  1. I would like to query splunk to see what disk-space the different indexes and buckets are currently using. This sort of page is a great place to reference this query. So far, I can't find it.
    Posted by zz2056 on May 06 2008, 5:40am

Log in to comment.