Find and index data

There are several methods to get your data into Splunk. Add data via Splunk Web, Splunk's CLI, a configuration file, with scripts, or 3rd party software.

Here's a brief intro on getting data into Splunk. For more detailed instructions, follow any of the links above.

Add Data

When you first log into Splunk Web, you're given three options to begin indexing data:

Option 1: Upload a file
Option 2: Start tailing /var/log
Option 3: Start watching /etc

There are many other ways to specify data inputs in Splunk. This section is a high-level description of these techniques. For more detailed methods, see the data inputs section.

Tail a file

When you specify a file to tail, Splunk processes the entire file and then watches the file and processes additions to it. When you give a directory name to process, Splunk recursively searches all subdirectories looking for files resembling log files. You can explicitly include or exclude files with whitelisting and blacklisting.

Tailing files via Splunk Web

Click the Admin link in the upper right-hand corner.
Click the Data Inputs tab.
The first row is the Files and Directories option. Click the Add input link under the Action heading.
Under the Source heading:
- select Tail from the Data Access drop-down.
- type in the path to the file in the text box.

Tailing files via the CLI

Use the splunk add command. These commands assume you have set a Splunk environment variable. If you have not, you must navigate to $SPLUNK_HOME/bin and run the ./splunk command.

For example:

splunk add tail /var/log/

This command tails all files in /var/log/.

Find logfiles

Splunk has a built-in CLI command to search for potential log files to index:

splunk find "searchpath1;searchpath2;..."

find searches logs within the specified searchpaths. You can narrow the search by defining restrictions in $SPLUNK_HOME/etc/findlogs.ini. Restrictions can include types of directories and file to ignore, maximum file size, and modification date. After logs are found, you can index some, all, or none of the files. If you answer "Some", Splunk will prompt you file-by-file.

Previous: Change defaults | Next: Add more users

Comments

If your log rotation is altering the file (e.g. compressing) then yes Splunk will reindex the file because you are altering the CRCs of the file. If your log rotation is simply renaming the file (e.g. foo.log.1) the Splunk will detect that it has read that file and not index it. More information on how Splunk deals with log rotation can be found here: http://www.splunk.com/doc/latest/admin/logrotation

Also rather than create seperate inputs for each of the files in the directory you can create a white or blacklist rule to ensure that Splunk only indexes the files that it should be. More information on white and black list rules can be found here: http://www.splunk.com/doc/3.2.3/admin/adminfilewhiteblacklist
Posted by m@ on Jun 11 2008, 3:33pm
Note that if you use logrotate and add /var/log as an input as suggested, splunk will attempt to index any rotated logs in /var/log resulting in redundant/duplicate indexing. It is better to add specific log files, eg. /var/log/syslog to avoid this.
Posted by dennydaugherty on Jun 11 2008, 12:17pm