Generate data

Generate data

Use data-generating commands to generate data to use with other Splunk commands. Generate search results by searching the Splunk index, executing a script that creates data, or by gathering them directly from a file.

Important: When beginning your search with a data generating command, use the pipe, |, character; DO NOT begin with "* |". For example:

This also applies when you begin CLI searches with data generating commands. Additionally, if you want to view your results, use outputcsv to save it in a CSV file. For example:

./splunk search " | savedsearch foo ... | outputcsv"

./splunk search " | file /path/path/log.file ... | outputcsv"

crawl

Use crawl to search your filesystem for new data sources to add to your index. Configure one or more types of crawlers in crawl.conf to define the crawler that runs when you run | crawl. Override the default crawl settings by specifying crawl options at search time.

Learn more about Splunk's crawl feature.

Note: The crawl command produces a log of crawl activity that's stored in $SPLUNK_HOME/var/log/splunk/crawl.log.

Syntax

| crawl [crawl option]...

Note:If you have any other command ahead of crawl in a search pipeline, Splunk automatically discards the data-generated ahead of crawl and outputs data generated from crawl. For example: If you have a search command ahead of a crawl command in your search, Splunk automatically discards the search results and outputs data generated from crawl.

Arguments

Note: The default values for crawl options are found in crawl.conf.spec.

crawl options

crawl option	bad_directories_list | bad_extensions_list | bad_file_matches_list | packed_extensions_list | collapse_threshold | days_sizek_pairs | big_dir_filecount | index | max_badfiles_per_dir | root	Specify values to override key values in crawl.conf.
bad_directories_list	bad_directories_list=string, string, ...	Specify directories to exclude.
bad_extensions_list	bad_extensions_list=string,string,...	Specify file extensions to exclude.
bad_file_matches_list	bad_file_matches=(string|string*|*string|*string*|*string*string|string*string*), ...	Specify a string, or a comma-separated list of strings that filenames must contain to be excluded. You can use wildcards (examples: foo*.*,foo*bar, *baz*).
packed_extensions_list	packed_extensions_list=string, string, ...	Specify extensions of compressed files to include. Splunk unpacks compressed files before it reads them. It can handle tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z files. Leave this empty if you don't want to add any archive filetypes.
collapse_threshold	collapse_threshold=integer (default=3)	Specify the minimum number of files a source must have to be considered a directory.
days_sizek_pairs	days_sizek_pairs=integer(days)-integer(kb), ... (default= 7-0, 30-1000)	Specify a comma-separated list of age (days) and size (kb) pairs to constrain what files are crawled. For example: days_sizek_pairs_list = 7-0, 30-1000 tells Splunk to crawl only files last modified within 7 days and at least 0kb in size, or modified within the last 30 days and at least 1000kb in size.
big_dir_filecount	big_dir_filecount=integer (default=10000)	Set the maximum number of files a directory can have in order to be crawled. crawl excludes directories that contain more than the maximum number you specify.
index	index=string (default=main)	Specify the name of the index to add crawled file and directory contents to.
max_badfiles_per_dir	max_badfiles_per_dir=integer (default=100)	Specify how far to crawl into a directory for files. If Splunk crawls a directory and doesn't find valid files within the specified max_badfiles_per_dir, then Splunk excludes the directory.
root	root=directory;directory;...(default= /;/Library/Logs)	Specify directories for a crawler to crawl through.

Examples

The following command tells Splunk to browse for:

• Directories that have no more than 100 files
• Directories that were modified within the last 3 to 7 days and are between 0 to 1000kb in size

file

Use file to read a file or directory into Splunk to see what its contents would look like if it were indexed. If you specify a file to read into Splunk, Splunk will return events from the file as if it were indexed in a Splunk index. If you specify a directory, Splunk will display a list of files in the directory and their identified source types.

Note: You must use a | (pipe) when you begin searches with file.

Note: You can't use file to process archived (compressed) files or directories.

Note: If you have a search command ahead of a file command in your search, Splunk will automatically discard the search's results and output data generated from file.

Syntax

file path

Arguments

path

.../.../...

Path to the file or directory to process.

Examples

This example displays the file message.1 as if it were indexed in Splunk.

Splunk Web:

CLI:

./splunk search "| file /var/log/message.1"

metadata

The metadata command generates a list of source, sourcetypes or hosts from the index, order controls how the results will be ordered and start controls the offset from which results will be returned.

Syntax

metadata [count] [type] [order] [start] [index]

Argument	Value	Description
count	<integer>	The number of results to return.
type	source | sourcetype	Controls the type of metadata to return.
order	last | recent | total	Controls how the metadata is sorted: alpha = sort by name; last = sort by latest time; recent = sort by recent time; and total = sort by total count.
start	<integer>	Controls the offset from which the results will be returned.
index	<index_name>	Specifies the index from which to give results.

Examples

Example 1: Display 16 of the total count of hosts that exist in the _internal index.

savedsearch

Return the search results of a saved search. You can set a saved search to run using on-disk cache.

Note: The savedsearch command is not supported in distributed configurations of Splunk.

Note: You must use a | (pipe) when you begin searches with savedsearch.

Syntax

savedsearch savedsearch [savedsearch-option]

Arguments

savedsearch

string

Name of the saved search to run.

savedsearch-option

savedsearch-option	cache-option | macro-option | ttl-option | replacement-option	Options for savedsearch.
cache-option	usecache=T | F (F)	Sets whether or not to use on-disk cache for running the savedsearch.
macro-option	nosubstitution=T | F (F)	If set, turns off any macro replacements.
ttl-option	maxage=integer(60)	Specifies the number of minutes cached search results should be valid.
replacement-option	key=value	A key value pair for macro replacement.

Examples

Splunk Web:

This example runs the mysecurityquery saved search and keeps those results with a count field greater than zero.

CLI:

This example shows how you can use savedsearch with a macro (Macro search).

• Create the saved search named ABC=$rotsky$ host="petpeeve"

Run the saved search:

./splunk search "| savedsearch ABC usecache=T rotsky=loose | outputcsv"

This search creates a saved search called "ABC" that searches for host="petpeeve" whenever the macro replacement "rotsky" is specified in the savedsearch. The saved search then is run with disk cache turned on, searching for "loose" on the host is "petpeeve".

search

The search command is the core data-generating command in Splunk. It performs a search by retrieving results from the Splunk index. What you can do with search is explained in the search syntax section of the User manual.

Note: When you use Splunk Web, you don't have to type in the word search before your search terms. Type the search terms as you would in any typical search engine (ex: Google).

Syntax

search search string

Arguments

search string

search string

terms | commands | modifiers | (search string) | search string [OR] search string | comparison expression

Components of a search string.

components

terms	Search terms. Syntax for literal, quoted, and simple search term expressions.
modifiers	Search modifiers. Used to narrow searches based on various value and time constraint specification.
fields	Search fields. Fields are either indexed when your data is put into the Splunk index, or they are extracted at search time.
comparison expression	Comparison expressions that are used to compare values of fields, and constrain searches based on comparisons to other fields or numbers.

terms

terms	keywords | wildcards | literals | quotes | logical expression	Supported types of search terms.
keywords	string	Any string of characters to search for. Narrow searches based on simple character matching.
wildcards	*[string | modifiers] | [string | modifiers]* | [string | modifiers]*[string | modifiers]	Place wildcards at the beginning, middle, or ends of character strings or modifiers to broaden search constraints.
literals	+string	Precede strings with "+" to search for any string literally. Equivalent to searching for "string".
quotes	"string"	Matches the exact character string between the quotations.
logical expression	terms [AND] terms | terms OR terms | terms NOT terms | ( logical expression (logical expression ) )	Logical expressions consist of: AND (implicit between terms by white-space), OR, NOT, or parentheses (to nest expressions). Logical expressions are evaluated with the following precedence. 1. ( ) , 2. OR , 3. AND,NOT.

commands

• See the complete search command reference for syntax and description of each command.

modifiers

modifiers	search modifiers | time modifiers	Types of modifiers to constrain your search with.
search modifiers	Constrain your search based on event types, host, or other core fields. See the search modifier reference in our online user manual for details.
search modifiers	Constrain your search based on time constraints of various types. See the search modifier reference in our online user manual for details.

comparison expression

comparison expression	(literal value | field) comp operator (literal value | field)	Valid comparison expression syntax.
literal value	number	Specified number value to compare.
field	field name	Field to take values from to compare.
comp operator	= | != | < &#124 <= | > | >=	Valid comparison operators used to evaluate logical expressions of fields.

Examples

Splunk Web:

This example searches for the term "Password accepted" and filters to show results where the user is "mary".

CLI:

This example shows the top results of a search for the term "404" on "monkeyBox."

./splunk search "404 host="monkeyBox" | top uri"