How search commands work

How search commands work

A Splunk search consists of one or more data-generating commands and their arguments, which can include literal keywords, wildcards, Boolean expressions, field name and value expressions, and subsearches. The generated data (search results) can then be used as inputs into other search commands in a search pipeline.

For the most part, search commands fall into categories based on what they do, such as: filter unwanted information, extract more information, evaluate your data, transform your data into statistical results, and reorder your results. The specific commands themselves may fit more than one category depending on the arguments you use. Refer to the Search Reference manual for the complete list of search commands.

To better understand how search commands act on your data, it helps to visualize all your indexed data as a table. Each search command redefines the shape of your table. This topic illustrates how the different types of search commands act on your data.

Also, if you want to just jump right in and start searching, the Search command cheatsheet is a quick reference complete with descriptions and examples.

Note: This topic assumes you have indexed data. For more information read About data and indexes in the User Manual.

Start with a table of indexed data

Before you search, think of your indexed data as a table. In this table, each indexed event is a row. Each of these events contain indexed or extracted fields, which are name and value pairs of information. In this table, the field names are columns, and the field values are the individual cells in each row.

This beginning table includes columns for the default fields Splunk automatically adds to the data. These default columns are followed by columns for all other extracted fields. The following is an example of a beginning table for an arbitrary number of events and fields.

_raw	_time	host	sourcetype	source	field 1	field 2	...	field y
event 1	time 1	host 1	sourcetype 1	source 1	field 11	field 12	...	field 1y
event 2	time 2	host 2	sourcetype 2	source 2	field 21	field 22	...	field 2y
⋮	⋮	⋮	⋮	⋮	⋮	⋮	...	⋮
event x	time x	hsot x	sourcetype x	source x	field x1	field x2	...	field xy

Search at the beginning or elsewhere

You can search at any point in the search command pipeline. A search results in a smaller table that contains the exact same number of columns minus the rows of events that did not match the search conditions. Searches do not change any cell values.

Searching commands: crawl, file, savedsearch, search.

Example: Search for matching host.

Let's say you have this beginning table:

time	host	sourcetype	source
2008-11-03T14:08:16-0800	host1	syslog	syslog.log
2008-11-03T14:08:16-0800	http1	access_common	http1access.log
2008-11-03T14:08:15-0800	http2	access_common	http2access.log
2008-11-03T14:07:00-0800	host1	syslog	syslog.log
2008-11-03T14:04:00-0800	http2	access_common	http2access.log

You want to find all the HTTP servers in your events:

_time	host	sourcetype	source
2008-11-03T14:08:16-0800	http1	access_common	http1access.log
2008-11-03T14:08:15-0800	http2	access_common	http2access.log
2008-11-03T14:04:00-0800	http2	access_common	http2access.log

Filter unwanted information

Filtering commands produce the same results as a search: a smaller table. However, depending on the search command, the smaller table may have fewer rows or fewer columns. Filtering commands also do not change any cell values.

Filtering commands: dedup, fields, head, localize, regex, search, set, tail, where.

The following 3 examples use the same beginning table from the previous search example.

Example: Remove duplicates of cell values in a column with dedup.

You want to remove duplicate events based on the hostname:

_time	host	sourcetype	source
2008-11-03T14:08:16-0800	host1	syslog	syslog.log
2008-11-03T14:08:16-0800	http1	access_common	http1access.log
2008-11-03T14:08:15-0800	http2	access_common	http2access.log

Example: Remove or keep columns with fields.

You want to see only the host and sourcetype information:

host	sourcetype
host1	syslog
http1	access_common
http2	access_common
host1	syslog
http2	access_common

Example: Remove all rows after the number specified with head.

You want to see only the first three results of your search:

_time	host	sourcetype	source
2008-11-03T14:08:16-0800	host1	syslog	syslog.log
2008-11-03T14:08:16-0800	http1	access_common	http1access.log
2008-11-03T14:08:15-0800	http2	access_common	http2access.log

Evaluate your data

Evaluating commands can change specific column names or cell values. Depending on the command, evaluating commands may or may not add columns.

Evaluating commands: abstract, addtotals, bucket, cluster, collect, convert, correlate, diff, eval, eventstats, format, fillnull, format, kmeans, makemv, mvcombine, mvexpand, nomv, outlier, overlap, replace, strcat, transaction, typelearner, xmlunescape.

The next example uses this beginning table; each succeeding example builds on it.

host	sourcetype	count1	count2
host1	syslog	200	80
http1	access_common	300	80
http2	access_common	300	60
host2	syslog	200	90
http2	access_common	300	60

Example: Create a new column where the cells are the results of an eval expression.

You want to create a new field for the sum of count1 and count2 values.

host	sourcetype	count1	count2	sum
host1	syslog	200	80	280
http1	access_common	300	80	380
http2	access_common	300	60	360
host2	syslog	200	90	290
http2	access_common	300	60	360

Example: Change one or more column names with rename. This does not create a new column.

Using the previous resulting table, you want to change the column name of sum to total

host	sourcetype	count1	count2	total
host1	syslog	200	80	280
http1	access_common	300	80	380
http2	access_common	300	60	360
host2	syslog	200	90	290
http2	access_common	300	60	360

Example: Overwrite cell values with replace. This does not create a new column.

Using the previous resulting table, you want to change all host values that are host1 to localhost.

host	sourcetype	count1	count2	total
localhost	syslog	200	80	280
http1	access_common	300	80	380
http2	access_common	300	60	360
host2	syslog	200	90	290
http2	access_common	300	60	360

Example: Create new columns for the concatenated string value of other columns with strcat.

Using the previous resulting table, you want to add a new column called hosttype that combines the host and sourcetype values, separated by a hyphen.

host	sourcetype	count1	count2	total	hosttype
localhost	syslog	200	80	280	localhost-syslog
http1	access_common	300	80	380	http1-access_common
http2	access_common	300	60	360	http2-access_common
host2	syslog	200	90	290	host2-syslog
http2	access_common	300	60	360	http2-access_common

Reorder your results

Reording commands sort the rows of the entire table based on the values of the specified column name. These commands do not add or remove rows and do not change any cell values.

Reordering commands: reverse, sort.

Example: Reorder the table with sort.

Using the previous resulting table, reorder the rows in ascending order of total.

host	sourcetype	count1	count2	total	hosttype
localhost	syslog	200	80	280	localhost-syslog
host2	syslog	200	90	290	host2-syslog
http2	access_common	300	60	360	http2-access_common
http2	access_common	300	60	360	http2-access_common
http1	access_common	300	80	380	http1-access_common

Extract more information

Extracting commands create new rows or columns from information found in the _raw column for each row.

Extracting commands: addinfo, extract/kv, iplocation, multikv, rex, top, typer, xmlkv.

Example: Create new columns from key/value pairs in your events with extract/kv.

Example: Create new rows from information found in multi-line or tabular events with multikv.

Transform your data into statistical results

Transforming commands creates an entirely new table of data. These commands change the specified cell values for each event into numerical values that Splunk can use for statistical purposes.

Transforming commands: chart, contingency, highlight, rare, stats, timechart, top.

Example: chart