Documentation:
3.2.3
This page last updated: 08/27/08 04:08pm
Filter and re-order
Use filtering commands to filter your search results by removing duplicate results (based on user-defined constraints), using regular expression matching, using SQLite (expressions or set operations), or by specifying a time range.
Use re-ordering commands to re-order how results are displayed. Display a chosen subset of search results based on when they were found, or sort them based according to values in chosen fields.
Note: Filtering and re-ordering commands don't change the raw data within search results.
dedupRemove duplicate events from your search results. Use dedup to obtain distinct sample events from searches that return many duplicate events. dedup keeps the first occurring event for each combination field values you specify.
Note: dedup filters what events are displayed in your search results. De-duplicated events aren't removed from the index.
Syntaxdedup [keep-empty] [num] field-list [sortby clause]
Arguments| keep-empty | keepempty=T | F (F) | If set, keepempty will keep events that contain a null value. If set to false, then events with null values are discarded. |
| num | integer (default = 1) | Optionally specify the number of results (num >0) to keep for each combination of values of the specified fields. |
| field-list | field1,field2,... | Comma-delimited list of fields to remove duplicates from. |
| sortby clause | sortby field1,field2,... (list of fields) | Specify a list of fields to sort the results by. This is equivalent to executing sort before dedup. |
Splunk Web:
This example searches for all events with eventtype="useraccess", then removes duplicates of events with the same host value and reports the total count of remaining events.
* eventtype="useraccess" | dedup host | stats (count)
This example searches for all events with eventtype="useraccess", then removes duplicates and keeps three events with the same host value and reports on the total count of the remaining events.
* eventtype="useraccess" | dedup 3 source by +_time | stats (count)This data-processing command returns the first number n of specified results.
Syntaxhead [number]
Arguments| number | integer(10) | Specify the number of results to return. |
Splunk Web:
This example searches for events with the term "rotsky" on host1 and user is identified as "amrit". Then returns the first 20 events found.
rotsky AND (src="host1") AND (user="amrit") | head 20This data-processing command takes a list of search results as input, and returns a list of time ranges in which the results were found. maxspan specifies a time range that is the maximum span of time range to return. maxpause specifies a maximum pause time to wait between results. maxresults specifies the maximum number of results to localize. Use localize to process data for the page command (it allows page to process each span of time as an iteration).
Syntaxlocalize [maxspan] [maxpause] [maxresults]
Arguments| maxspan | maxspan=integer(5m)(s | m | h | d) | Specifies the maximum span of time range to return. s=seconds, m=minutes, h=hours, d=days |
| maxpause | maxpause=integer(1m)(s | m | h | d) | Specifies the maximum amount of time to allow for a pause between search results before causing a break. |
| maxresults | maxresults=integer(10000)(s | m | h | d) | Specifies the maximum number of results to feed into the command. |
Splunk Web:
This example searches for the most recent events involving "root" within 30 seconds of a "fail" event. How it works: searches for "fail", gets a 30 second window around all of the events containing "fail", pick the 5 most recent 30 second windows around "fail" events, and for each time window around a fail event, search for "root".
fail | localize maxspan=30s maxpause=30s | head 5 | map search="search root starttimeu::$starttime$ endtimeu::$endtime$"This data-processing command removes results that do not match the specified regular expression. It accomplishes this by matching the values of fields to the expression. You can specify that the regular expression keep events that match the expression, or keep those that don't match.
Syntaxregex field (= | !=) regular expression
Arguments| field | string(field name) | Field to match to the regular expression. |
| regular expression | "string" | string | A PCRE (Perl Compatible Regular Expression) supported by the pcre library to match field values to. |
Note: if you want to use the "or" ("|") command in a regex argument, the whole regex expression must be surrounded by quotes (ie. regex "expression").
ExamplesSplunk Web:
This example selects events whose _raw field contains ip addresses in the non-routable class A (10.0.0.0/8).
* | regex _raw="(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)"CLI:
This example is the same as above but shown for the CLI.
./splunk search '* | regex _raw="(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)" 'This data-processing command reverses the display order of the entire result set. Optionally specify the number of results to return by adding a number value argument.
Syntaxreverse [number]
Arguments| number= | integer (default=10) | Specify the number of results to return. |
Splunk Web:
This example searches for the term "rotsky" where the src="host1" and the user="amrit", then reverses the order that the first 20 results are returned.
rotsky (src="host1") AND (user="amrit") | reverse 20Perform set operations on sets of search results. Pass two subsearches as arguments, and return the union, diff, or intersection of the result sets..
Syntaxset set-operator [subsearch1] [subsearch2]
Arguments| set-operator | UNION | DIFF | INTERSECT | Define the set operation to perform. Union = "+", diff= "-", intersect = "^". |
| subsearch1 | string | The first search string to pass to the set operator. |
| subsearch2 | string | The second search string to pass to the set operator. |
Splunk Web:
This example return all URLs that have 404 errors or 303 errors, but not both.
index=sampledata | set diff [search 404 | fields url] [search 303 | fields url]CLI:
This example is the same as above but shown for the CLI.
./splunk search 'index=sampledata | set intersect [search 404 | fields url] [search 303 | fields url]'
This data-processing command sorts the search results by the given list of fields. It will return the original results, ordered according to the specified arguments.
If given more than one field, the first field passed will be the primary sort order(second=secondary... etc.). If the field is preceded by "+", then the sort will be ascending. If the field is preceded by "-", then the sort will be descending. If no "+" or "-" is specified, the sort will be ascending by default.
Syntaxsort [+ | -]field-list... [d | desc]
Arguments| + | +(default) | Causes the sort to be ascending. |
| - | - | Causes the sort to be descending. |
| field-list | field1,field2,... | Space or comma-separated list of fields to pass to the sort. |
| d | desc | (d | desc) | If specified, causes sort order to be reversed. |
Splunk Web:
This example sorts results by IP and then URL.
404 | sort ip, urlCLI:
This example sorts results by IP ascending and then URL descending.
./splunk search "* | sort +ip, -url"
This data-processing command returns the last n number of specified results.
Syntaxtail [number]
Arguments| number | integer(10) | Specify the number of results to return. |
Splunk Web:
This example searches for the term "rotsky" on "host1" and matches events that have the user "amrit". Only the last 20 results are returned.
rotsky (src="host1") AND (user="amrit") | tail 20This data-processing command performs arbitrary filtering on results using an SQLite WHERE clause syntax.
Note: You can now perform most of the comparison operations you had to use where for in previous versions of Splunk using the search command. where is still useful for performing statistical evaluations.
Use SQLite syntax for the arguments to where. Refer to the SQLite online documentation for a reference on SQLlite expressions.
Note: where doesn't support the SQLite commands "|" and "| |" (double-bar and single-bar). "|" is used in Splunk's search language to separate commands in a search string. Quotes should be used to signify a literal string. Numbers should not be quoted.
Note: SQLite evaluates comparison expressions with non-identical type parameters. where treats quoted literal expressions as strings ( "123" is a string, NOT a number). When where performs a non-identical type comparison, the following precedence rule applies:
- NULL < numeric type < string type (or "quoted" literal)
where filtering-expressions
Arguments| filtering-expressions | A valid SQLite WHERE clause expression. If no filter clauses are specified, results are not filtered. Quotes are optional with where. In SQLite, they are mandatory. Example: "where == 80" is the same as "where == '80'" |
Splunk Web:
This example performs a search on host="CheckPoint" on src and dest fields looking for an ip address range of: 10.9.165.x/24. Notice that where uses SQLite syntax wildcards "%" instead of Splunk search syntax wildcards "*".
host="CheckPoint" | where (src LIKE "10.9.165.%") OR (dst LIKE "10.9.165.%")CLI:
This example returns any events whose host field matches localhost.
./splunk search "* | where host="localhost""
Comments
for efficiency sake, it is preferable to use '| search <condition>' rather than '| where <condition>' as long as the condition does not involve arithmetic or other SQLite provided functions. also, '| search' supports wildcards in a much more pleasant way than SQLite: '| search src=10.9.165.* OR dst=10.9.165.*' in the above example case.
Posted by ssorkin on Mar 14 2008, 3:19pm