dedup
This documentation applies to the following versions of Splunk: 4.0 , 4.0.1 , 4.0.2 , 4.0.3 , 4.0.4 , 4.0.5 , 4.0.6 , 4.0.7 , 4.0.8 , 4.0.9 , 4.0.10
dedup
Synopsis
Removes the subsequent results that match specified criteria.
Syntax
dedup [<N>] <field-list> [keepevents=<bool>] [keepempty=<bool>] [consecutive=<bool>] [sortby <sort-by-clause>]
Arguments
- consecutive
- Syntax: consecutive=<bool>
- Description: Specify whether to only eliminate duplicate events that are consecutive (true). (default, false).
- <field-list>
- Syntax: <string>, ...
- Description: Comma-delimited list of field names.
- keepempty
- Syntax: keepempty=<bool>
- Description: If an event contains a null value for one or more of the specified fields, the event is either retained (T) or discarded (default, F).
- keepevents
- Syntax: keepevents=<bool>
- Description: When true, keeps all events and removes specific values. By default, keepevents=false.
- <N>
- Syntax: <int>
- Description: Specify the first N (where N > 0) number of events to keep, for each combination of values for the specified field(s). The non-option parameter, if it is number, is interpreted as N.
- <sort-by-clause>
- Syntax: ( - | + ) <sort-field>
- Description: List of fields to sort by and their order, descending ( - ) or ascending ( + ).
Sort field options
- <sort-field>
- Syntax: <field> | auto(<field>) | str(<field>) | ip(<field>) | num(<field>)
- Description: Options for sort-field.
- <field>
- Syntax: <string>
- Description: The name of field to sort.
- auto
- Syntax: auto(<field>)
- Description: Determine automatically how to sort the field's values.
- ip
- Syntax: ip(<field>)
- Description: Interpret the field's values as an IP address.
- num
- Syntax: num(<field>)
- Description: Treat the field's values as numbers.
- str
- Syntax: str(<field>)
- Description: Order the field's values lexigraphically.
Description
Keep in the first N (where N > 0) events for each combination of values for the specified field(s) The non-option parameter, if it is number, is interpreted as N. If this number is absent, N is assumed to be 1. Optionally, sort by some list of fields. Equivalent doing a sort command before the dedup command except that it is executed more efficiently. Optionally, keep all events, but for events with duplicate values, remove those values instead of the entire event.
Examples
Example 1: Remove duplicates of results with the same host value.
... | dedup host
Example 2: Remove duplicates of results with the same source value and sort the events by the '_time' field in ascending order.
... | dedup source sortby +_timeExample 3: Remove duplicates of results with the same source value and sort the events by the '_size' field in descending order.
... | dedup group sortby -_sizeExample 4: For events that have the same 'source' value, keep the first 3 that occur and remove all subsequent events.
... | dedup 3 source