Search language and knowledge extensions

Search language and knowledge extensions

Dynamic field lookups

In this feature, users can create lookup tables that can be updated and used by searches (for example, mapping a list of ip addresses to mac addresses). Lookup lists can be static (stored in a csv file), dynamically generated from a search, or retrieved from an external database. Lookups typically add or replace fields in search results. Functionality for this feature includes:

• Create simple static lookup lists in a csv file, on a per application basis
• Create lookup lists from search-generated results
• Create lookup lists with values that change over time
• Conduct external database lookups via a script (ex. DNS server lookup)

Learn more about adding dynamic information to your searches in the Knowledge Manager Manual.

Relative time modifiers

In addition to Splunk's standard built-in time ranges relative to search time (last day, last week, last month etc.), Splunk now allows users to specify more flexible time ranges. These additional time periods include those that are:

• Relative to current the current calendar day, week, month or year (ie Today, this week, this month, this year, etc.)
• Relative to a previous period (ie Yesterday, previous week, previous month, previous year, etc.)
• Relative to a business week (ie Previous business week, this business week, etc.)

Administrators can configure these relative time ranges for individual apps via a configuration file.

Learn more about using relative time modifiers in the User Manual.

Improved and new search commands

This feature includes new search commands, as well as enhancements to existing search commands:

• The 'convert' command is used to change the format of field values at search time. 'Convert' now allows you to remove commas from real numbers
• The 'eval' command performs an arbitrary expression evaluation, including mathematical and Boolean evaluations. 'Eval' now allows:
  • Faster search-time calculations within Splunk
  • More precise calculations, with support for scientific calculations
  • Improved string comparison, matching, and replacement
• The 'where' command performs arbitrary filtering on your data. 'Where' no longer uses strict SQLite syntax, but now includes support for a larger number of SQLlite-like functions
• Search for IP addresses in a particular subnet using CIDR syntax
• The 'dedup' command removes duplicate events or field values from your search results. Now, you can specify that 'dedup' remove events or fields only if they are consecutive
• The 'addtotals' command computes row totals, or the arithmetic sum of the all the numeric fields for each event. Now, you can compute column totals with 'addtotals'
• New delta" command which computes the numerical difference between a field's current value and its previous value
• New "streamstats" command which adds summary statistics to all search results in a streaming manner
• New "accum" command that calculates the running total for a given numeric field
• New "analyzefields" command which analyzes numerical fields for their ability to predict another discrete field
• New "trendline" command which computes the moving averages of fields
• New set of accelerator commands that make it easier to populate summary indexes
• Ability to create search language macros that modularize search expressions and allow them to take arguments

Learn more about new and improved search commands in the Search Reference Manual.

Improvements to field renaming, tagging, and aliasing

This feature includes several improvements to Splunk's handling of fields and sourcetypes:

• Rename sourcetypes to either an entirely new or existing sourcetype (replaces 'sourcetype aliasing' from previous versions)
• Ability to tag sourcetypes
• Create an alias of a field (allows easier support for common information model)
• Search for eventtypes and tags with wildcards

Learn more about aliasing fields in the Knowledge Manager Manual.

Learn more about tagging fields in the User Manual.

Advanced anonymization

In previous versions of Splunk, masking confidential data within events required administrators to write custom transforms. Splunk now supports the same syntax as 'sed', a Unix utility which reads a file and modifies the input as specified by a list of commands. Capabilities include:

• Anonymization and replacement of strings based on regular expression matching
• Substitution of characters within log data

Benefits

For users:

• Instead of running sub-searches to retrieve information, users can now store information in a list where it is available for any number of searches and can be reported on. For example:
  • A scheduled search might periodically be looking for offending entities in the log files and then store them in a blacklist
  • Another scheduled search could then check this list on a frequent basis to monitor what these blacklisted entities are doing.
• Users can also lookup values that may be contained in an external database
  • For example, users could translate numerical ip address to dns names by performing lookups through a DNS server
• In the case that Splunk is misconfigured or a sourcetype is incorrectly applied during indexing, users can quickly change the sourcetype label without having to re*index data
• Ability to tag sourcetypes gives app developers more flexibility in writing searches
• Ability to alias fields adds the capability to use different namespaces for field names
  • For example, users can make field names compatible with other namespaces, such as CEE (cee.mitre.org)
• The ability to use wildcards when searching on eventtypes and tags makes constructing complex searches easier
• Searches that specify full calendar days, weeks, or months in the time range allow for easier historical comparisons within reports (ie how am I tracking today, as compared to yesterday, and the day before yesterday)
• Searches limited to business week activity allow users to better display and analyze data for only relevant time periods (ie What does web traffic look like during business hours?)
• Easier for administrators to mask and replace confidential data such as social security numbers, credit card numbers, passwords, etc. at index time using the more commonly known 'sed'-like syntax