Massively scalable search

Massively scalable search

Not only is search about ten times faster than the previous release, but we've added several new features that empower users to search smarter and faster; a few of these features are highlighted in this Splunk developer's blog post. Read on for more information.

Analyze large data sets

This feature allows users to run reports over hundreds of millions of results and terabytes of data from within Splunk's new user interface. While previous versions of Splunk allowed users to perform searches over 50,000 results from the command line (the 'dispatch' command), this capability has been enhanced and has been now incorporated into the new user interface. As part of this integration with the new user interface, users no longer need to wait for searches to complete. They can interact with results as as search runs in real-time, add/remove fields, and page through results.

Other scaling-related enhancements include:

• Ability to search across multiple indexes
• Ability to import/export indexes to compressed csv files
• Better control of "live" splunk data files, allowing for more granular data retention control through time-based settings

For information about Splunk's powerful search tools, refer to "About search"in the User Manual

New highly responsive user interface

This feature includes a completely redesigned and more intuitive core Splunk search interface. Capabilities include:

• Completely redesigned look and feel of Splunk's core search interface
  • Faster rendering of events and immediate feedback on search progress
  • Preview fields and top values as a search is running
  • Preview statistical reports while a search is running
  • Improved ability to zoom in/out on data from a timeline
  • Advanced time range selector that allows users to manually choose subsecond-level time ranges and relative time ranges
  • New field picker that allows users to quickly search through thousands of fields, sort fields by frequency, and set the ordering of fields within events
• Support for more advanced and dynamic form searches (also see the Easy UI Customization feature)
• Quick access to multiple search "views" and dashboards (also see the Easy UI Customization feature)
• Ability to organize large numbers of saved searches into folders
• Ability to add descriptions to saved searches
• Ability to view events in their raw format or as a table
• Improved (faster) type-prediction from within the search box
• More detailed and predictive search command help from within the search box
• Improved control over the number of lines shown in events
• Ability to switch between linear and log scale in the timeline
• Support for subseconds in events and the timeline

Learn more about using Splunk's redesigned core search interface.

Job management and control

This feature allows users and administrators greater flexibility in managing more concurrent searches and longer running searches.

Through Splunk's new job management user interface, users and administrators will be able to:

• Background, pause, finalize, or cancel search jobs
• Get a list of jobs and sort them job type, owner, status, and expiration
• Switch to a currently running or previously run job and bring it to the foreground
• Save or delete an existing job
• Access previously completed searches and report on them without having to rerun a search
• Manage the jobs of all Splunk users (Administrators only)

Additionally, search jobs now run in a separate process, allowing searches to run independently of the Splunk indexer. Individual search jobs will be accessible to the administrator directly from the user's operating system.

Learn more about managing your search jobs.

Faster complex searches

This feature includes significant back-end improvements to Splunk's search speed, especially for more complex searches. Improvements include:

• Get results back up to 10x faster (depending on search length) due to optimizations made to handle complex boolean expressions and more efficient key-value extraction
• Splunk's transaction processor will typically run up to 100% faster compared to previous versions, work over hundreds of millions of events, and work efficiently in distributed environments
• Improved splunk startup speed in clustered file systems

Faster distributed search

This feature includes significant back-end improvements to Splunk's performance in distributed environments. Improvements include:

• Search loads are more evenly distributed over Splunk servers (map-reduce optimizations), increasing search speed substantially
• Users can specify a set of distributed servers to search using the search language
• Transaction processing now scales to hundreds of millions of events over distributed architectures

Learn more about using distributed search.

Benefits

For users:

• Run statistical analysis and summary reports over long periods (months, or even years) over massive amounts of search results (hundreds of millions) on an ad hoc basis
• The ability to report directly from single searches over massive datasets simplifies analyzing large volume sources such as firewall, web traffic, and business intelligence data
• The ability to run multiple concurrent searches and switch back-and forth between them improves time to problem identification and repair
• Pausing or finalizing searches allows users to analyze intermediate results without having to wait for a search to complete
• Accessing results from previous searches gives users the flexibility to report on long running searches at a later date or share them with other users
• Users can immediately begin analyzing search results before a search is completed, allowing for more rapid problem resolution and data correlation
• New field picker allows users to search through and select from hundreds of fields that may be embedded in large, complex data sets
• Faster loading of events and event progress monitor improves the overall product interaction
• Users can group and quickly find relevant saved searches by assigning search descriptions and organizing them through tags
• Advanced time picker enables users to select from relative time ranges (today, this week, last business week, etc.), as well as select manual time ranges (with subsecond support) without needing to know advanced search language syntax
• Improved type-prediction and search command help makes it easier for new users to learn Splunk commands
• Reduce time to problem resolution through a faster search experience
• Improved speed of "transaction" command allows for the broader usage of transactions analysis over larger datasets
• Reduce time to problem resolution in larger, geographically distributed environments through a faster distributed search experience
• Improved configurability over distributed search defaults on a per user basis can reduces unnecessary server load and improve search response time
• Improved speed of 'transaction' command allows for the broader usage of transactions analysis over larger data sets

For administrators:

• Easily import and export index data out of Splunk for use in other applications, thereby making it easier to add, edit, and delete data from Splunk indexes
• The ability to control what Splunk considers 'live' data by time gives administrators greater control over Splunk disk usage and data retention
• Manage server load across many Splunk servers, users and searches
• Manage individual searches from the OS level and assign job priority