This document last updated: 07/03/08 03:07pm

Print User Manual

About this manual

What's in this guide?

This manual teaches end users about Splunk and how to use Splunk. To help you start using Splunk, run through the Tutorial.

Before using Splunk, review the Installation manual for system requirements and step-by-step installation guides.
System administrators should refer to the Administration and Deployment manuals for advanced configuration and data management topics.
Those interested in customizing Splunk and creating applications for Splunk should refer to the Developer manual.

You can share custom Splunk applications with other members of the Splunk community via SplunkBase.

About Splunk

Splunk is an IT search engine.

You can use Splunk to search and navigate IT data from applications, servers, and network devices in real-time.

Data sources include logs, configurations, messages, alerts, scripts, code, metrics, etc.

Splunk lets you search, navigate, alert, and report on all your IT data in real-time using Splunk Web.

About Splunk licenses

Each instance of Splunk server must have its own license. This topic discusses the different Splunk licenses, how to install or update a license, and what to do when you have a violation on your license.

Note: You must purchase a separate license for every instance of Splunk that you deploy.

Which license?

Splunk provides two standard types of licenses, a Free license and an Enterprise license. To evaluate Enterprise features, you can request a trial Enterprise license before purchasing.

Note: If you evaluate a Splunk Preview release, it will include the required license.

Free versus Enterprise

When you download Splunk for the first time, you are asked to register. Your registration authorizes you to receive the Free license, which allows a maximum indexing volume of 500 MB/day. The Free license is not a trial license and does not have an expiration date. The Enterprise license enables higher data indexing volume and the following additional features:

Multiple user accounts and access controls.
Distributed search and data routing.
Deployment management.

To evaluate these features before you purchase an Enterprise license, you can request a 30-day trial Enterprise license.

Find more information about the different license features here. Also, read Splunk's Free license agreement.

Trial license

You can request trial Enterprise licenses of varying size and duration. The default evaluation period is 30 days. If you are running with a trial license and your license expires, Splunk continues to index your data. However, you will not be able to search until you install a new license.

Preview license

Splunk's Preview releases require a different license that is not compatible with other Splunk releases. Also, if you are evaluating a Preview release of Splunk, it will not run with a Free or Enterprise license. Preview licenses typically enable Enterprise features, they are just restricted to Preview releases.

Forwarding license

Each instance of Splunk server must have its own license. To configure Splunk for forwarding and receiving, contact Splunk Support and request a 1 MB/day license to install on each forwarder instance (any instance that is not indexing locally). For additional security, this license lets you configure different username and password pairs on each forwarder.

Note: This 1 MB/day license is not subtracted from your existing license(s) and can be applied to multiple forwarders.

Install your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license (splunk-free.license) or an Enterprise license (splunk.license). You can install and update your licenses with the CLI or from Splunk Web's Admin > License & Usage page.

Refer to the Installation Manual for instructions to install or update your Splunk license.

License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have more than 7 violations in a rolling 30-day period, search will be disabled. Search capabilities return when you have less than 7 violations in the previous 30 days or when you apply a new license with a larger volume limit.

Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks access while you exceed your license.

If you have other issues with your license, refer to the Administration Manual for troubleshooting tips.

Use Splunk Web

About Splunk Web

Splunk Web is Splunk's graphical user interface. It runs off of the splunkweb process, which is a Python-based application server. Use Splunk Web to search your IT data and manager your Splunk deployment. Access Splunk Web via a Web browser. Refer to the system requirements for our list of supported operating systems and browsers.

Splunk Web contains dashboards and configuration pages. You can run a custom search from any of the dashboards. Access the Preferences panel and the Admin pages with links on the top right corner of the dashboards, above the search bar. Access different dashboards from a drop-down menu located under the search bar and on the right.

Dashboards

Dashboards are customizable pages in Splunk Web. You can add and remove components to and from each dashboard. These components may be lists of all indexed data, snapshots of different saved searches, or a list of saved searches.

Splunk ships with three default dashboards: getting started, main, and admin.

Getting started

The getting started dashboard is the default landing page for Splunk Web. It provides information and links to help new users learn how to use Splunk. There are many upcoming changes to the getting started dashboard. In 3.3, you can now watch feature videos in Splunk Web and add files and more data to your indexes, directly from the getting started dashboard.

Watch Splunk feature videos

When you first log into Splunk Web, the Learn about Splunk window displays over the getting started dashboard. This window provides demo videos of Splunk features for viewing. If you don't want to view this window next time you start Splunk, check the "Do not show this window on startup". When you're ready to begin using Splunk, click "Close this window and start using Splunk".

Note: After closing the Learn about Splunk window, you can open it again from the link under Get help on the getting started dashboard.

Index some data

In 3.3, the buttons for indexing data take you to the index manager:

Clicking Index Files redirects you to the Admin > Data Inputs: Files & Directories: New Input page. You can enter Source, Host, and Source Type information to define local or remote files and directories to index.
Clicking Index More Data redirects you to the Admin > Data Inputs: All page. You can select the type data to configure and add to your inputs.

Read more about the index manager and adding inputs.

Main

The main dashboard provides default modules, which include:

Lists of all indexed data, sorted by Sources, Sourcetypes, and Hosts.
A timeline of errors in the last hour.
A list of saved searches.

Admin

The admin dashboard provides graphs that report information a Splunk administrator may find useful:

Messages by minute in the last 3 hours.
The volume of data (KB/hr) over the last 24 hours.
The number of Splunk errors over the last 24 hours.
The daily indexing volume by server.

Custom dashboard

Instead of editing the default dashboards, we recommend creating a new dashboard to customize. From the dashboard drop-down menu, select create new dashboard... to name your new dashboard and add it to Splunk Web. Use the Edit and Delete options, located next to the drop-down menu, to customize or remove dashboards from Splunk Web.

Preferences

Use the Preferences panel to configure Splunk Web's default search properties and general appearance and behavior. For more information, read Change Splunk Web preferences.

Search

Use the Search preferences tab to define:

The default time range; You can always change the timerange from the dashboard.
A maximum limit for the number of events Splunk indexes when you search.
How Splunk Web handles segmentation.

Note: Splunk Web's segmentation setting has nothing to do with indexing segmentation. This setting affects how the browser interacts with Splunk and may speed up the display of search results.

General

Use the General preferences tab to define:

A default theme for Splunk Web. If you want to customize Splunk Web's appearance, refer to the Developer manual.
Click behavior. You can click on sections of your search results to add or replace terms in your search. "Click behavior" configures either ctrl or ctrl-click to add and replace when narrowing your search.

Admin pages

In 3.3, when you click on the Admin link, to the top right of the page, the Server settings page opens. Instead of navigating a tabbed menu layout, you now access the Admin pages from a list located on the left side of the page. Click on the top-level section names to view the pages included in that section. You have access to the same pages as before (Server, Data Inputs, Distributed, Users, Saved Searches, and License & Usage) with the addition of Indexes and Applications.

Server

Use the Admin > Server pages to view and change server settings, restart the Splunk server, and change and reload Splunk's authentication method. Read more About Splunk server's settings and changing Splunk server's settings.

Data Inputs

Use the Admin > Data Inputs pages to add new and edit existing inputs in Splunk Web. You can view and manage all of your files and directories, FIFO queues, network ports, and crawls from this page. Read more About Inputs and using Data Inputs to add inputs.

Indexes

Use the Admin > Indexes pages to view a list of your indexes, edit individual index properties, and add new indexes. Read more About Indexes.

Applications

Use the Admin > Applications pages to manage existing applications and browse SplunkBase for new applications to install. Read more About Applications.

Distributed

Use the Admin > Distributed pages to view your network topology and configure search distribution, data forwarding, and data receiving between multiple Splunk instances. Read more About Data Distribution.

Note: You can only set up forwarding from this page if you are running Splunk with a Free license. To configure distributed search and data receiving, you must have an Enterprise license.

Users

Use the Admin > Users page to view a list of users and their search history, edit each user's properties, and add new users. Read more about Users and User Roles.

Note: You cannot access this page if you are running Splunk with a Free license; you must have an Enterprise license to modify user's properties.

Saved Searches

Use the Admin > Saved Searches page to view a list of your saved searches and edit their properties, create new searches, or delete existing searches. Read more about Managing saved searches.

License & Usage

Use the Admin > Licenses & Usage pages to view view your current license and replace it with a new one. This page displays the type of license you're running, the maximum indexing volume allowed, and when the license expires. This page also provides some useful statistics, such as: number of days before you need to renew, the peak usage in GB/day, and peak percentage. Read more About Licenses.

Change Splunk Web preferences

Use the Preferences panel to configure Splunk Web's default search properties and general appearance and behavior. The Preferences panel is a dialog box that opens when you click the Preferences link in the upper righthand corner of Splunk Web. Access the link on any of the dashboard pages.

The Preferences panel has two tabbed options: Search and General.

Search

Use the Search preferences tab to change:

The default time range.
The maximum search results.
How Splunk Web handles segment selection.

Change default time range

To set the default time range for search, select one of the following options:

Last 15 minutes
Last 60 minutes
Last 3 hours
Last 24 hours
Last 7 days
Last Month
Last 3 Months
All time

You can always change the time range at search time from the dashboard.

Change maximum search results

The maximum search results is the maximum limit for the number of events splunk indexes when you search. By default, it is 50000. Increasing or decreasing this number will affect Splunk's search speed.

Change segment selection

Splunk Web's segmentation setting has nothing to do with indexing segmentation. This setting affects how the browser interacts with Splunk and may speed up the display of search results.

You can set segment selection to:

Raw: Disables segmentation entirely. This is the fastest setting, but disables clicking on events.
Inner: Enables only minor breakers.
Outer: Enables only major breakers.
Full: Default setting, enables both inner and outer segmentation.
Pyramid: Enables visualizing of event's segmentation.

General

Use the General preferences tab to define:

A default theme for Splunk Web.
Click behavior.

Change default theme

Splunk Web is defined with HTML, CSS, Javascript, and XSL. You can customize a theme with CSS files to override the default styles for font, color, and images. If you want to customize Splunk Web's appearance, refer to the Developer manual.

Change click behavior

You can click on sections of your search results to add or replace terms in your search. "Click behavior" configures either ctrl or ctrl-click to add and replace terms when narrowing your search.

About Splunk server settings

Use the Admin > Server pages to view and change the Splunk server settings, restart the Splunk server, and change and reload Splunk's authentication method.

Important: When you change any of the server settings, you must restart Splunk for your changes to take effect.

View server settings

The Admin > Server: View Settings page is divided into three sections. Refer to Change Splunk server default settings for instructions on changing these settings.

Basic settings

Under the Basic Settings heading, you can change your server name and splunkd port.

The Splunk server name, which is the identity of your Splunk instance, defaults to the DNS or IP address of the Splunk sever host.
Splunk management port number, though which your Splunk instance communicates with the splunkd daemon, defaults to port 8089.

Note: You cannot modify the installation path.

Splunk Web settings

Under the Splunk Web heading, you can enable or disable Splunk Web, enable SSL (HTTPS) in Splunk Web, and change the Web port.

By default, Splunk Web is enabled. However, in a distributed environment, you may choose to disable Splunk Web on your forwarding instances.
By default, SSL is not enabled.
The default Web port number is 8000.

Datastore settings

Under the Datastore heading, you can change the default host name, the datastore path, and the minimum free disk space.

By default, your Splunk server name is your Default host name. This name defines the value for the host field in your search results; it tags all events with this host value.
The datastore location is the top-level directory where the Splunk server stores all indexed data, user information, and working files. By default, this path is set to $SPLUNK_HOME/var/lib/splunk.
The minimum free disk space is a value that tells Splunk to stop indexing if the storage space reaches this limit (in MB). The default disk space is 2000 MB.

Control server

Use the Admin > Server: Control Server page to restart the Splunk server and reload the Authentication method.

Note: Unless you are running Splunk with an Enterprise license, you will not see the Reload Authentication method option.

Configure authentication method

Use the Admin > Server: Authentication Configuration page to change Splunk's authentication method.

Note: This feature requires an Enterprise license. If you are running Splunk with a Free license, Splunk Web will tell you how to update your license.

Change Splunk server default settings

After you start a new installation of Splunk, you may want to change a number of the default settings. This is a quick guide for using Splunk Web to change the admin default password, your Splunk server name, the Web and splunkd network ports, the datastore location, and the minimum free disk space.

You can make all of these changes from the Admin > Users and Admin > Server: View Settings pages in Splunk Web. Refer to the User Manual for more information About Splunk Web.

Note: You can also make these changes using Splunk's CLI.

Change the password

Splunk with a Free license does not require login authentication. However, Splunk with an Enterprise license does require authentication and ships with a default administrator account with username admin and password changeme.

To change the administrator password:

1. Navigate to the Admin > Users page in Splunk Web.

2. For the username admin, click Edit from the Action column.
The admin user properties page opens.

3. Enter your new password twice (under Password and Confirm Password).

4. Click Save.
You return to the Admin > Users page with a note at the top, "User admin updated."

Change Splunk server name

The Splunk server name is the identity of that particular instance. This name is displayed within Splunk Web and is sent to other Splunk servers in a distributed setting. The default name is taken from either the DNS or IP address of the Splunk server host.

To change the Splunk server name:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Basic Settings section, enter a new Splunk server name.

3. Click Save.

Change network ports

Splunk uses two network ports that default to:

8000: HTTP or HTTPS socket for Splunk Web.
8089: Splunk management port.

Note: Splunk Web, the command line interface, and any distributed connections from other servers use the Splunk management port to communicate with the splunkd daemon.

To change the network ports:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Basic Settings section, enter a new Splunkd port number (under Splunkd port #).

3. In the Splunk Web section, enter a new Web port number (under Web port).

4. Click Save.

Change datastore location

The datastore path is the top-level directory where the Splunk server stores all indexed data, user information, and working files. If you turn off local indexing and only forward data in a distributed setup, this server still requires a few megabytes of available space in the datastore path.

Important: If you change this directory, the server does not migrate old datastore files. Instead, it starts over in the new location. To migrate your data to another directory, refer to the instructions in Move an index.

To change the datastore location:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Datastore section, enter a new Datastore path.

3. Click Save.

Change minimum free disk space

The minimum free disk space defines a storage space limit before Splunk stops indexing. Splunk resumes indexing when more space becomes available. For more information on managing Splunk disk usage, see Disk usage.

To change the minimum free disk space:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Datastore section, enter a new limit under "Pause indexing if free disk space falls below".

3. Click Save.

Tutorial

About this tutorial

This section briefly explains what you need to run Splunk and complete this tutorial.

Requirements

A supported browser (See system requirements and release notes).
A copy of Splunk:
- On an individual machine, or elsewhere in the organization. (Download and install.)
- On Splunk hosted demo servers.

Log in

Splunk does not require a login when using a Free license. An Enterprise license requires a login:

For an individual server, the default Splunk username is admin and password is changeme.
For the Splunk hosted demo server, the username and password are guest and guest.

Splunk Web

Splunk Web runs by default on port 8000 of the host on which it is installed.

If you're running Splunk on your local machine, the URL to access Splunk Web is http://localhost:8000
If you're running Splunk on a different machine, the URL to access Splunk Web is http://<hostname>:8000

where <hostname> is the name of the machine Splunk is running on.

Logging in to Splunk takes you to the dashboard and Splunk Web. For this tutorial, you only need to know that:

You type your search query in the search bar at the top of the dashboard.
- The down-arrow button, to the left of the search bar, is a drop-down menu.
- The green button, to the right of the search bar, enters your search.
You change the time range with the drop-down menu below the search box.
Splunk displays the number of occurrences of your search results graphically on the timeline.
Splunk lists each individual event that results from your search underneath the timeline and ordered by timestamp.

Read the other sections of the Splunk User Manual for more information.

Command line interface (CLI)

Splunk includes a command line interface (CLI) that runs from a shell on the server host. The Splunk CLI is a great way to integrate Splunk into admin scripts.

Read Use the Splunk CLI for more information.

Simple searches

See a few simple searches in action.

Index data

Splunk comes with pre-indexed sample data, called sampledata, which we will use throughout this tutorial. You can search the index that has the sample data in it instead of the main index by including index=sampledata in the search criteria.

For help indexing your own data, see the data inputs section of the Admin Manual.

Search

To start, enter your search in the search bar at the top of the page. To search for all the data in the sampledata index, type the following into the search bar:

index=sampledata

The timeline should show bars indicating when matching results occurred. If there are no results displayed, change the time range until you see results.

Now, lets search for HTTP requests that resulted in an internal server error (code 500). Type this simple search:

index=sampledata http 500

Narrow your search

You can use arguments in the search command to narrow your search. Add Boolean logic between terms and modifiers, use logical comparison operators for field values, or use search modifiers. You can also use the timeline to zoom in on particular events. This section discusses two ways to apply Boolean modifiers to your search. We'll discuss using the timeline to narrow your results later.

Read Search results for more ways to manipulate search results.

Search with Booleans

Splunk supports the Boolean operators: AND, OR, and NOT (must be capitalized).

Enter the search:

index=sampledata http AND 500

Your results should match the previous example search. Similar to Google and other search engines, Splunk implicitly inserts an AND between terms by default.

Note: If your search produces no results, try zooming out, clearing the time range, or resetting the time range using the drop-down menu.

Search for all HTTP requests that do not contain error code 500:

index=sampledata http NOT 500

Search for all HTTP requests that do not contain error codes 500 or 503 (service unavailable).

index=sampledata http NOT (500 OR 503)

Note: Splunk uses parentheses to group Boolean expressions.

Click on results

As you scroll through your results and mouse over sections of each event, you'll notice the sections are highlighted. You can highlight and click items in the results to add and remove terms in your search string.

Search for:

index=sampledata http

Scroll through the list of results. Click on "500" in one of the search results.

Notice that Splunk highlights and updates the search to add "500" as a term (in the search bar). This is a shortcut for applying the "AND" operator to your search.

index=sampledata http 500

Click on another instance of "500". Splunk removes the term from your search string and your search results include all HTTP results again:

index=sampledata http

Now, alt-click on "200" in any search result (option-click for Mac, shift-click for some popular *nix windows managers).

Splunk now updates your search with "NOT 200"; This is a shortcut for applying the "NOT" operator.

index=sampledata http NOT 200

Follow a relationship

While you scroll through the list of results, you may find interesting events. For example, if you want to look only at activity on one particular IP address:

Ctrl-click on an IP address (cmd-click on a Mac).
Check "Wrap results" to turn on line-wrapping for the long single line events that result from searches.

Your search has been replaced with the IP address. This is an effective way to follow relationships between events.

Use the timeline

The timeline shows bars and a red line (or flag). The bars indicate the volume of search results and when they occur along the span of your time range.

You can change the time range with the drop-down menu:

Choose Custom.
Specify a start and end time.

You can also customize the time range by clicking on any bar in the timeline and zooming in on a particular cluster of events:

Set the time range to "Last month".
Execute the search for all sampledata.

Notice that each bar is equivalent to one day of data.

Click on the bar in the timeline showing the cluster of data.
Click Zoom in.
Repeat the last two actions to narrow the time range until you see a few more bars in the cluster of data.

Notice that each bar is equivalent to one minute of data.

Shift-click or drag your mouse across all of the bars and zoom in further.

Note: The red flag marks the location of the results you are currently viewing along the timeline. As you scroll through your search results, the red flag shifts to follow.

The timeline now spans several minutes, with one bar equal to one second.

Note: Your Splunk instance will perform faster if you narrow the time range of your search. Searching over all time may result in slow search performance.

Search results

Splunk allows you to navigate search results by following links and using interactive field filters. Filtering is an efficient method to organize the results of a search.

Events and fields

Your search results appear below the timeline as a list of events ordered by timestamp. A field is a name/value pair distinguished from the free-form indexed segments that you see in an event.

You can add and remove field filters, extract new fields from the results, and tag fields to group results.

Filter on fields

Search for all the sampledata index events:

index=sampledata

Splunk includes three default filters in your search results: host, source, and sourcetype. These interactive field filters are drop-down menus located below the timeline.

Each field's filter menu lists (up to) 10 values, ordered by the frequency at which they occur in the search results.

Host

The host field, which lists the originating hosts of events, lets you target one specific host in the filter. The host field is stored and indexed with each raw event.

Click on the host menu.

From the list, select the first host value, http2. The search results filter to show only results for the selected host.

Let's look at another host value and add it to our search:

To remove the first host filter, click Clear filter. The search results revert to your previous search.
Select the next host value from the menu.
To add this filter to your search string, click Add filter to search.

The search bar and search results update to include the host value restriction you applied, http1:

index=sampledata host=http1

Source

The source field lists the location where an event is accessed; a file, network port, script, etc.

Sourcetype

The sourcetype field characterizes all sources that have similar formats. For example, all Apache access logs in W3C common format have the sourcetype value access_common. The sample data contains four distinct sourcetypes - syslog, access_common, db2 and websphere_activity.

Show more fields

You can include many more field filters in addition to host, source, and sourcetype, in your searches. The fields are listed in the Fields... drop-down menu.

Search for all the sampledata index events:

index=sampledata

Let's add a couple more field filters to our search:

To display the list of field filters, click the Fields... menu.
Scroll through the list.
Check eventtype and punct.
Click Apply.

The interactive field filters list updates to include eventtype and punct menus. You can use these field filters exactly the same way you used host.

To remove a field filter menu:

Click the Fields.. menu.
Uncheck the fields you want to remove.
Click Apply.

The eventtype and punct fields are discussed further in Event types.

Define custom fields

Splunk lets you interactively define and extract fields from your search results. Let's define a field to extract the IP addresses from our search for all events in sampledata.

index=sampledata

You may need to scroll through the results or use the timeline to find events that contain an IP address.

Below the timestamp of every event is a drop-down menu. Click the down-arrow and select Extract field.

The Extract fields window opens.

Notice the panel at the top of this window:

Sample Event lists the event that you selected from the search results.
Example Value(s) provides a textarea for you to define the field value.

To define the IP address field for extraction:

Highlight the IP address from your sample event. Copy and paste (or type) it into the Example Value(s) textarea.
Click Preview.

In the Rules panel:

Select a field to restrict your search (either host, source, or sourcetype),
Splunk generates the regex rules to define your field.
Splunk provides a preview of values it has extracted from your results.

Splunk also provides a preview of other events that contain your custom field. Use this Preview panel to validate the results of your field definition.

To save your custom field definition, click Save. The Save FIeld Definition dialog box opens.

Name the field "ipaddress".
Click Save.

Now, your custom field (ipaddress) is listed in the Fields menu. You can activate and apply your field filter in exactly the same way you used host.

Tag fields

You can tag fields to group together results that share field values. Use tagging to attach a name, or tag, to a group of results that share the same value of a field, event type, host, or source. You can apply as many tags as you want to a single field, event type, host, or source. A tag cannot contain spaces.

To access the tagging dialog, click Tag field name in the drop-down menu of any field in your search results.
To apply a tag to the field you selected, type a tag name into the Tags field.
To apply multiple tags, enter a space-delimited list of tag names in the Tags field.

Note: Tags that you create for a field are displayed in italics next to that field name in your search results.

Collect snapshots

Splunk allows you to save your results in a "Snapshot Container" that houses your collection. Each snapshot includes an image of the time graph and your search string.

You can add and remove snapshots from your collection. However, after adding a snapshot, you cannot modify the time graph within the container.

If you want to modify a snapshot in your collection:

In the Snapshot Container, click Restore search.
Modify your graph.
Click Snapshot.

Your modified graph has been added to your snapshot collection.

Event types

Splunk allows you to classify events that have common characteristics and save them as a value in the eventtype filter field. Examples of event types include all ssd logins and all sendmail syslog messages. Editing, tagging, and naming such event types helps the Splunk server learn and improve its performance.

In this tutorial, you will search for a specific event and save your results as an event type which you can later apply to a new search.

Find similar events

Search for all events in the sampledata index:

index=sampledata

Because the format of an event is often a powerful part of defining an event type, Splunk indexes the punctuation characters of events as a field called punct. This field, while it looks cryptic at first, is a powerful way of finding similar events quickly.

To activate the punct field:

Click the FIelds... menu.
From the list of fields, check the punct field.
Select Apply.

The punct filter menu now appears below the timeline.

Click on this menu to view a list of the 10 punct values that are most common in your results:

Add a filter to your search for the most common punct value in sampledata, which might be:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__

Your results update to include only events in the sample data that have a matching punctuation scheme.

Save as event type

We want to save the last search as an event type. First, let's add "logout" to the search string:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__ logout

Now, the results are all sampledata events with the punctuation scheme that are logout actions.

To save your results as an event type:

Click the search bar menu.
Choose Save as event type...
In the Save Event Type dialog box, name your event type "trade_app_logouts".
Click Save.

Now you can include your new event type in a search.

Note: If your event type name contains spaces, the spaces will be replaced with underscores and the tags will not be saved. When naming the event type, do not include spaces.

Search for an event type

Let's search for all events in sampledata again.

After the search results load, activate the eventtype field filter.

From this field filter, choose trade_app_logouts.

Notice that eventtype=trade_app_logout appears as a field in your results, underneath each event:

Now, you can add the filter to your search:

Click on Add filter to search, or
In the events list, click on a highlighted instance of the field.

You search string updates to:

index=sampledata eventtype=trade_app_logouts

Note: Identifying and saving an event type means you can search for it directly. In fact, if you know the field name and value, you do not need to activate the field filter to add it to your search.

You can also click on the down-arrow to the right of the eventtype instance to perform the following options:

Report on all event types
Search for this event type
Tag event type
Rename event type
Delete event type

Tag an event type

You can tag event types that have very different search terms and punctuation patterns with common words, then find all events of types that have any tag. This is a great way to create higher level classifications like "logouts" that cross different logout event types from different applications.

You can also tag hosts with one or more words describing their function or type, enabling users to easily search for all activity on a group of similar servers. Tagging hosts is useful for knowledge capture and sharing and for crafting more precise searches.

See the section in our Admin manual on Host tagging.

Automated event type discovery

Splunk automatically discovers event types based on seeing a large number of events sharing common characteristics. You can edit, delete, rename, and tag event types that Splunk discovers. You can also make your own event types by saving any search as an event type.

Learn how to use automated event type discovery in Splunk Web.

You can also change the settings that determine which keywords are considered in event type discovery in its eventdiscoverer.conf configuration file. Learn how to configure eventdiscoverer.conf.

Save options

You can save any of your searches, schedule your saved searches, and define alert conditions for your scheduled searches.

Save a search

Search for the trade_app_logouts events in the sampledata:

index=sampledata eventtype=trade_app_logouts

To save a search:

1. Click on the search bar menu.

2. Select Save search... from the menu.
This opens the Save search dialog box.

3. In the "Search options" tab, name your search.

4. Click Save.

Note: When saving your search, you can choose to add it to one or more dashboards.

Splunk lets you delete or modify your saved searches and add them to the dashboard. For more information on how to manage saved searches, refer to the User Manual's Save, Schedule, and Alert page.

Schedule the search

From the search bar menu:

1. Choose Save search...

2. Click the Alert options tab.

3. Under Schedule, check "Run this search on a schedule".

Note: You can define the schedule frequency with the Basic or Cron options.

Schedule an alert

After you schedule a search, you can define alert conditions based on thresholds in the number of events, sources, and hosts in your results. You can receive these alerts via RSS feed or email.

You can also trigger a shell script, such as a script to generate an SNMP trap or call an API to send the event to another system. If you need additional email options (like setting the From: address) see the Alerts page in the Admin manual.

Reports

Splunk allows you to summarize the results of any search as a report in a separate window.

You can access the reports window in three ways:

After running a search, click Report on results >> located below the search bar.
Select Report on this field >> from any interactive field filter menu.
Pipe your search results into a report command, such as stats, top, and rare.

We'll cover pipes and other commands in More searches.

Report on results

Search for all firewall deny events in sampledata.

index=sampledata deny

After the results load, click Report on results >> above the timeline options. This takes you to a separate window where you can build your report.

Notice that:

You can enter new search strings from the search bar at the top of the window.
Splunk identifies fields from your search results and lists the field names in the Fields panel.

Select dst from the Fields list.

Splunk updates your search string to:

index=sampledata deny | top limit=100 dst

The report displays:

A chart graphing the results (the top 100 values of dst).
A summary of the count and events matching your search.

Notice that the options in the Series panel defines the data series for your chart. You can also choose a different chart to display your results.

Let's tune this search to report only the top 10 dst values of firewall deny events and display the series in a pie graph.

In the search bar, change the limit boundary to ten and enter the search:

index=sampledata deny | top limit=10 src

In the series panel, select display as "pie graph".

When you mouseover each wedge of the pie graph, an information box appears. The box lists the dst value and event count. If you click on the wedge, Splunk takes you back to the search results and updates your search string to include the specific field name and value you selected from the chart. Try it out!

Report on fields

Return to the search window and search for all firewall deny events in sampledata.

index=sampledata deny

To report on fields:

Click on the Fields... menu.
From the list, check and apply src.
From the src filter menu, choose Report on this field >>.

Splunk takes you to the report window and updates your search string:

index=sampledata deny | top limit=100 src

Now, you can modify your report settings.

Build new reports

From the reports window, you can also enter a new search and build new reports.

Search for all "access_common" data in sampledata.

index=sampledata sourcetype=access_common

From the resulting list of Fields, select bytes.

Under Series, define your data series to "show the sum of bytes vs. time split by action":

You can define a custom time range for your chart. Here, it's zoomed in to a day of data.

Note: The chart updates as you define your series.

Pick different charts

Change chart styles by selecting a type from the display as drop-down menu above the current chart.

Choose from the following chart types:

column
line
area
scatter
stacked column
stacked area
pie
donut
bubble
heatmap

See a sample of these charts in the report gallery on our website.

Add a report to your dashboard

You can save a report just as you would any other search. When you save the search, add it to your default dashboard by checking the box at the bottom of the save dialog.

You'll see the report on the dashboard after clicking the logo to return to the home page. Dashboard searches are refreshed every tenth of the time interval (for example, a 4 hour search every 24 minutes) or every hour, whichever is shorter.

You can read more about saving searches to the dashboard in Manage saved searches.

Note: You won't see your report on your dashboard if you haven't loaded any data to your main index yet. As soon as you have data in your main index, the "getting started" links are replaced with a default dashboard including modules that are predefined in the product, plus additional searches and reports you've added.

CLI searches

Splunk includes a command line interface (CLI) that runs from a shell on your Splunk server. Use the CLI to execute searches and integrate Splunk into admin scripts.

Learn how to use the CLI.

Add inputs

About inputs

Splunk can access and process any format of IT data from different sources on your filesystem. Data sources include logfiles, FIFO queues, network ports, databases, and scripts. You can add most of these input types to your index using Splunk Web.

This topic discusses the different input types you can add to Splunk's index using Splunk Web. For information about using other methods to define inputs (such as using inputs.conf), refer to the Admin manual's data inputs page.

Files and directories

When adding a new file or directory to your data inputs, you can monitor a directory, upload a local file, or index a file on the Splunk server. Use monitor to add continuous and non-destructive inputs. Upload or index files to add one-time and destructive inputs.

Monitor

Splunk's monitor command is similar to the UNIX tail -f command for file monitoring. When you monitor a directory, Splunk detects subdirectories and recursively examines them for new files. As new files are added to the directory, Splunk detects the changes and updates your indexes.

When you use Splunk Web to monitor a directory, Splunk modifies your inputs.conf file in /system/local to include a stanza that defines your new input. If you monitor /var/log, Splunk adds the following stanza to your local inputs.conf:

[monitor:///var/log]
disabled = false
host = <hostname>

Also, you can view and edit the input properties of your monitored directory from Admin > Data Inputs: Files & Directories in Splunk Web.

Upload

You can browse for a local file and add it directly to your inputs. If you have a previous version of the file as an input, uploading a new file will overwrite the existing version. Unlike a monitored file, the uploaded file does not continuously update. Therefore, use the upload option for one-time and destructive inputs.

Uploading a local file does not modify inputs.conf. Instead, it uploads the specified file into /var/run/splunk/upload and then moves it into /var/spool/splunk for indexing. The file does not show up as a new data input in Admin > Data Inputs: Files & Directories.

Note: When you upload a local file, if necessary, Splunk unpacks and uncompresses the file before processing it.

Index

Indexing a file on the Splunk server copies the file directly into /var/spool/splunk, where it exists while Splunk processes the data. Similar to uploading a local file, this operation does not modify inputs.conf and the indexed file does not show up as a new data input in Admin > Data Inputs: Files & Directories.

FIFO queues

Splunk accesses the data in a FIFO, or named pipe, queue as though it were a file. When defining a FIFO input in Splunk Web, provide the path that directs Splunk to the queue. When choosing the FIFO data input method, consider the following:

FIFO queues can be a high performance method to get data into Splunk, since the system does not have the I/O burden of writing to both a file on disk and Splunk's index on disk (like monitor).
FIFO access is very fast, but FIFOs are vulnerable when there are processing disruptions because the in-memory data may be lost.
You do not have to worry about log file rotation and archiving because the data goes straight from the logging application into Splunk via the queue. There is nothing on disk to manage except for Splunk's index.
Most syslog implementations can write to FIFO queues in addition to or instead of files.
Other applications can write to FIFO queues instead of files by just changing a logfile name parameter from a filename to a defined FIFO queue.

Note: FIFOs are not recommended for application servers forwarding data to Splunk in a distributed setting. Monitor is a more reliable, stable method.

Network ports

Splunk supports UDP and TCP connections. When configuring network ports, keep in mind that you cannot use ports lower than 1024 if you have not installed Splunk as root.

UDP

UDP is a best effort protocol; you might not get messages if the network is clogged or has a hiccup. You also can't be absolutely sure the messages aren't spoofed or altered in transit. UDP should be reserved for logging implementations focused on day-to-day troubleshooting rather than compliance or security.

Splunk with an Enterprise license can read directly from the network on any UDP port. Use this configuration to make Splunk act directly as a syslog server by reading remote syslog events on UDP port 514. You can also send any other UDP source of logging data, including SNMP.

Like all network streaming approaches, direct UDP input is higher performance than reading files from disk.

TCP

TCP is a reliable, high-performance choice for most situations, since this protocol includes checks to ensure that data has arrived safely and intact. Splunk with an Enterprise license can receive data on any TCP port, allowing Splunk to receive remote data from syslog-ng and other syslog implementations that use TCP for security or reliability. TCP is the foundation of Splunk's distributed data access.

Note: If the sending process buffers data such that events are broken into multiple pieces, Splunk may interpret the parts as multiple events. This is more likely if events are being generated intermittently, as there may be long pauses (several seconds or longer) between blocks of buffered data. If you notice truncated events, try forcing the process to send events atomically.

Use Data Inputs

This topic discusses how to use Splunk Web's Admin > Data Inputs page to add new inputs and edit existing inputs. These inputs include files, directories, FIFO queues, and network ports.

For information about adding inputs using crawl and using Live Tail, refer to their respective pages.

For more information about the different inputs you can add to Splunk, read About inputs.

Access Data Inputs page

In Splunk Web, you can add and manage all your data inputs from the Admin page:

1. On the upper righthand corner of any of the dashboards, click Admin.

2. From the lefthand navigation list, click Data Inputs.

This takes you to the Admin > Data Inputs: All page which tells you how many inputs you have in each category: Files & Directories, FIFO Queue, Network Ports, and Crawls.

You can add new inputs directly from this page by clicking Add input in the "Actions" column. If you want to view and edit the actual inputs, click on the input category.

Add files and directories

Use the Data Inputs: Files & Directories page to view and edit properties for monitored directories and uploaded files. Configure new inputs by clicking New Inputs. Change existing inputs by clicking on the input's path in the File or Directory column.

To add a new input:

1. Click New Input.

2. Under Data access, choose one of the following options:

Monitor a directory: to index a directory and continuously update when changes are made to that directory.
Upload a local file: to upload a file from your machine and index it.
Index a file on the Splunk server: to copy a file from the server directly into the your index.

3. Specify a pathname to the file or directory. If you choose to Upload a local file, you can browse for the source.

4. Under Host, select the host type under Set host and supply the required host value. Your host options depend on the data access method you selected in Step 2:

If you chose Monitor a directory, the Set host options include:

Constant value: requires a fully qualified domain name or IP address.
Regex on path: requires a regular expression for the host path.
Sement in path: requires a segment number.

If you chose Upload a local file or Index a file on the Splunk server, you can only set Set host to Constant value. This requires a fully qualified domain name or IP address.

Note: Refer to the Admin manual for more information about assigning host values to an input.

5. Under Source Type, set the source type to:

Automatic: to let Splunk define the sourcetype value.
From list: to select from existing sourcetype values.
Manual: to supply a custom sourcetype value.

6. Click Submit to save your new input.

Note: Refer to the Admin manual for more information about setting the source type for an input.

Add FIFO queues

Use the Data Inputs: FIFO Queues page to view and edit properties of each FIFO prcessed by Splunk.

Add network ports

Use the Data Inputs: Network Ports page to view and edit properties for UDP or TCP ports watched by Splunk.

Run crawl searches

Use the Data Inputs: Crawls page to search for new inputs to add or update existing inputs. Refer to the Use crawl page for more information on this search feature.

Use crawl

crawl searches your filesystem for new data sources to add to your index. Configure one or more types of crawlers in crawl.conf to define the type of data sources to include in or exclude from your results. Save this crawl search and schedule it to run regularly to update your indexes.

This topic explains how to use the crawl command and how to save and schedule a crawl search. Refer to the Admin manual for instructions on configuring crawl.

Note: Splunk Preview currently supports one type of crawler, labeled file_crawler. As yet, you cannot define a custom crawler.

Run a crawl

In Splunk Web, you can access and run the crawl command from the Splunk search bar and the Admin > Data Inputs: Crawls page.

The Splunk search bar
You can run the crawl command directly from the search bar:

| crawl

If you run a crawl without arguments, Splunk searches your filesystem with the settings defined in crawl.conf. To override these default settings, specify crawl options at search time.

The Admin page
You can manage all your saved crawls from the Admin > Data Inputs: Crawls page. From this page, you can also run the default crawl search by clicking New Crawl:

| crawl | search NOT *personal*

After the crawl completes you can add or remove options to narrow your search.

Results of a crawl

For each item listed in your crawl results, Splunk displays whether or not it is a file, a timestamp indicating when it was last modified, its size, and its status (whether it is added or not added to your inputs). You can perform two actions on each data source: Add input and Preview file/directory.

Preview file or directory

To review the contents of the data source before adding it as an input, click Preview file or Preview directory.

A new window opens:

If you click Preview file on a file, Splunk returns events from the file.
If you click Preview directory on a directory, Splunk displays a list of the files in the directory and lets you drill-down further and preview each file.

Add input

To add the selected data source as an input, click Add input.

Now, when you go to the Admin page and select the Data Inputs tab, your selected data source is listed.

Note: Adding data inputs with crawl modifies your inputs.conf file to include a stanza describing the new source. For example, if crawl discovers /var/log, clicking Add input adds the following stanza to inputs.conf:

[monitor:///var/log]
disabled = false
index = main
class = crawl
generator = ui

Save a crawl

After you run a crawl search, save the search by clicking the Save this Crawl... link located above your search results. This action opens the Admin > Data Inputs: Crawls: Create Crawl page which prompts you to:

Name your crawl search.
If necessary, edit your search.
If desired, elect to run your crawl on a schedule.
Click Cancel to return to the Admin > Data Inputs.
Click Save to save your crawl search.

Note: Your crawl won't save, if you don't provide a name.

Manage saved crawls

Manage your saved crawl searches from the Admin > Data Inputs: Crawls page. You can run a new crawl or select one or more saved crawls to:

Run Now and update your indexes.
Enable or Disable so that you can start or stop updating particular indexes.
Delete to remove the search from your list.

Edit the search and schedule properties of an individual crawl by clicking on its Name.

Note: You cannot change the name of your saved crawl.

Schedule saved crawls

When scheduling your saved crawls, you can define the type of schedule and how frequently to run it. You can also set alert options and define fields to include in summary indexes. These options are exactly the same as options provided for saving regular (non-crawl) searches.

Use Live Tail

Live Tail lets you monitor data that is coming into Splunk in real-time. Live Tail for Splunk Web works just like tail -f (in Linux/Unix). Search for any text in data as it is indexed into Splunk. Live Tail streams data to your browser based on a simple text search.

You can use Live Tail for a lot of different things, for example:

Passive monitoring
- You'll know the moment specific events occur.
Troubleshooting
- Set up Live Tail to search for a particular type of event and set it to monitor your environment.
- Change your environment and monitor the effects in the Live Tail stream.
- For example, send an email and see whether it passes your spam filter.

Use Live Tail in Splunk Web

To start Live Tail, select the View in Live Tail menu item in the search bar drop-down menu.

Live Tail launches in a new window (or new tab - depending on your browser configuration). The Live Tail processor takes the search terms you input in the search bar(before they are piped to data processing commands), creates a search based on them, and streams data to your browser that matches the search.

The Live Tail interface

The Live Tail interface is a separate window opened when you click View in Live Tail in the search bar drop-down menu. The controls available to you in the Live Tail window are listed here.
Live Tail interface controls:

Enter search terms in the Live Tail search bar.
Click the green button to launch a new stream based on the search terms entered in the Live Tail search bar.
Press ctrl-c to terminate the current stream (just like with tail -f in a Linux or Unix shell).
Use the Wrap results check box to word-wrap the search results just like in Splunk Web.
Press the Enter key anywhere outside the search box to insert a new line in the current stream.
Press ctrl + shift + b to pause or un-pause Live Tail.
- On a Mac, use cmd + ctrl + b.

Start Live Tail from the Splunk CLI

1. Log into Splunk. ./splunk login
2. Use the live-tail CLI command to start Live Tail.
3. Type: ./splunk live-tail "your search string", where "your search string" is whatever simple search terms you want to search for (surrounded by quotes).

Current limitations

The following are the current limitations of Live Tail:

You can only perform a simple text search while using Live Tail. You can't use any Splunk search commands or any data extractions in a search.
If the client is overloaded by the volume of the data coming in to the processor, it will arbitrarily omit chunks of data. This means that with a very high volume of data, some events may never be displayed on screen for Live Tail.
User permissions must be configured properly to allow access to Live Tail. By default, Live Tail will not work for the User role. It should work by default for users assigned to the Power or Admin role.

Index

About indexes and indexing

We use the term "index" to refer to:

How Splunk manipulates and prepares raw event data for searching.
The act of processing raw event data as Splunk prepares it for searching.
The directory where Splunk stores all the event data.

Splunk indexes data in real time. It accesses data using a variety of input methods, applies universal processing techniques to handle different formats of IT data, and persists the original raw data along with indexes and additional fields added during processing.

Note: Refer to the About inputs page for more information about input types and methods.

Events, segments, and fields

Events are a single record of activity or instance of data -- for example, a single log entry. Fields are attribute and value pairs that make up segments of events. As part of indexing, events are broken into segments; Splunk uses breaking characters and rules to define how events are divided.

Usually, Splunk can detect event boundaries for different data formats. However, if event boundary recognition is not working as desired, you can customize your rules in props.conf. Refer to the Admin Manual for how to configure event boundaries.

The are two types of segments: major and minor. Major segments are words, phrases, or terms in the data that are surrounded by breaking characters such as white space and newline characters. Minor segments are breaks within a major segment. For example, the IP address 192.168.1.254 may be indexed as a major segment and then separated into the following minor segment: 192, 192.168, and 192.168.1.

Edit your segment recognition rules in segmenters.conf and apply them to different fields via props.conf. Refer to the Admin Manual for how to configure segmentation.

Search and indexes

Splunk stores all processed data in a collection of database directories, also called an index. Each database directory is located in $SPLUNK_DB and named db_<starttime>_<endtime>_<seq_num>. $SPLUNK_DB defaults to $SPLUNK_HOME/var/lib/splunk. The following is a list of Splunk's preconfigured indexes and a brief description of what they store:

history: search history.
main: all processed data. Unless otherwise specified, this is the default database.
sampledata: sample event data used for training.
splunklogger: Splunk internal logs.
summary: summary indexing searches.
_audit: events from the file system change monitor and auditing.
_blocksignature: event block signatures.
_internal: metrics from Splunk's processors.
_thefishbucket: internal information on file processing.

You can create new indexes, edit index properties, remove unwanted indexes, or relocate existing indexes. You can manage (create, view, and edit) indexes from Splunk Web. For more information, refer to the User Manual's topic on managing and creating indexes. You can only remove and relocate existing indexes via the CLI. For more information, refer to the Admin Manual's topic on index management.

Unless specified, Splunk automatically searches through the default index, main. You can restrict your search to another index by specifying it in the search bar. For example, to search for HTTP requests that occurred only in sampledata:

index=sampledata http

Manage your indexes

In previous Splunk releases, you used the command line interface (CLI) to manage your indexes. Now, you can view your indexes, edit their properties, and add new indexes from the Admin page of Splunk Web.

View and manage indexes

The Admin > Index: View/Manage Indexes page displays a table of all your indexes and their properties, including:

The home path, or directory.
The current size in MB.
The maximum size in MB.
A count of events.
A timestamp for the latest event.
A timestamp for the earliest event.

Edit index properties

From the Admin > Index: View/Manage Indexes page, click an index name to view and edit that index's properties. Properties that you cannot change are grayed out and include:

The index's name.
The path to the fields and hot/warm databases.
The path to the cold databases.
The path to the thawed databases.

Properties that you can redefine include:

The maximum size (in MB) of the hot database.
The maximum size (in MB) of an index.

After you make your changes, click Update. Then, restart Splunk to apply your changes.

Note: To apply any changes that you make to the indexes, such as editing properties or adding a new index, you must restart Splunk. In Splunk Web, you can restart the Splunk server from Admin > Server: Control Server. Just click Restart Now.

Create new index

Splunk ships with an index called main that, by default, holds all your events. Splunk with an Enterprise license lets you add an unlimited number of additional indexes. One of them serves as the default index for any input and search command that don't specify and index. You can add indexes via Splunk Web, Splunk's CLI or indexes.conf.

via Splunk Web

1. The Admin > Indexes: Create Index page lets you define the properties for a new index. To create a new index, enter:

A name for the index.
The maximum size (in MB) of the hot database.
The maximum size (in MB) of the index.

2. If you check Advanced settings, the list of properties expands. Advanced properties include:

The maximum number of search results.
The maximum number of warm database directories.
The maximum number of cold databases open at any given time.
The frequency that new hot database are to be created.
The frequency that cold databases are to be frozen.
The script and directory to archive the index's data.
The number of concurrently running optimize processes.
Whether to wait for optimize processes to finish or just kill them.
The number of extra threads to use during indexing.
The amount of memory (in MB) to allocate for indexing.
The number of events to make up a block for block signatures.

3. After setting the index's properties, click Add. Then, restart Splunk to save and apply your changes.

You can also edit an index at any time by clicking on the index name within the Indexes tab of the Admin section of Splunk Web. Properties that you cannot change are grayed out. To change these properties, use indexes.conf.

via Splunk's CLI

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.

To add an index, first shutdown Splunk with splunk stop. Then navigate to Splunk's CLI. Then type:

# ./splunk add index [name] [directory (optional)]

Note: Do not use capital letters in your index name; this is a known problem that will be fixed.

The optional directory argument lets you set up an index outside of the default $SPLUNK_DB location.

The add index command brings you to a dialog session. Specify the configurations of your new index:

./splunk add index hatch

Hit enter to accept the default values in parenthesis, or enter your own values.

Delete an index

Use the CLI to delete an index from your Splunk instance:

# ./splunk remove index [name]

Search

Searching is easy - type any term you'd expect to find in your data into the search box and hit Enter. A Splunk search lets you search indexed data in real-time, extract data from search results, and produce meaningful reports from the data you put into Splunk. For example, to search for all events containing a given IP address, type it into the box. Try it with usernames, error codes, transaction IDs, or whatever else you are looking for.

See the Search syntax page to learn about Splunk search syntax.

A search is pipeline of commands (similar to a Unix "|" pipeline) that starts with a command that gathers data (typically a search on data in a Splunk index), followed by data-processing commands that operate on the data to yield search results.

See the Search pipeline syntax page for details about the syntax of the search pipeline.

Generate search results

You can generate search results in three ways:

Use the search command to get new search results (that match your search criteria) from a Splunk index.
Use the savedsearch command to execute a saved search (access a previous search).
Use the file command to import previously gathered search results directly from a file.

Construct searches

Use the search command to construct simple keyword searches on data in your Splunk index (just like a Google search). Narrow your keyword searches with modifiers, fields, Boolean operators, and logical comparison operators.

You can also construct more powerful searches by using additional commands to extract data, perform statistical operations, and build reports. Learn about the search commands in the search command reference.

Here are some valuable points to remember when constructing a search:

When generating data

To get more results:

Increase the time range (search over all time to get the most number of results).
Use wildcards and partial keywords instead of exact keywords.

If you want a faster search:

Narrow the time range of your search.
Reduce the number of fields that are extracted by un-selecting fields in the Fields picker menu.
Reduce the segmentation by selecting inner, outer, or raw in the Preferences menu (reducing segmentation makes extracting and reporting more difficult).

When narrowing your search

Filter results by event type if possible.
Use the source and sourcetype fields to narrow a search to only a specified source.
Use a combination of logical and Boolean expressions of keywords, modifiers, and fields.

Form searches

Form searches are reusable searches that are pre-defined by a Splunk administrator. Form searches allow you to run complex searches by simply inputting variables in form fields. Learn more about Form searches.

Macro searches

Macro searches allow macro substitution of variables in saved searches. This allows you to run a complex search repeatedly with different variables. Learn more about Macro searches.

Transaction searches

Transaction searches let you search for groups of related events that are pre-defined as a transaction by your Splunk administrator. Use the transaction command to execute a transaction search. You can override specifications of a pre-defined transaction, or define a new transaction with the transaction command. Learn more about Transaction search.

Live Tail

Live Tail allows you to see data as its being indexed into Splunk in real-time (similar to Unix's tail -f command). Live Tail allows you to execute a simple search in its stand-alone window, and monitor events that match the search. Find out more about Live Tail.

Asynchronous searches

The Splunk CLI allows you to run multiple searches asynchronously. Use this if you have a search or report you want to run on a large amount of data where the search could take days and you still want to be able to run other searches with Splunk. Use the dispatch CLI command to execute asynchronous searches. Learn more about asynchronous searches.

CLI searches

Schedule and save searches

Tune search performance

Splunk's searches are optimized for text-based searching of raw event data. Search speed is dependent on how your Splunk install is configured. You can improve the speed of your searches by editing configuration files, and by downloading various add-ons from SplunkBase. Read more about tuning search performance.

Form search

A form search is a saved search that has form fields that you must fill in before you run a search. Save any complicated search, and make it reusable as a form search (learn how to create form searches).

Form searches are saved searches that appear as forms when run. Save any search with form fields that a user running the search must fill out with parameters to run the search. You can create a sophisticated saved search and save it as a form with as many form fields as you like.

For example, you can define a search that returns all Web server errors for any username to be specified at search time:

503 OR 500 OR 404 sourcetype=access_common $user$

When run, this search appears as a form labeled user.

The search 503 OR 500 OR 404 sourcetype=access_common is still part of the search, but does not appear to the user.

Note: Form search works via text substitution, so the form fields can consist of anything, not just an indexed or an extracted field.

Run a form search

Form searches are saved searches. Run a form search by selecting it from the "Saved searches" menu in the search bar drop-down in Splunk Web.

If the saved search you select is a form search, then you'll be prompted with a form dialog like this:

Fill out the values in the form.

Note: You can substitute any text (not just a field) in a free-form text box in the form.

Refer to the Admin guide section on form searches to learn how to create form searches.

Transaction search

Transaction search enables you to search, and report on transactions in Splunk. A transaction is a grouping of events that contain related pieces of information. Transaction search is useful for a single observation of any physical event stretching over multiple logged events.

Here is an example of what can make up a transaction:

An event in a Web access log
An event in an application server log
An event in an asynchronous fulfillment application that generates a message queue that subsequently identifies a business transaction

In this case, the Web access log might share a session ID with the event in the application server log; the application server log might contain the account ID, transaction ID, and product ID; the transaction ID may live in the message queue with a message ID, and the fulfillment application may log the message ID along with the shipping status. All of this data represents a single user transaction.

Useful transactions

There are many cases where a transaction search may be useful. Here are some use cases for transaction search:

Find quarantined mail messages where:
- All events printed to a mail log for the different steps in processing a given message are considered to be a transaction.
- All events in the transaction share a message ID or a process ID with events that have both a process ID and message ID providing the transition.
- Queue entry events containing the sender's email address, and separate delivery events for each recipient of the message.

To find a security issue:
- A sequence of 3 specific error messages in a particular order within a time frame on a given host that may be consistent with a known failure or attack pattern.

To combine any combination of specific event types:
- Any transaction from a single data source that generates multiple log entries for a single event.

The transaction command

Search for transactions using the transaction search command. transaction yields groupings of events which may then be used in reports. To use transaction, either call a pre-configured transaction type (that your Splunk Administrator configured), or define transaction constraints in your search by setting the specification options of transaction.

Example transaction searches

Run a search that groups together all of the pages a single user (or client IP address) looked at over a time range.

This search takes events from the access logs, and creates a transaction from events that share the same clientip value that occurred within 5 minutes of each other (within a 3 hour time span).

sourcetype=access_combined | transaction fields=clientip maxpause=5m maxspan=3h

Transactions and macro search

Transactions and macro search are a powerful combination that allow substitution into your transaction searches. Make a transaction search and then save it with $field$ to allow substitution.

When to not use transactions

It is almost always more efficient to use the stats command when computing aggregate statistics over transactions parameterized by a unique identifier. For example, to compute the statistics of the duration of a transaction parameterized by the field trade_id:

* | stats min(_time) as earliest max(_time) as latest by trade_id | eval duration = latest-earliest | stats min(duration) max(duration) avg(duration) median(duration) perc95(duration)

Similarly, if you want to compute the number of hits per clientip in an access log:

sourcetype=access_combined | stats count by clientip | sort -count

You can also compute the number of distinct sessions (parameterized by cookie) per clientip in an access log:

sourcetype=access_combined | stats dc(cookie) as sessions by clientip | sort -sessions

Macro search

Save searches with macro fields, which are values you set at search time. You can create sophisticated saved searches with as many macro fields as you like. Use macro searches in Splunk Web or in Splunk's CLI. Macro searches work similarly to form searches, except there is no graphical user interface.

Configure a macro search

1. Create a saved search. Use $TERM$ to specify a macro field for substitution. You can specify any number of macro fields.

host=swan OR host=pearl $user$ $trans$

2. Save the search and name it. The fo