This document last updated: 11/24/08 03:11pm

Print User Manual

About this manual

What's in this guide?

This manual teaches end users about Splunk and how to use Splunk search. This manual focuses primarily on accomplishing tasks through the web interface, Splunk Web, and the command line interface.

To help you start using Splunk, run through the Tutorial. The chapters following the tutorial focus on tasks, such as adding inputs, searching, setting alerts, and building reports. The manual ends with reference pages for search syntax, commands, and modifiers.

Refer to the search cheatsheet to help you construct useful searches right away.

Take note:

• Before using Splunk, review the Installation manual for system requirements and step-by-step installation guides.
• System administrators should refer to the Administration and Deployment manuals for advanced configuration and data management topics.
• Those interested in customizing Splunk and creating applications for Splunk should refer to the Developer manual.
• You can also share custom Splunk applications with other members of the Splunk community via SplunkBase.

About Splunk

About Splunk

Splunk is an IT search engine.

• You can use Splunk to search and navigate IT data from applications, servers, and network devices in real-time.

• Data sources include logs, configurations, messages, alerts, scripts, code, metrics, etc.

• Splunk lets you search, navigate, alert, and report on all your IT data in real-time using Splunk Web.

Tutorial

About this tutorial

This section briefly explains what you need to run Splunk and complete this tutorial.

Requirements

• A supported browser (See system requirements and release notes).
• A copy of Splunk:
  • On an individual machine, or elsewhere in the organization. (Download and install.)
  • On Splunk hosted demo servers.

Log in

Splunk does not require a login when using a Free license. An Enterprise license requires a login:

• For an individual server, the default Splunk username is admin and password is changeme.
• For the Splunk hosted demo server, the username and password are guest and guest.

Splunk Web

Splunk Web runs by default on port 8000 of the host on which it is installed.

• If you're running Splunk on your local machine, the URL to access Splunk Web is http://localhost:8000
• If you're running Splunk on a different machine, the URL to access Splunk Web is http://<hostname>:8000

where <hostname> is the name of the machine Splunk is running on.

Logging in to Splunk takes you to the dashboard and Splunk Web. For this tutorial, you only need to know that:

• You type your search query in the search bar at the top of the dashboard.
  • The down-arrow button, to the left of the search bar, is a drop-down menu.
  • The green button, to the right of the search bar, enters your search.
• You change the time range with the drop-down menu below the search box.
• Splunk displays the number of occurrences of your search results graphically on the timeline.
• Splunk lists each individual event that results from your search underneath the timeline and ordered by timestamp.

Read the other sections of the Splunk User Manual for more information.

Command line interface (CLI)

Splunk includes a command line interface (CLI) that runs from a shell on the server host. The Splunk CLI is a great way to integrate Splunk into admin scripts. Read Use the Splunk CLI for more information.

Simple searches

See a few simple searches in action. This section introduces keyword searches and how to narrow your search with Boolean logic and using the timeline. Before you begin, you can watch Splunk's developer video about search; it demonstrates how to search with a just a few words and clicks of the mouse.

Index data

Splunk comes with pre-indexed sample data, called sampledata, which we will use throughout this tutorial. You can search the index that has the sample data in it instead of the main index by including index=sampledata in the search criteria.

For help indexing your own data, see the data inputs section of the Admin Manual.

Search

To start, enter your search in the search bar at the top of the page. To search for all the data in the sampledata index, type the following into the search bar:

index=sampledata

The timeline should show bars indicating when matching results occurred. If there are no results displayed, change the time range until you see results.

Now, lets search for HTTP requests that resulted in an internal server error (code 500). Type this simple search:

index=sampledata http 500

Narrow your search

You can use arguments in the search command to narrow your search. Add Boolean logic between terms and modifiers, use logical comparison operators for field values, or use search modifiers. You can also use the timeline to zoom in on particular events. This section discusses two ways to apply Boolean modifiers to your search. We'll discuss using the timeline to narrow your results later.

Read Search results for more ways to manipulate search results.

Search with Booleans

Splunk supports the Boolean operators: AND, OR, and NOT (must be capitalized).

Enter the search:

index=sampledata http AND 500

Your results should match the previous example search. Similar to Google and other search engines, Splunk implicitly inserts an AND between terms by default.

Note: If your search produces no results, try zooming out, clearing the time range, or resetting the time range using the drop-down menu.

Search for all HTTP requests that do not contain error code 500:

index=sampledata http NOT 500

Search for all sampledata events of sourcetype access_common or syslog.

index=sampledata (sourcetype=access_common OR sourcetype=syslog)

Note: Splunk uses parentheses to group Boolean expressions.

Click on results

As you scroll through your results and mouse over sections of each event, you'll notice the sections are highlighted. You can highlight and click items in the results to add and remove terms in your search string.

Search for:

index=sampledata http

Scroll through the list of results. Click on "500" in one of the search results.

Notice that Splunk highlights and updates the search to add "500" as a term (in the search bar). This is a shortcut for applying the "AND" operator to your search.

index=sampledata http 500

Click on another instance of "500". Splunk removes the term from your search string and your search results include all HTTP results again:

index=sampledata http

Now, alt-click on "200" in any search result (option-click for Mac, alt-shift-click for some popular *nix windows managers).

Splunk now updates your search with "NOT 200"; This is a shortcut for applying the "NOT" operator.

index=sampledata http NOT 200

Follow a relationship

While you scroll through the list of results, you may find interesting events. For example, if you want to look only at activity on one particular IP address:

• click on an IP address (option-click on a Mac).
• Check Wrap results to turn on line-wrapping for the long single line events that result from searches.

Now, your results are a chronological list of events that occurred on this IP address. You can use this to trace a sequence on events. This is an effective way to follow relationships between events.

Use the timeline

The timeline shows bars and a red line (or flag). The bars indicate the volume of search results and when they occur along the span of your time range.

You can change the time range with the drop-down menu:

• Choose Custom.
• Specify a start and end time.

You can also customize the time range by clicking on any bar in the timeline and zooming in on a particular cluster of events:

• Set the time range to "Last month".
• Execute the search for all sampledata.

Notice that each bar is equivalent to one day of data.

• Click on the bar in the timeline showing the cluster of data.
• Click Zoom in.
• Repeat the last two actions to narrow the time range until you see a few more bars in the cluster of data.

Notice that each bar is equivalent to one minute of data.

• Shift-click or drag your mouse across all of the bars and zoom in further.

Note: The red flag marks the location of the results you are currently viewing along the timeline. As you scroll through your search results, the red flag shifts to follow.

The timeline now spans several minutes, with one bar equal to one second.

Note: Your Splunk instance will perform faster if you narrow the time range of your search. Searching over all time may result in slow search performance.

Search results

Splunk allows you to navigate search results by following links and using interactive field filters. Filtering is an efficient method to organize the results of a search.

Events and fields

Your search results appear below the timeline as a list of events ordered by timestamp. A field is a name/value pair distinguished from the free-form indexed segments that you see in an event.

You can add and remove field filters, extract new fields from the results, and tag fields to group results.

Filter on fields

Search for all the sampledata index events:

index=sampledata

Splunk includes three default filters in your search results: host, source, and sourcetype. These interactive field filters are drop-down menus located below the timeline.

Each field's filter menu lists (up to) 10 values, ordered by the frequency at which they occur in the search results.

Host

The host field, which lists the originating hosts of events, lets you target one specific host in the filter. The host field is stored and indexed with each raw event.

1. Click on the host menu.

2. From the list, select the first host value, http2.
The search results filter to show only results for the selected host. Note that this does not add the filter to your search; instead, it shows you a preview of your results. If these aren't the results you want, you can remove this filter and revert to your earlier search.

3. To remove the first host filter, click Clear filter.
The search results revert to your previous search.

4. Select the next host value from the menu.

5. To add this filter to your search string, click Add filter to search.
The search bar and search results update to include the host value restriction you applied, http1:

index=sampledata host=http1

Source

The source field lists the location where an event is accessed; a file, network port, script, etc.

Source type

The sourcetype field characterizes all sources that have similar formats. For example, all Apache access logs in W3C common format have the sourcetype value access_common. The sample data contains four distinct sourcetypes - syslog, access_common, db2 and websphere_activity.

Show more fields

You can include many more field filters in addition to host, source, and sourcetype, in your searches. The fields are listed in the Fields... drop-down menu.

Search for all the sampledata index events:

index=sampledata

Let's add a couple more field filters to our search:

1. To display the list of field filters, click the Fields... menu.

2. Scroll through the list.

3. Check eventtype and punct.

4. Click Apply.

The interactive field filters list updates to include eventtype and punct menus. You can use these field filters exactly the same way you used host.

To remove a field filter menu:

1. Click the Fields.. menu.

2. Uncheck the fields you want to remove.

3. Click Apply.

The eventtype and punct fields are discussed further in Event types.

Define custom fields

Splunk lets you interactively define and extract fields from your search results. Let's define a field to extract the IP addresses from our search for all events in sampledata.

index=sampledata

You may need to scroll through the results or use the timeline to find events that contain an IP address.

Below the timestamp of every event is a drop-down menu. Click the down-arrow and select Extract field.

The Extract fields window opens.

Notice the panel at the top of this window:

• Sample Event lists the event that you selected from the search results.
• Example Value(s) provides a textarea for you to define the field value.

To define the IP address field for extraction:

1. Highlight the IP address from your sample event. Copy and paste (or type) it into the Example Value(s) textarea.

2. Click Preview.

3. In the Rules panel:

• Select a field to restrict your search (either host, source, or sourcetype),
• Splunk generates the regex rules to define your field.
• Splunk provides a preview of values it has extracted from your results.

Splunk also provides a preview of other events that contain your custom field. Use this Preview panel to validate the results of your field definition.

4. To save your custom field definition, click Save.
The Save FIeld Definition dialog box opens.

2. Under Name, enter a name for the field. Type in "ipaddress".

3. Click Save.

Now, your custom field (ipaddress) is listed in the Fields menu. You can activate and apply your field filter in exactly the same way you used host.

Tag fields

You can tag fields to group together results that share field values. Use tagging to attach a name, or tag, to a group of results that share the same value of a field, event type, host, or source. You can apply as many tags as you want to a single field, event type, host, or source. A tag cannot contain spaces.

• To access the tagging dialog, click Tag field name in the drop-down menu of any field in your search results.
• To apply a tag to the field you selected, type a tag name into the Tags field.
• To apply multiple tags, enter a space-delimited list of tag names in the Tags field.

Note: Tags that you create for a field are displayed in italics next to that field name in your search results.

Collect snapshots

Splunk allows you to save your results in a "Snapshot Container" that houses your collection. Each snapshot includes an image of the time graph and your search string.

You can add and remove snapshots from your collection. However, after adding a snapshot, you cannot modify the time graph within the container.

If you want to modify a snapshot in your collection:

1. In the Snapshot Container, click Restore search.

2. Modify your graph.

3. Click Snapshot.
Your modified graph has been added to your snapshot collection.

Event types

Splunk allows you to classify events that have common characteristics and save them as a value in the eventtype field. Examples of event types include all ssd logins and all sendmail syslog messages. Editing, tagging, and naming such event types helps the Splunk server learn and improve its performance.

In this tutorial, you will search for a specific event and save your results as an event type which you can later apply to a new search.

Find similar events

Search for all events in the sampledata index:

index=sampledata

Because the format of an event is often a powerful part of defining an event type, Splunk indexes the punctuation characters of events as a field called punct. This field, while it looks cryptic at first, is a powerful way of finding similar events quickly.

To activate the punct field:

1. Click the FIelds... menu.

2. From the list of fields, check the punct field.

3. Select Apply.
The punct filter menu now appears below the timeline.

4. Click on this menu to view a list of the 10 punct values that are most common in your results:

5. Add a filter to your search for the most common punct value in sampledata, which might be:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__

Save as event type

We want to save the last search as an event type. First, let's add "logout" to the search string:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__ logout

Now, the results are all sampledata events with the punctuation scheme that are logout actions.

To save your results as an event type:

1. Click the search bar menu.

2. Choose Save as event type...

3. In the Save Event Type dialog box, name your event type "trade_app_logouts".

4. Click Save.

Now you can include your new event type in a search.

Note: If your event type name contains spaces, the spaces will be replaced with underscores and the tags will not be saved. When naming the event type, do not include spaces.

Search for an event type

Let's search for all events in sampledata again.

1. After the search results load, activate the eventtype field filter.

2. From this field filter, choose trade_app_logouts.

Notice that eventtype=trade_app_logout appears as a field in your results, underneath each event:

3. Now, you can add the filter to your search:

• Click on Add filter to search, or
• In the events list, click on a highlighted instance of the field.

You search string updates to:

index=sampledata eventtype=trade_app_logouts

Note: Identifying and saving an event type means you can search for it directly. In fact, if you know the field name and value, you do not need to activate the field filter to add it to your search.

You can also click on the down-arrow to the right of the eventtype instance to perform the following options:

• Report on all event types
• Search for this event type
• Tag event type
• Rename event type
• Delete event type

Tag an event type

You can tag event types that have very different search terms and punctuation patterns with common words, then find all events of types that have any tag. This is a great way to create higher level classifications like "logouts" that cross different logout event types from different applications.

You can also tag hosts with one or more words describing their function or type, enabling users to easily search for all activity on a group of similar servers. Tagging hosts is useful for knowledge capture and sharing and for crafting more precise searches.

See the section in our Admin manual on Host tagging.

Automated event type discovery

Splunk automatically discovers event types based on seeing a large number of events sharing common characteristics. You can edit, delete, rename, and tag event types that Splunk discovers. You can also make your own event types by saving any search as an event type.

Learn how to use automated event type discovery in Splunk Web.

You can also change the settings that determine which keywords are considered in event type discovery in its eventdiscoverer.conf configuration file. Learn how to configure eventdiscoverer.conf.

Save options

You can save any of your searches, schedule your saved searches, and define alert conditions for your scheduled searches. For more information, refer to the User Manual topic about Save, schedule, and alert options.

Save a search

Search for the trade_app_logouts events in the sampledata:

index=sampledata eventtype=trade_app_logouts

To save a search:

1. Click on the search bar menu.

2. Select Save search... from the menu.
The Save Search dialog box opens.

3. In the "Search options" tab, name your search. (In 3.3, this is Search.)

4. Click Save.

Note: When saving your search, you can choose to add it to one or more dashboards.

Splunk lets you delete or modify your saved searches and add them to the dashboard. For more information on how to manage saved searches, refer to the User Manual's Find and manage saved searches page.

Schedule the search

From the search bar menu:

1. Choose Save search...

2. Click the Schedule & Alerts tab. (In 3.3, this is Schedule and Alert.)

3. Under Schedule, check "Run this search on a schedule".

Note: You can define the schedule frequency with the Basic or Cron options.

Schedule an alert

You can also trigger a shell script, such as a script to generate an SNMP trap or call an API to send the event to another system. If you need additional email options (like setting the From: address) see the Alerts page in the Admin manual.

Reports

Splunk allows you to summarize the results of any search as a report in a separate window.

You can access the reports window in three ways:

1. After running a search, click Report on results >> located below the search bar.

2. Select Report on this field >> from any interactive field filter menu.

3. Pipe your search results into a report command, such as stats, top, and rare.

We'll cover pipes and other commands in More searches.

Report on results

Let's build a report for all firewall deny events in sampledata:

1. Search for all firewall deny events in sampledata.

index=sampledata deny

2. After the results load, click Report on results >> above the timeline options. This takes you to a separate window where you can build your report.

Notice that:

• You can enter new search strings from the search bar at the top of the window.
• Splunk identifies fields from your search results and lists the field names in the Fields panel.

3. Select dst from the Fields list.

Splunk updates your search string to:

index=sampledata deny | top limit=100 dst

The report displays:

• A chart graphing the results (the top 100 values of dst).
• A summary of the count and events matching your search.

Notice that the options in the Series panel defines the data series for your chart. You can also choose a different chart to display your results.

Let's tune this search to report only the top 10 dst values of firewall deny events and display the series in a pie graph.

4. In the search bar, change the limit boundary to 10 and enter the search:

index=sampledata deny | top limit=10 src

5. In the series panel, select display as "pie graph".

5. When you mouseover each wedge of the pie graph, an information box appears.
The box lists the dst value and event count. If you click on the wedge, Splunk takes you back to the search results and updates your search string to include the specific field name and value you selected from the chart. Try it out!

Report on fields

Return to the search window and search for all firewall deny events in sampledata.

index=sampledata deny

To report on fields:

1. Click on the Fields... menu.

2. From the list, check and apply src.

3. From the src filter menu, choose Report on this field >>.

Splunk takes you to the report window and updates your search string:

index=sampledata deny | top limit=100 src

Now, you can modify your report settings.

Build new reports

From the reports window, you can also enter a new search and build new reports.

1. Search for all "access_common" data in sampledata.

index=sampledata sourcetype=access_common

2. From the resulting list of Fields, select bytes.

3. Under Series, define your data series to "show the sum of bytes vs. time split by action":

You can define a custom time range for your chart. Here, it's zoomed in to a day of data.

Note: The chart updates as you define your series.

Pick different charts

Change chart styles by selecting a type from the display as drop-down menu above the current chart. Choose from the following chart types:

• column
• line
• area
• scatter
• stacked column
• stacked area
• pie
• donut
• bubble
• heatmap

See a sample of these charts in the report gallery on our website.

Add a report to your dashboard

You can save a report just as you would any other search. When you save the search, add it to your default dashboard by checking the box at the bottom of the save dialog.

You'll see the report on the dashboard after clicking the logo to return to the home page. Dashboard searches are refreshed every tenth of the time interval (for example, a 4 hour search every 24 minutes) or every hour, whichever is shorter.

You can read more about saving searches to the dashboard in Manage saved searches.

Note: You won't see your report on your dashboard if you haven't loaded any data to your main index yet. As soon as you have data in your main index, the "getting started" links are replaced with a default dashboard including modules that are predefined in the product, plus additional searches and reports you've added.

More searches

A Splunk search consists of one or more data-generating commands and their arguments, which can include literal keywords, wildcards, Boolean expressions, modifier name and value pairs, and subsearches. The generated data (search results) can then be used as inputs into other search commands in a search pipeline.

Splunk search commands are categorized by the type of operations they perform. You've already seen some examples of data generating commands. There are also commands that allow you to:

• Evaluate each result.
• Filter and re-order your results.
• Summarize large result sets.
• Add or remove fields in your results.
• Save data in different output formats.
• Perform administrative functions.

The following examples will demonstrate some of these capabilities. Refer to Search Commands for the complete list.

Report

Report commands, such as timechart, stats, top, and rare, summarize your results in the report window.

timechart

timechart returns statistics bucketed by time and is good for driving line charts. Try these examples.

Count of deny events graphed by time.

index=sampledata deny | timechart count(_raw)

Sum of bytes for GET requests:

index=sampledata sourcetype="access_common" GET | timechart sum(bytes)

Average bytes by method:

index=sampledata sourcetype=access_common | timechart avg(bytes) by method

stats

stats provides summary calculations by any field.

Total bytes sent by destination.

index=sampledata sourcetype=syslog | stats sum(sent) by dst

top

Let's get the top denied source IP addresses. Try it with a column graph.

index=sampledata netscreen deny | top src

rare

You can also get the 10 least common source IPs (by using rare).

index=sampledata netscreen deny | rare limit=10 src

Transform

Transform commands, such as transaction and diff , allow you manipulate the fields and values in your search results.

transaction

This search takes events from the access logs and creates a transaction from events that share the same clientip value that occurred within 5 minutes of each other (within a 3 hour time span).

index=sampledata sourcetype=access_combined | transaction fields=clientip maxpause=5m maxspan=3h

diff

Search for errors in syslog and diff the first and third results.

index=sampledata error sourcetype=syslog | diff pos1=1 pos2=3

Compare the host field of the last search.

index="sampledata" error sourcetype="syslog" | diff pos1=1 pos2=3 attribute="host"

Re-order

You can modify the order of your results based on different fields.

sort

Use the sort command to re-order the top 100 src field values of netscreen deny events.

index="sampledata" netscreen deny | top limit=100 src | sort src

Filter

You can define constraints to modify your search results.

set

Return all URLs that have 404 errors but no 303 errors (using set).

index=sampledata | set diff [search 404 | fields url] [search 303 | fields url]

regex

Use the regex command to filter results out of your search results. Specify a regular expression in regex to remove results that do not match.

Note: if you want to use the "or" ("|") command in a regex argument, the whole regular expression must be surrounded by quotes (ie. regex "<expression>").

The following example gets sendmail events that contain IP addresses in the non-routable class A (10.0.0.0/8).

index=sampledata sendmail | regex _raw=(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)

Note: The regex command supports inclusion of PCREs (Perl Compatible Regular Expressions).

Evaluate

You can perform operations directly on your data while searching.

fields

Use the fields command to specify the particular fields you want to see in your results. Here we will display only the src and dst fields of the sampledata netscreen deny events.

index="sampledata" netscreen deny | fields src, dst

Add a comparison

Let's go back to our top source IP addresses and filter for ones with more than 5 denies by using a logical comparison in the search command.

index=sampledata netscreen deny AND (count>5) | top limit=100 src

Refer to the Search Syntax for more information on Comparison Operators.

Use subsearches

Now we're going to put it all together by doing another search to find which of the actions with more than 2 500 http status codes also had 200 successes.

index=sampledata 200 [search index=sampledata 500 AND (count>2) | top action | fields + action]

CLI searches

Learn how to use the CLI.

Use the Splunk Command Line Interface (CLI)

About Splunk's CLI

You can use the Splunk CLI to monitor, configure, and execute searches on your Splunk server. Your Splunk role configuration dictates what actions (commands) you can execute. Most actions require you to be a Splunk administrator.

To access Splunk CLI, you need either:

• Shell access to a Splunk server, or
• Permission to access the SOAP port on a remote server.

Search your data in the Splunk CLI just as you can in Splunk Web.

Note: (If you have administrator or root privileges) To save a lot of typing, add the top level directory of your Splunk installation to your shell path. The $SPLUNK_HOME variable refers to the top level directory. Set a SPLUNK_HOME environment variable and add $SPLUNK_HOME/bin to your shell's path.

This example works for Linux/BSD/Solaris users who accepted the default installation location:

# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

This example works for Mac users who accepted the default installation location:

# export SPLUNK_HOME=/Applications/Splunk
# export PATH=$SPLUNK_HOME/bin:$PATH

Access help in the CLI

Find a complete CLI help reference by using the command help. Access the default CLI help page by typing the following in the command line while Splunk is running:

./splunk help

Access help on a specific CLI command, or topic by typing:

./splunk help command name | topic name 

For example, access a help page about Splunk search commands:

./splunk help search-commands

Note: Notice the dash (-) between the words: "search" and "commands". This is because the Splunk CLI interprets spaces as breaks. Use dashes between multiple words for topic names that are more than one word.

CLI commands

If you have administrator privileges, you can configure and monitor your Splunk server (or servers) using the CLI. Refer to the CLI command reference in this manual for a full list of capabilities.

Note: CLI commands are NOT search commands. Search commands are arguments in the search and dispatch CLI commands.

auth and uri parameters

Use the auth and uri parameters with any CLI command.

auth

Use auth with commands that require authentication to execute. auth is useful if you need to run a command
that requires different permissions to execute than the currently logged in user has.

Note: auth must be the last parameter specified in a CLI command argument.

Syntax:

./splunk command object [-parameter value]... -auth username:password 

uri

Use uri to send commands to another Splunk server.

Syntax:

./splunk command object [-parameter value]... -uri specified-server (=  [http|https]://name_of_server:port)

Note for Mac users

Mac OS X requires you to have superuser level access to run any command that accesses system files or directories. Run CLI commands using sudo or "su -" for a new shell as root. The recommended method is to use sudo. (By default the user "root" is not enabled but any administrator user can use sudo.)

Enable Splunk to start on system start-up

Enable Splunk to start at system start-up by executing the command: ./splunk enable boot-start.

Splunk automatically creates a script and configuration file in the directory: /System/Library/StartupItems. This script is run at system start, and automatically stops Splunk at system shutdown.

Note: If you are using a Mac OS, you must have root level permissions (or use sudo).

Note: You need administrator access to use sudo..

Example:
Enable Splunk to start at system start up on Mac OS using:

just the CLI::

./splunk enable boot-start

the CLI with sudo:

sudo ./splunk enable boot-start

Start live tail

You can use live tail in the CLI.

To start live tail:
1. Log into Splunk. ./splunk login

2. Use the live-tail CLI command to start live tail.

3. Type: ./splunk live-tail "your search string", where "your search string" is whatever simple search terms you want to search for (surrounded by quotes).

Search in the CLI

Run searches in the CLI using the CLI search command. Searches in the CLI work the same way as searches in Splunk Web except there is no timeline rendered with the search results, and a time range isn't specified by default.

Access these CLI help pages from the command line for help with CLI searches:

For a complete list of search commands available in the CLI type:

./splunk help search-commands

For a quick reference on CLI search command syntax type:

./splunk help commands

CLI Search syntax

In general, the syntax you use for search commands and arguments in the CLI is the same as you use in Splunk Web. In Splunk Web the search command is automatically prepended to a search when you use the search bar. You can search for anything you would normally search for in the CLI by using the CLI search command.

When you search using the CLI search command, quote the search string (argument of the search command) with single-quotes. You can use double-quotes, but this can cause parsing errors when you use double-quotes to express field arguments.

General form of a CLI search command string:

./splunk search 'search string' [-parameter] 

Differences between searching in Splunk Web and the CLI:

• Remember to quote search strings using single quotes to avoid any parsing errors.
• You can't use search commands that produce charts or graphs in the CLI.
• Use * to search for all events in a CLI search (example: ./splunk search '* | top' - returns top 10 events from all your data).
• Use time-based search modifiers to specify time ranges for your CLI searches.

Examples of CLI search

These examples illustrate the difference between searches in Splunk Web and searches in the CLI.

Example 1

This example selects events whose _raw field contains IP addresses in the non-routable class A (10.0.0.0/8).

Splunk Web:

| regex _raw=(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)

CLI:

./splunk search '* | regex _raw="(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)" '

Example 2

This example returns all URLs that have 404 errors but not 303 errors.

Splunk Web:

index=sampledata | set diff [search 404 | select url] [search 303 | fields url]

./splunk search ' index="sampledata" | set intersect [search "404 | select url"] [search "303 | fields url"] '

Example 3

This example extracts the COMMAND field only when it occurs in rows that contain "splunkd".

Splunk Web:

| multikv fields COMMAND filter splunkd

./splunk search ' * | multikv fields COMMAND filter splunkd '

Dispatched searches

The Splunk CLI allows you to run multiple searches asynchronously using the dispatch CLI command. Use dispatch to report on a large amount of data where the search could take days and you still want to be able to run other searches with Splunk. dispatch works the same way as search except you are not limited to running a single search. Set when to end a dispatch search by setting a maximum time (-maxtime) or a maximum number of results to output (-maxout).

To run more than one dispatch (search):
1. Execute a dispatch command in your current shell window.
2. (Leave your current window open.) Open a new shell window.
3. Execute another dispatch command.
Repeat!

Access the dispatch CLI help page at the command line for full detail of its syntax:

./splunk help dispatch

Note: Use dispatch to search without a maximum result limit.

Syntax

dispatch uses the same syntax as the CLI search command.

./splunk dispatch 'search string' [-parameter]

Optional parameters

Caution: Do not set the optional parameter -maxout to a large value near the max value (use a value that is less than 500 for -maxout). Setting a large value causes Splunk to hang indefinitely in some cases.

Examples

This example starts a search for events on all sources that contain "hot" and returns the count of events. The maxtime parameter sets the search to only run for 3 seconds.

./splunk dispatch "source=*hot* | stats count" -maxtime 3

This example searches for events from the access source types with byte counts greater than 1000. The maxout parameter sets the search to run until its returned 200 search results.

./splunk dispatch "sourcetype=access* bytes>1000" -maxout 200 

CLI search parameters

maxresults

By default, Splunk returns 100 search results when you search in the CLI. Splunk also passes only 100 search results as arguments to search commands in your search pipeline. Change the maximum number of results passed to search commands by using the -maxresults parameter of the CLI search command.

Set a new value for maxresults (0-50000) by adding the maxresults parameter after your search string.

Caution: Setting -maxresults to a high value causes searches to run very slow. Splunk recommends that you search in Splunk Web if you want your search to return a higher number of results.

Syntax:

./splunk search 'search string' -maxresults <value>

Example:

This example searches for 404's from web server events and returns only 5000 events (by setting -maxresults 5000).

./splunk search '404 host=webserver bigcompany.com | top source'  -maxresults 5000

Use form search

Form search

A form search is a saved search that has form fields that you must fill in before you run a search. Save any complicated search, and make it reusable as a form search (learn how to create form searches).

Form searches are saved searches that appear as forms when run. Save any search with form fields that a user running the search must fill out with parameters to run the search. You can create a sophisticated saved search and save it as a form with as many form fields as you like.

For example, you can define a search that returns all Web server errors for any username to be specified at search time:

503 OR 500 OR 404 sourcetype=access_common $user$

When run, this search appears as a form labeled user.

The search 503 OR 500 OR 404 sourcetype=access_common is still part of the search, but does not appear to the user.

Note: Form search works via text substitution, so the form fields can consist of anything, not just an indexed or an extracted field.

Run a form search

Form searches are saved searches. Run a form search by selecting it from the "Saved searches" menu in the search bar drop-down in Splunk Web.

If the saved search you select is a form search, then you'll be prompted with a form dialog like this:

Fill out the values in the form.

Note: You can substitute any text (not just a field) in a free-form text box in the form.

Refer to the Admin guide section on form searches to learn how to create form searches.

Use Live Tail

Live tail

Live tail lets you monitor data that is coming into Splunk in near real-time. See streaming search results; search for any text in data as soon as it is indexed into Splunk. Live tail streams data to your browser based on a simple text search.

You can use live tail for a lot of different things, for example:

• Passive monitoring
  • You'll know when specific events occur.
• Troubleshooting
  • Set up live tail to search for a particular type of event and set it to monitor your environment.
  • Change your environment and monitor the effects in the live tail stream.
  • For example, send an email and see whether it passes your spam filter.

Use live tail in Splunk Web

To start live tail, select the View in live tail menu item in the search bar drop-down menu.

Live tail launches in a new window (or new tab - depending on your browser configuration). The live tail processor takes the search terms you input in the search bar(before they are piped to data processing commands), creates a search based on them, and streams data to your browser that matches the search.

The live tail interface

The live tail interface is a separate window opened when you click View in live tail in the search bar drop-down menu. The controls available to you in the live tail window are listed here.
Live tail interface controls:

• Enter search terms in the live tail search bar.
• Click the green button to launch a new stream based on the search terms entered in the live tail search bar.
• Press ctrl-c to terminate the current stream (just like with tail -f in a Linux or Unix shell).
• Use the Wrap results check box to word-wrap the search results just like in Splunk Web.
• Press the Enter key anywhere outside the search box to insert a new line in the current stream.
• Press ctrl + shift + b to pause or un-pause live tail.
  • On a Mac, use cmd + ctrl + b.

Start live tail from the CLI

1. Log into Splunk. ./splunk login
2. Use the live-tail CLI command to start live tail.
3. Type: ./splunk live-tail "your search string", where "your search string" is whatever simple search terms you want to search for (surrounded by quotes).

Current limitations

The following are current limitations of live tail:

• You can only perform a simple text search while using live tail. You can't use any Splunk search commands or any data extractions in a search.
• If the client is overloaded by the volume of the data coming in to the processor, it will arbitrarily omit chunks of data. This means that with a very high volume of data, some events may never be displayed on screen for live tail.
• If you are monitoring a large number of files, there may be a slight lag before data is displayed in the live tail window.
• User permissions must be configured properly to allow access to live tail. By default, live tail will not work for the User role. It should work by default for users assigned to the Power or Admin role.

Use transaction search

Transactions

Transactions enable you to search, and report on transactions in your data indexed in Splunk. A transaction is a grouping of events that contain related pieces of information. Transaction search is useful for a single observation of any physical event stretching over multiple logged events.

Here is an example of what can make up a transaction:

• An event in a Web access log
• An event in an application server log
• An event in an asynchronous fulfillment application that generates a message queue that subsequently identifies a business transaction

In this case, the Web access log might share a session ID with the event in the application server log; the application server log might contain the account ID, transaction ID, and product ID; the transaction ID may live in the message queue with a message ID, and the fulfillment application may log the message ID along with the shipping status. All of this data represents a single user transaction.

Example use cases

There are many cases where a transaction search may be useful. Here are some use cases for transaction search:

• Find quarantined mail messages where:
  • All events printed to a mail log for the different steps in processing a given message are considered to be a transaction.
  • All events in the transaction share a message ID or a process ID with events that have both a process ID and message ID providing the transition.
  • Queue entry events containing the sender's email address, and separate delivery events for each recipient of the message.

• To find a security issue:
  • A sequence of 3 specific error messages in a particular order within a time frame on a given host that may be consistent with a known failure or attack pattern.

• To combine any combination of specific event types:
  • Any transaction from a single data source that generates multiple log entries for a single event.

The transaction search command

Search for transactions using the transaction search command. transaction yields groupings of events which may then be used in reports. To use transaction, either call a pre-configured transaction type, or define (or override) transaction constraints during your search via transaction command options.

Example transaction searches

Run a search that groups together all of the pages a single user (or client IP address) looked at over a time range.

This search takes events from the access logs, and creates a transaction from events that share the same clientip value that occurred within 5 minutes of each other (within a 3 hour time span).

sourcetype=access_combined | transaction fields=clientip maxpause=5m maxspan=3h

Transactions and macro search

Transactions and macro search are a powerful combination that allow substitution into your transaction searches. Make a transaction search and then save it with $field$ to allow substitution.

When to not use transactions

Transactions aren't the most efficient method to compute aggregate statistics on transactional data. If you want to compute aggregate statistics over transactions that are defined by data in a single field, use the stats command.

For example, to compute the statistics of the duration of a transaction defined by the field trade_id:

* | stats min(_time) as earliest max(_time) as latest by trade_id | eval duration = latest-earliest | stats min(duration) max(duration) avg(duration) median(duration) perc95(duration)

Similarly, if you want to compute the number of hits per clientip in an access log:

sourcetype=access_combined | stats count by clientip | sort -count

Also, compute the number of distinct sessions (parameterized by cookie) per clientip in an access log:

sourcetype=access_combined | stats dc(cookie) as sessions by clientip | sort -sessions

Use tagging

About tags

Tags help you group search results that share field values. Attach a name, or tag, to a group of results that share the same value of an indexed field (for example: event type, host, or source), or an extracted field. Apply any number of tags to any field, event type, host, or source (Learn how to tag fields), with the exception of sourcetypes (see Source type aliases, below).

You can use tags to:

• Help you track abstract values of extracted fields, like IP addresses or ID numbers. For example, you could have an IP address related to your main office with the value 192.168.1.2. Tag that IPaddress value as mainoffice, and then search on that tag to find events with that IP address.
• Use one tag to group a set of extracted field values together, so you can search on them with one simple command. For example, you might find that you have two host names that relate to the same computer. You could give both of those values the same tag. When you search on that tag, Splunk returns events involving both host name values.
• Give specific extracted fields multiple tags that reflect different aspects of their identity, which enable you to perform tag-based searches that help you quickly narrow down the results you want. To understand how this could work, see the following example.

Example:
Let's say you have an extracted field called IPaddress, which refers to the IP addresses of the data sources within your company intranet. You can make IPaddress useful by tagging each IP address based on its functionality or location. You can tag all of your routers' IP addresses as router. You can also tag each IP address based on its location, for example: SF or Building1. An IP address of a router located in San Francisco inside Building 1 could have the tags router, SF, and Building1.

To search for all routers in San Francisco that are not in building 1, you'd search for the following:

tag=router tag=SF NOT (tag=Building1)

For another example of using tags to search, you can watch this Splunk developer video.

Source type aliases

Source type aliases are similar to tags with the exception that you can only apply a single alias to a specific source type (though you can apply the same alias to a set of source types). Read more about sourcetype aliasing.

Search for extracted fields associated with tags

Search for tags by using the tag search modifier. The following examples show how to search for extracted fields (such as events, hosts, or sources) using the tag modifier.

When you tag an extracted field value (such as an event type or a host name), Splunk adds the tag next to the value in the main dashboard. Search using the tags in the main dashboard by clicking on them. If you're searching for a host tag, Splunk adds hosttag="tagname" in the search bar when you click on a host tag, and it adds tag::source="tagname" to the search bar when you click on a source tag.

The following examples search for events that have date_year field values associated with the tag "2007."

tag::field=tag

Example:

tag::date_year=2007

tag::field::tag

tag::date_year::2007

This example would return all events with field values that have been tagged with "2007." It would return all of the events found in the preceding two examples, as well as any other events with fields whose values were tagged with "2007."

tag=tag

Example:

tag=2007

Configure tags

The tags.conf file holds all of the tag definitions that you create through Splunk Web. You can use tags.conf to add or remove tags directly, create tag backups, and share sets of tags among Splunk servers. Learn how to configure tags via tags.conf.

Configure roles for tagging

Your role configuration and those of others can include specific role-based tagging capabilities. A Splunk administrator must define the ability to create, edit, or delete tags in your role configuration by editing authorize.conf.

Use reporting

Chart gallery

You can use Splunk to generate many types of reports. Run reports on your data by:

• clicking on Report on results >> after you search.
• selecting a field from the fields menu to report on.
• using a reporting command from Splunk's search language.

Check out the tutorial to learn how to create reports.

Column (or bar) chart

Use a column chart to compare the frequency of values of fields in your data. The x-axis values are typically field values (or time) and the y-axis can be any other field value, count of values, or statistical calculation of a field value.

Example:
This report shows the total kb (Kb indexed) into Splunk over time (the past 24hrs.).

Stacked column (or bar) chart

You can also use a stacked column chart to compare the frequency of values of fields in your data. A stacked column chart is the same as a column chart, except all of the columns are segments of a single column. The value of the total column is the sum of the segments.

Note: You can use a stacked column chart to highlight the relative weight (importance) of data within a set of your data.

Example:
This report shows the number of orders placed over time for each of the products offered by an online store.

Line chart

Use a line chart to show trends in your data over time or another field. You can display more than one series in a line chart.

Example:
This report shows which machines were the most requested (by destination IP address) over time. The series is split by the dst_ip field to make it easy to compare them.

Area chart

Use an area graph to show trends in your data over time or compared to another field value. The shaded areas under the data points help emphasize quantities.

Example:
This report shows the top occurances of the buddy field.

Stacked area chart

Use a stacked area chart to show multiple series the trends in your data the way an area chart can. You can show how a series of data relates to the whole set of data.

Example:
This report shows CPU utilization, broken out by individual processes, on all machines that report process statistics.

Scatter plot

Use a scatter plot to show trends in the relationships between discrete values of your data. Generally, a scatter plot shows discrete values that do not occur at regular intervals or belong to a series. This is different from a line graph, which usually plots a regular series of points.

Example:
This report shows machine and service access over time. The scatter plot counts the number of distinct accesses by external machines.

Pie graph

Use a pie graph to show the relationship of parts of your data to the whole set of data. The size of a slice in a pie graph is determined by the size of a value of part of your data as a percentage of the total of all values.

Example:
This example shows the top number of occurrences of a user in the last 24 hours.

Doughnut graph

You can also use a doughnut graph to show the relationship of parts of your data to the whole set of data. Doughnut graphs are functionally identical to pie graphs. The size of a slice in a doughnut graph is determined by the size of a value of part of your data as a percentage of the total of all values.

Example:
This example shows the top number of occurrences of a user in the last 24 hours.

Bubble graph

Use a bubble graph to show trends and the relative importance of discrete values in your data.

Note: The size of a bubble indicates a value's relative importance.

Example:
This report shows the distinct count of events seen over time for the fields: Foreign, Local, and Path.

Heat map

Use a heat map to show the relative importance of field values or statistical calculations done on your data. A heat map shows the importance of a value relative to other comparable values using gradients of color.

Example:
This report shows the top values of the field buddy. The x-axis contains the discrete values of the field, and the y-axis is the count of how many times those values occurred.

Examples of useful reports

This section shows you some reports you can execute using data from the sampledata and _internal default indexes. Paste any of these reports into the search bar in Splunk Web to see it in action (unless otherwise noted). Some of the examples require you to substitute your own data to see meaningful results. If noted, some examples require you to install certain applications.

Note: Extract fields from search results to create reports using interactive field extraction in Splunk Web, or with the rex command in a search.

Internal Splunk log data

Create reports on Splunk internal data. Report on things like indexing performance and the CPU utilization of Splunk processes.

Indexing performance

See the indexing kps (indexing thruput):

index=_internal "group=thruput" | timechart avg(instantaneous_kbps)

Plot the average eps and kps by time:

index=_internal "group=thruput" | timechart avg(instantaneous_eps) avg(instantaneous_kbps)

CPU utilization of Splunk processes

See the CPU utilization of Splunk processes in a bar chart sorted in descending order:

index=_internal "group=pipeline" | stats sum(cpu_seconds) by processor | sort sum(cpu_seconds) desc

System monitoring data

Create reports on data from your system monitoring utilities with the Splunk for UNIX application. You must install Splunk for UNIX to run these reports. These samples illustrate what some of the system monitoring reports look like.

Download and install Splunk for UNIX from SplunkBase. Splunk for UNIX configures Splunk to index UNIX system monitoring utility data.

See the % CPU usage over time:

index=monitoring source=ps | multikv | timechart avg(CPU) by COMMAND

See the % CPU for processes that contain the word "splunkd":

index=monitoring source=ps | multikv filter splunkd | timechart avg(CPU) by COMMAND

See the % CPU used per user:

index=monitoring source=ps | multikv | timechart avg(CPU) by USER

Web access data

Create reports on your web access data with these examples, or use the Splunk for Web Access application. Download and install Splunk for Web Access from SplunkBase. The Splunk for Web Access application has a large number of saved reports that you can run on your Web access data.

The following examples illustrate some of the kinds of reports you can run on Web access data with Splunk.

See the top Web clients used:

index=sampledata sourcetype=access* | top limit=100 clientip

See a chart of data transferred (in bytes) by time:

index=sampledata sourcetype=access* | timechart sum(bytes)

Web traffic stats

Here are some examples of Web traffic reports you can create.

Note: You must extract fields for the clientip and referer data from your search results to run these reports.

See the number of unique visitors by hour of the day:

index=sampledata sourcetype=access* | dc(clientip) by date.hour

See the top traffic peak hours of a day:

index=sampledata sourcetype=access* | top date.hour

See the number of unique visitors by day of week:

index=sampledata sourcetype=access* | dc(clientip) by date.wday

See the top traffic peak days of a week:

index=sampledata sourcetype=access* | top date.wday

See the top visits by referrer:

index=sampledata sourcetype=access* | top referrer

Web application data

Create reports from data of Web applications.

See the top Web applications and their violations from a log:
(This example assumes you have the web_app and violation fields extracted.)

sourcetype="webbapp" | top web_app violation

See the top Web applications and the top 3 violations for each application (using the sort command at the end to optionally sort the results):

sourcetype="webbapp" | stats count by web_app violation | sort - count | dedup 3 web_app | sort web_app

See the top Web applications and the top 3 violations for each application, and restrict the search to only include the top 10 Web applications (using the subsearch to pre-filter the results).

sourcetype="webbapp" [search <data source> | top 10 web_app | fields + web_app] | stats count by web_app violation | sort - count | dedup 3 web_app | sort web_app

Firewall (or connection) activity

Create reports on the connection activity of your server or firewall. You can report on any type of activity: # of connections, # of accepts, # of denies, bytes transferred.

See the top port activity:

index=sampledata | top port

See the top activity of a source host ID:

index=sampledata | top SourceId

See the most active destination ports:

index=sampledata | top limit=100 dst_port

See the ports with the top number of denials:

index=sampledata action=Deny | top src_port

Email activity or email transactions

Create reports of email activity by applying these examples to your own email data.

Note: These examples use the rex command to extract the fields on which to report. You can also extract fields by configuring field extraction at index time.

Note: The sampledata index doesn't contain email data used to demonstrate these reports. To try these, supply email data from your own sources and tailor these examples to try these reports.

See the top senders:

sourcetype=*mail* | rex field=_raw "from: (?<from>.*)" | top from

See the unique number of recipients to whom a sender sent messages:

sourcetype=*mail* | rex field=_raw "from: (?<from>.*) to:(?<to>.*)" | stats dc(to) by from

See the top recipients of messages:

sourcetype=*mail* | rex field=_raw "from: (?<to>.*)" | top to

Transaction data

For example, let's say that you have a set of purchase transaction data that looks like this:

1216725748296,I,toystore,S,1
1216725748396,I,toystore,S,1
1216725748497,E,toystore,S,30
1216725748598,E,toystore,S,40
1216725748698,I,toystore,S,1
1216725754024,I,toystore,S,2
1216725754126,I,toywarehouse,S,2

Now, let's assume that you configure Splunk to extract the following fields from this comma-delimited data:

• The timestamp in milliseconds since Jan 1, 1970 epoch.
• A character code indicating logger priority (I = info, E = error).
• The supplier name.
• A character code indicating business stage (S = customer is searching).
• A numerical event code indicating the outcome of the process.

With these fields, you can generate the following reports:

See a table showing the count of each supplier by logger code (priority):

* | chart count by supplier, priority

See a table showing the count of each supplier by logger code (priority). Also, rename the priority=I as successCount, and priority=E as failCount, then fill the empty values with 0:

* | chart count by supplier, priority | rename I as successCount, E as failCount | fillnull value=0

See a table showing the count of each supplier by logger code with the sum of successCount and failCount as totalCount (in a 3rd column):

* | chart count by supplier, priority | rename I as successCount, E as failCount | fillnull value=0 | eval totalCount = successCount + failCount

Search reference

Splunk search

Searching in Splunk is easy - type any term you'd expect to find in your data into the search box and hit Enter. A Splunk search lets you search indexed data in real-time, extract data from search results, and produce meaningful reports from the data you put into Splunk. For example, to search for all events containing a given IP address, type it into the box. Try it with usernames, error codes, transaction IDs, or whatever else you are looking for.

See the Search syntax page to learn about Splunk search syntax.

A search is pipeline of commands (similar to a Unix "|" pipeline) that starts with a command that gathers data (typically a search on data in a Splunk index), followed by data-processing commands that operate on the data to yield search results.

See the Search pipeline syntax page for details about the syntax of the search pipeline.

You can also watch this Splunk developer video about searching with Splunk.

Generate search results

Generate search results by using a data-generating command. Generate search results by using:

• search to get new search results from a Splunk index.
• savedsearch to execute a saved search.
• file to import previously gathered search results directly from a file.
• crawl to search your filesystem for new data sources to add to your index (returns them as search results).

Construct searches

Use the search command to construct simple keyword searches on data in your Splunk index (just like a Google search). Narrow your keyword searches with modifiers, fields, Boolean operators, and logical comparison operators.

You can also construct more powerful searches by using additional commands to extract data, perform statistical operations, and build reports. Learn about the search commands in the search command reference.

As you construct your search, Splunk's typeahead functionality will prompt you with predictive text based on the commands you are using and the contents of your data.
Note: Wildcards (*) are not supported. You can simply type and wait for the predictive text to complete your term.

Here are some valuable points to remember when constructing a search:

When generating data

To get more results:

• Increase the time range (search over all time to get the most number of results).
• Use wildcards and partial keywords instead of exact keywords.

If you want a faster search:

• Narrow the time range of your search.
• Reduce the number of fields that are extracted by un-selecting fields in the Fields picker menu.
• Reduce the segmentation by selecting inner, outer, or raw in the Preferences menu (reducing segmentation makes extracting and reporting more difficult).

When narrowing your search

• Filter results by event type if possible.
• Use the source and sourcetype fields to narrow a search to only a specified source.
• Use a combination of logical and Boolean expressions of keywords, modifiers, and fields.

Types of search

Form searches

Form searches are reusable searches that are pre-defined by a Splunk administrator. Form searches allow you to run complex searches by simply inputting variables in form fields. Learn more about Form searches.

Macro searches

Macro searches allow macro substitution of variables in saved searches. This allows you to run a complex search repeatedly with different variables. Learn more about Macro searches.

Transaction searches

Transactions let you search for groups of related events that are pre-defined as a transaction by your Splunk administrator. Use the transaction command to execute a transaction search. You can override specifications of a pre-defined transaction, or define a new transaction with the transaction command. Learn more about Transactions.

Live tail

Live tail allows you to see data as its being indexed into Splunk in real-time (similar to Unix's tail -f command). Live tail allows you to execute a simple search in its stand-alone window, and monitor events that match the search. Find out more about live tail.

Asynchronous searches

The Splunk CLI allows you to run multiple searches asynchronously. Use this if you have a search or report you want to run on a large amount of data where the search could take days and you still want to be able to run other searches with Splunk. Use the dispatch CLI command to execute asynchronous searches. Learn more about asynchronous searches.

CLI searches

Run searches in the CLI with the search CLI command. Searches in the CLI work the same way as searches in Splunk Web except there is no timeline rendered with the search results, and a time range isn't specified by default. Search for anything by including your search as the 'search string' argument of a CLI search command.

Learn more about CLI searching.

Save and schedule searches

After you save a search, you can set your searches to run regularly and schedule alert conditions. Read more about Save, schedule, and alert options.

Tune search performance

Splunk's searches are optimized for text-based searching of raw event data. Search speed is dependent on how your Splunk install is configured. You can improve the speed of your searches by editing configuration files, and by downloading various add-ons from SplunkBase. Read more about tuning search performance.

Search syntax

Splunk searches are designed to make searching your data easy by allowing you to perform a variety of simple term and phrase searches with Boolean and comparison logic using the Splunk search command. Use a Splunk search to explore your indexed data, and operate on it by piping search results to a variety of powerful commands to perform statistics and structured analysis on the results. Refer to the Search pipeline syntax reference for more about the search pipeline.

For example:
Search for error events from access logs.

error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) )

Keyword search

Search your data for any whole or part of a keyword by typing it into the search bar. Search for any number of keywords that you like in a single search. Separate multiple keywords by using whitespace. Splunk treats the whitespace between keywords as an AND operator.

Keywords are not case-sensitive.

Examples:
Search for "10." anywhere in your data.

10.

apache error

Literals ("quotes")

Search for any literal term by wrapping it in double quotes. To search for the literal equal sign, you must wrap the search term in double quotes. For example, to search for the string a=b rather than the field a with the value b you must search:

"a=b"

In general, Splunk recommends that you wrap terms that have punctuation or whitespace in double quotes.

Make search case sensitive

Use the regex command to match events generated in a search to an exact string of characters (including case sensitive terms pronouns). Search for all events, and then filter your search results using a regex. For example, search for the word "ERROR" in all of your events:

* | regex _raw=ERROR

Search for an asterisk (*)

You can't search for an asterisk (*) character as it is always treated as a wildcard within the search command. To search for a literal asterisk, you need to search for all data and filter out results that don't contain an asterisk (using the regex command).
Here's the search you need to execute to search for an asterisk:

* | regex _raw= \*

Wildcards

Use wildcards to search for keywords or phrases that match a partial string of characters. Place wildcards at