This document last updated: 01/08/09 03:01pm

Print User Manual

About this manual

What's in this guide?

This manual teaches end users about Splunk and how to use Splunk search. This manual focuses primarily on accomplishing tasks through the web interface, Splunk Web, and the command line interface.

To help you start using Splunk, run through the Tutorial. The chapters following the tutorial focus on tasks, such as adding inputs, searching, setting alerts, and building reports. The manual ends with reference pages for search syntax, commands, and modifiers.

Refer to the search cheatsheet to help you construct useful searches right away.

Take note:

• Before using Splunk, review the Installation manual for system requirements and step-by-step installation guides.
• System administrators should refer to the Administration and Deployment manuals for advanced configuration and data management topics.
• Those interested in customizing Splunk and creating applications for Splunk should refer to the Developer manual.
• You can also share custom Splunk applications with other members of the Splunk community via SplunkBase.

About Splunk

About Splunk

Splunk is an IT search engine.

• You can use Splunk to search and navigate IT data from applications, servers, and network devices in real-time.

• Data sources include logs, configurations, messages, alerts, scripts, code, metrics, etc.

• Splunk lets you search, navigate, alert, and report on all your IT data in real-time using Splunk Web.

About Splunk licenses

Each instance of Splunk server must have its own license. This topic discusses the different Splunk licenses, how to install or update a license, and what to do when you have a violation on your license.

Note: You must purchase a separate license for every instance of Splunk that you deploy.

Which license?

Splunk provides two standard types of licenses, a Free license and an Enterprise license. To evaluate Enterprise features, you can request a trial Enterprise license before purchasing.

Note: If you evaluate a Splunk Preview release, it will include the required license.

Free versus Enterprise

When you download Splunk for the first time, you are asked to register. Your registration authorizes you to receive the Free license, which allows a maximum indexing volume of 500 MB/day. The Free license is not a trial license and does not have an expiration date. The Enterprise license enables higher data indexing volume and the following additional features:

• Multiple user accounts and access controls.
• Distributed search and data routing.
• Deployment management.

To evaluate these features before you purchase an Enterprise license, you can request a 30-day trial Enterprise license.

Find more information about the different license features here. Also, read Splunk's Free license agreement.

Trial license

You can request trial Enterprise licenses of varying size and duration. The default evaluation period is 30 days. If you are running with a trial license and your license expires, Splunk continues to index your data. However, you will not be able to search until you install a new license.

Preview license

Splunk's Preview releases require a different license that is not compatible with other Splunk releases. Also, if you are evaluating a Preview release of Splunk, it will not run with a Free or Enterprise license. Preview licenses typically enable Enterprise features, they are just restricted to Preview releases.

Forwarding license

Each instance of Splunk server must have its own license. To configure Splunk for forwarding and receiving, contact Splunk Support and request a 1 MB/day license to install on each forwarder instance (any instance that is not indexing locally). For additional security, this license lets you configure different username and password pairs on each forwarder.

Note: This 1 MB/day forward-only license is not subtracted from your existing license(s) and can be applied to multiple forwarders.

Install your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license (splunk-free.license) or an Enterprise license (splunk.license). You can install and update your licenses with the CLI or from Splunk Web's Admin > License & Usage page.

Refer to the Installation Manual for instructions to install or update your Splunk license.

License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have more than 7 violations in a rolling 30-day period, search will be disabled. Search capabilities return when you have less than 7 violations in the previous 30 days or when you apply a new license with a larger volume limit.

Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks access while you exceed your license.

If you have other issues with your license, refer to the Admin Manual for troubleshooting tips.

Use Splunk Web

About Splunk Web

Splunk Web is Splunk's dynamic and interactive graphical user interface. It runs off of the splunkweb process, which is a Python-based application server. Use Splunk Web to search your IT data and manage your Splunk deployment. Access Splunk Web via a Web browser. Refer to the system requirements for our list of supported operating systems and browsers.

Splunk Web contains the search bar, dashboards, and configuration pages. You can run a custom search from any of the dashboards. Access the Preferences panel and the Admin pages with links on the top right corner of the dashboards, above the search bar. Access different dashboards from a drop-down menu located under the search bar and on the right.

For advanced Splunk Web customizations, see the Developer manual.

Dashboards

Dashboards are customizable pages in Splunk Web. You can add and remove components to and from each dashboard. These components may be lists of all indexed data, snapshots of different saved searches, or a list of saved searches.

Splunk ships with three default dashboards: getting started, main, and admin.

You can also mask default dashboards by following these instructions on "dashboard customization" in the Developer manual.

Getting started dashboard

The getting started dashboard is the default landing page for Splunk Web. It provides information and links to help new users learn how to use Splunk. There are many upcoming changes to the getting started dashboard.

In 3.3, the buttons for indexing data take you to the index manager:

• Clicking Index Files redirects you to the Admin > Data Inputs: Files & Directories: New Input page. You can enter Source, Host, and Source Type information to define local or remote files and directories to index.
• Clicking Index More Data redirects you to the Admin > Data Inputs: All page. You can select the type data to configure and add to your inputs.

Read more about the index manager and adding inputs.

Main dashboard

The main dashboard provides default modules, which include:

• Lists of all indexed data, sorted by Sources, Sourcetypes, and Hosts.
• A timeline of errors in the last hour.
• A list of saved searches.

Admin dashboard

The admin dashboard provides charts that report information a Splunk administrator may find useful:

• Messages by minute in the last 3 hours.
• The volume of data (KB/hr) over the last 24 hours.
• The number of Splunk errors over the last 24 hours.
• The daily indexing volume by server.

Custom dashboard

Instead of editing the default dashboards, we recommend creating a new dashboard to customize.

• From the dashboard drop-down menu, select create new dashboard... to name your new dashboard and add it to Splunk Web.
• Use the Edit options, located next to the drop-down menu, to select the saved searches or reports you want to add to a dashboard.
• Use the Delete option, located next to the drop-down menu, to remove one or more dashboard from Splunk Web.

You can customize the layout of your dashboard by editing prefs.conf. Refer to the Developer Manual for Customized Dashboard examples.

Preferences

Use the Preferences panel to configure Splunk Web's default search properties and general appearance and behavior. For more information, read Change Splunk Web preferences.

Search

Use the Search preferences tab to define:

• The default time range; You can always change the timerange from the dashboard.
• A maximum limit for the number of events Splunk indexes when you search.
• How Splunk Web handles segmentation.

Note: Splunk Web's segmentation setting affects how the browser interacts with Splunk and may speed up the display of search results. This setting should not be confused with indexing segmentation.

General

Use the General preferences tab to define:

• A default theme for Splunk Web. If you want to customize Splunk Web's appearance, refer to the Developer manual.
• Click behavior. You can click on sections of your search results to add or replace terms in your search. "Click behavior" configures either ctrl or ctrl-click to add and replace when narrowing your search.

Admin pages

In 3.3, when you click on the Admin link, to the top right of the page, the Server settings page opens. Instead of navigating a tabbed menu layout, you now access the Admin pages from a list located on the left side of the page. Click on the top-level section names to view the pages included in that section. You have access to the same pages as before (Server, Data Inputs, Distributed, Users, Saved Searches, and License & Usage) with the addition of Indexes and Applications.

Server

Use the Admin > Server pages to view and change server settings, restart the Splunk server, and change and reload Splunk's authentication method. Read more About Splunk server's settings and changing Splunk server's settings.

Data Inputs

Use the Admin > Data Inputs pages to add new and edit existing inputs in Splunk Web. You can view and manage all of your files and directories, FIFO queues, network ports, and crawls from this page. Read more About Inputs and using the Data Inputs page to add inputs.

Indexes

Use the Admin > Indexes pages to view a list of your indexes, edit individual index properties, and add new indexes. Read more About Indexes.

Applications

Use the Admin > Applications pages to manage existing applications and browse SplunkBase for new applications to install. Read more About Applications.

Distributed

Use the Admin > Distributed pages to view your network topology and configure search distribution, data forwarding, and data receiving between multiple Splunk instances. Read more About Data Distribution.

Note: You can only set up forwarding from this page if you are running Splunk with a Free license. To configure distributed search and data receiving, you must have an Enterprise license.

Users

Use the Admin > Users page to view a list of users and their search history, edit each user's properties, and add new users. Read more about Users and User Roles.

Note: You cannot access this page if you are running Splunk with a Free license; you must have an Enterprise license to modify user's properties.

Saved Searches

Use the Admin > Saved Searches page to view a list of your saved searches and edit their properties, create new searches, or delete existing searches. Read more about Managing saved searches.

License & Usage

Use the Admin > Licenses & Usage pages to view view your current license and replace it with a new one. This page displays the type of license you're running, the maximum indexing volume allowed, and when the license expires. This page also provides some useful statistics, such as: number of days before you need to renew, the peak usage in GB/day, and peak percentage. Read more About Licenses.

Change Splunk Web preferences

Use the Preferences panel to configure Splunk Web's default search properties and general appearance and behavior. The Preferences panel is a dialog box that opens when you click the Preferences link in the upper righthand corner of Splunk Web. Access the link on any of the dashboard pages.

The Preferences panel has two tabbed options: Search and General.

Change default settings in search preferences

Use the Search preferences tab to change:

• The default time range.
• The maximum search results.
• How Splunk Web handles segment selection.

Change default time range

To set the default time range for search, select one of the following options:

• Last 15 minutes
• Last 60 minutes
• Last 3 hours
• Last 24 hours
• Last 7 days
• Last Month
• Last 3 Months
• All time

You can always change the time range at search time from the dashboard.

Change maximum search results

The maximum search results is the maximum number of events splunk will return when you search. By default, it is 50000. Increasing or decreasing this number will affect Splunk's search speed.

Change segment selection

This setting affects how the browser interacts with Splunk and may speed up the display of search results. Splunk Web's segmentation setting is not related indexing segmentation.

You can set segment selection to:

• Raw: Disables segmentation entirely. This is the fastest setting, but disables clicking on events.
• Inner: Enables only minor breakers.
• Outer: Enables only major breakers.
• Full: Default setting, enables both inner and outer segmentation.
• Pyramid: Enables visualizing of event's segmentation.

Change default settings in general preferences

Use the General preferences tab to define:

• A default theme for Splunk Web.
• Click behavior.

Change default theme

Splunk Web is defined with HTML, CSS, Javascript, and XSL. You can customize a theme with CSS files to override the default styles for font, color, and images. If you want to customize Splunk Web's appearance, refer to the Developer manual. You can also watch this Splunk developer video about it.

Change click behavior

You can click on sections of your search results to add or replace terms in your search. "Click behavior" configures either ctrl or ctrl-click to add and replace terms when narrowing your search.

About Splunk server settings

Use the Admin > Server pages to view and change the Splunk server settings, restart the Splunk server, and change and reload Splunk's authentication method.

Important: When you change any of the server settings, you must restart Splunk for your changes to take effect.

View server settings

The Admin > Server: View Settings page is divided into three sections: basic settings, Splunk Web settings, and Datastore settings. Refer to Change Splunk server default settings for instructions to change these settings.

Basic settings

Under the Basic Settings heading, you can change your server name and splunkd port.

• The Splunk server name, which is the identity of your Splunk instance, defaults to the DNS or IP address of the Splunk sever host.
• Splunk management port number, though which your Splunk instance communicates with the splunkd daemon, defaults to port 8089.

Note: You cannot modify the installation path.

Splunk Web settings

Under the Splunk Web heading, you can enable or disable Splunk Web, enable SSL (HTTPS) in Splunk Web, and change the Web port.

• By default, Splunk Web is enabled. However, in a distributed environment, you may choose to disable Splunk Web on your forwarding instances.
• By default, SSL is not enabled.
• The default Web port number is 8000.

Datastore settings

Under the Datastore heading, you can change the default host name, the datastore path, and the minimum free disk space.

• By default, your Splunk server name is your Default host name. This name defines the value for the host field in your search results; it tags all events with this host value.
• The datastore location is the top-level directory where the Splunk server stores all indexed data, user information, and working files. By default, this path is set to $SPLUNK_HOME/var/lib/splunk.
• The minimum free disk space is a value that tells Splunk to stop indexing if the storage space reaches this limit (in MB). The default disk space is 2000 MB.

Control server

Use the Admin > Server: Control Server page to restart the Splunk server and reload the Authentication method.

Note: Unless you are running Splunk with an Enterprise license, you will not see the Reload Authentication method option.

Configure authentication method

Use the Admin > Server: Authentication Configuration page to change Splunk's authentication method.

Note: This feature requires an Enterprise license. If you are running Splunk with a Free license, Splunk Web will tell you how to update your license.

Change Splunk server default settings

After you start a new installation of Splunk, you may want to change a number of the default settings. This is a quick guide for using Splunk Web to change the admin default password, your Splunk server name, the Web and splunkd network ports, the datastore location, and the minimum free disk space.

You can make all of these changes from the Admin > Users and Admin > Server: View Settings pages in Splunk Web. Refer to the User Manual for more information About Splunk Web.

Note: You can also make these changes using Splunk's CLI.

Change the password

Splunk with a Free license does not require login authentication. However, Splunk with an Enterprise license does require authentication and ships with a default administrator account with username admin and password changeme.

To change the administrator password:

1. Navigate to the Admin > Users page in Splunk Web.

2. For the username admin, click Edit from the Action column.
The admin user properties page opens.

3. Enter your new password twice (under Password and Confirm Password).

4. Click Save.
You return to the Admin > Users page with a note at the top, "User admin updated."

Change Splunk server name

The Splunk server name is the identity of that particular instance. This name is displayed within Splunk Web and is sent to other Splunk servers in a distributed setting. The default name is taken from either the DNS or IP address of the Splunk server host.

To change the Splunk server name:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Basic Settings section, enter a new Splunk server name.

3. Click Save.

Change network ports

Splunk uses two network ports that default to:

• 8000: HTTP or HTTPS socket for Splunk Web.
• 8089: Splunk management port.

Note: Splunk Web, the command line interface, and any distributed connections from other servers use the Splunk management port to communicate with the splunkd daemon.

To change the network ports:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Basic Settings section, enter a new Splunkd port number (under Splunkd port #).

3. In the Splunk Web section, enter a new Web port number (under Web port).

4. Click Save.

Change datastore location

The datastore path is the top-level directory where the Splunk server stores all indexed data, user information, and working files. If you turn off local indexing and only forward data in a distributed setup, this server still requires a few megabytes of available space in the datastore path.

Important: If you change this directory, the server does not migrate old datastore files. Instead, it starts over in the new location. To migrate your data to another directory, refer to the instructions in Move an index.

To change the datastore location:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Datastore section, enter a new Datastore path.

3. Click Save.

Change minimum free disk space

The minimum free disk space defines a storage space limit before Splunk stops indexing. Splunk resumes indexing when more space becomes available. For more information on managing Splunk disk usage, see Disk usage.

To change the minimum free disk space:

1. Navigate to the Admin > Server: View Settings page in Splunk Web.

2. In the Datastore section, enter a new limit under "Pause indexing if free disk space falls below".

3. Click Save.

Tutorial

About this tutorial

This section briefly explains what you need to run Splunk and complete this tutorial.

Requirements

• A supported browser (See system requirements and release notes).
• A copy of Splunk:
  • On an individual machine, or elsewhere in the organization. (Download and install.)
  • On Splunk hosted demo servers.

Log in

Splunk does not require a login when using a Free license. An Enterprise license requires a login:

• For an individual server, the default Splunk username is admin and password is changeme.
• For the Splunk hosted demo server, the username and password are guest and guest.

Splunk Web

Splunk Web runs by default on port 8000 of the host on which it is installed.

• If you're running Splunk on your local machine, the URL to access Splunk Web is http://localhost:8000
• If you're running Splunk on a different machine, the URL to access Splunk Web is http://<hostname>:8000

where <hostname> is the name of the machine Splunk is running on.

Logging in to Splunk takes you to the dashboard and Splunk Web. For this tutorial, you only need to know that:

• You type your search query in the search bar at the top of the dashboard.
  • The down-arrow button, to the left of the search bar, is a drop-down menu.
  • The green button, to the right of the search bar, enters your search.
• You change the time range with the drop-down menu below the search box.
• Splunk displays the number of occurrences of your search results graphically on the timeline.
• Splunk lists each individual event that results from your search underneath the timeline and ordered by timestamp.

Read the other sections of the Splunk User Manual for more information.

Command line interface (CLI)

Splunk includes a command line interface (CLI) that runs from a shell on the server host. The Splunk CLI is a great way to integrate Splunk into admin scripts. Read Use the Splunk CLI for more information.

Simple searches

See a few simple searches in action. This section introduces keyword searches and how to narrow your search with Boolean logic and using the timeline. Before you begin, you can watch Splunk's developer video about search; it demonstrates how to search with a just a few words and clicks of the mouse.

Index data

Splunk comes with pre-indexed sample data, called sampledata, which we will use throughout this tutorial. You can search the index that has the sample data in it instead of the main index by including index=sampledata in the search criteria.

For help indexing your own data, see the data inputs section of the Admin Manual.

Search

To start, enter your search in the search bar at the top of the page. To search for all the data in the sampledata index, type the following into the search bar:

index=sampledata

The timeline should show bars indicating when matching results occurred. If there are no results displayed, change the time range until you see results.

Now, lets search for HTTP requests that resulted in an internal server error (code 500). Type this simple search:

index=sampledata http 500

Narrow your search

You can use arguments in the search command to narrow your search. Add Boolean logic between terms and modifiers, use logical comparison operators for field values, or use search modifiers. You can also use the timeline to zoom in on particular events. This section discusses two ways to apply Boolean modifiers to your search. We'll discuss using the timeline to narrow your results later.

Read Search results for more ways to manipulate search results.

Search with Booleans

Splunk supports the Boolean operators: AND, OR, and NOT (must be capitalized).

Enter the search:

index=sampledata http AND 500

Your results should match the previous example search. Similar to Google and other search engines, Splunk implicitly inserts an AND between terms by default.

Note: If your search produces no results, try zooming out, clearing the time range, or resetting the time range using the drop-down menu.

Search for all HTTP requests that do not contain error code 500:

index=sampledata http NOT 500

Search for all sampledata events of sourcetype access_common or syslog.

index=sampledata (sourcetype=access_common OR sourcetype=syslog)

Note: Splunk uses parentheses to group Boolean expressions.

Click on results

As you scroll through your results and mouse over sections of each event, you'll notice the sections are highlighted. You can highlight and click items in the results to add and remove terms in your search string.

Search for:

index=sampledata http

Scroll through the list of results. Click on "500" in one of the search results.

Notice that Splunk highlights and updates the search to add "500" as a term (in the search bar). This is a shortcut for applying the "AND" operator to your search.

index=sampledata http 500

Click on another instance of "500". Splunk removes the term from your search string and your search results include all HTTP results again:

index=sampledata http

Now, alt-click on "200" in any search result (option-click for Mac, alt-shift-click for some popular *nix windows managers).

Splunk now updates your search with "NOT 200"; This is a shortcut for applying the "NOT" operator.

index=sampledata http NOT 200

Follow a relationship

While you scroll through the list of results, you may find interesting events. For example, if you want to look only at activity on one particular IP address:

• click on an IP address (option-click on a Mac).
• Check Wrap results to turn on line-wrapping for the long single line events that result from searches.

Now, your results are a chronological list of events that occurred on this IP address. You can use this to trace a sequence on events. This is an effective way to follow relationships between events.

Use the timeline

The timeline shows bars and a red line (or flag). The bars indicate the volume of search results and when they occur along the span of your time range.

You can change the time range with the drop-down menu:

• Choose Custom.
• Specify a start and end time.

You can also customize the time range by clicking on any bar in the timeline and zooming in on a particular cluster of events:

• Set the time range to "Last month".
• Execute the search for all sampledata.

Notice that each bar is equivalent to one day of data.

• Click on the bar in the timeline showing the cluster of data.
• Click Zoom in.
• Repeat the last two actions to narrow the time range until you see a few more bars in the cluster of data.

Notice that each bar is equivalent to one minute of data.

• Shift-click or drag your mouse across all of the bars and zoom in further.

Note: The red flag marks the location of the results you are currently viewing along the timeline. As you scroll through your search results, the red flag shifts to follow.

The timeline now spans several minutes, with one bar equal to one second.

Note: Your Splunk instance will perform faster if you narrow the time range of your search. Searching over all time may result in slow search performance.

Search results

Splunk allows you to navigate search results by following links and using interactive field filters. Filtering is an efficient method to organize the results of a search.

Events and fields

Your search results appear below the timeline as a list of events ordered by timestamp. A field is a name/value pair distinguished from the free-form indexed segments that you see in an event.

You can add and remove field filters, extract new fields from the results, and tag fields to group results.

Filter on fields

Search for all the sampledata index events:

index=sampledata

Splunk includes three default filters in your search results: host, source, and sourcetype. These interactive field filters are drop-down menus located below the timeline.

Each field's filter menu lists (up to) 10 values, ordered by the frequency at which they occur in the search results.

Host

The host field, which lists the originating hosts of events, lets you target one specific host in the filter. The host field is stored and indexed with each raw event.

1. Click on the host menu.

2. From the list, select the first host value, http2.
The search results filter to show only results for the selected host. Note that this does not add the filter to your search; instead, it shows you a preview of your results. If these aren't the results you want, you can remove this filter and revert to your earlier search.

3. To remove the first host filter, click Clear filter.
The search results revert to your previous search.

4. Select the next host value from the menu.

5. To add this filter to your search string, click Add filter to search.
The search bar and search results update to include the host value restriction you applied, http1:

index=sampledata host=http1

Source

The source field lists the location where an event is accessed; a file, network port, script, etc.

Source type

The sourcetype field characterizes all sources that have similar formats. For example, all Apache access logs in W3C common format have the sourcetype value access_common. The sample data contains four distinct sourcetypes - syslog, access_common, db2 and websphere_activity.

Show more fields

You can include many more field filters in addition to host, source, and sourcetype, in your searches. The fields are listed in the Fields... drop-down menu.

Search for all the sampledata index events:

index=sampledata

Let's add a couple more field filters to our search:

1. To display the list of field filters, click the Fields... menu.

2. Scroll through the list.

3. Check eventtype and punct.

4. Click Apply.

The interactive field filters list updates to include eventtype and punct menus. You can use these field filters exactly the same way you used host.

To remove a field filter menu:

1. Click the Fields.. menu.

2. Uncheck the fields you want to remove.

3. Click Apply.

The eventtype and punct fields are discussed further in Event types.

Define custom fields

Splunk lets you interactively define and extract fields from your search results. Let's define a field to extract the IP addresses from our search for all events in sampledata.

index=sampledata

You may need to scroll through the results or use the timeline to find events that contain an IP address.

Below the timestamp of every event is a drop-down menu. Click the down-arrow and select Extract field.

The Extract fields window opens.

Notice the panel at the top of this window:

• Sample Event lists the event that you selected from the search results.
• Example Value(s) provides a textarea for you to define the field value.

To define the IP address field for extraction:

1. Highlight the IP address from your sample event. Copy and paste (or type) it into the Example Value(s) textarea.

2. Click Preview.

3. In the Rules panel:

• Select a field to restrict your search (either host, source, or sourcetype),
• Splunk generates the regex rules to define your field.
• Splunk provides a preview of values it has extracted from your results.

Splunk also provides a preview of other events that contain your custom field. Use this Preview panel to validate the results of your field definition.

4. To save your custom field definition, click Save.
The Save FIeld Definition dialog box opens.

2. Under Name, enter a name for the field. Type in "ipaddress".

3. Click Save.

Now, your custom field (ipaddress) is listed in the Fields menu. You can activate and apply your field filter in exactly the same way you used host.

Tag fields

You can tag fields to group together results that share field values. Use tagging to attach a name, or tag, to a group of results that share the same value of a field, event type, host, or source. You can apply as many tags as you want to a single field, event type, host, or source. A tag cannot contain spaces.

• To access the tagging dialog, click Tag field name in the drop-down menu of any field in your search results.
• To apply a tag to the field you selected, type a tag name into the Tags field.
• To apply multiple tags, enter a space-delimited list of tag names in the Tags field.

Note: Tags that you create for a field are displayed in italics next to that field name in your search results.

Collect snapshots

Splunk allows you to save your results in a "Snapshot Container" that houses your collection. Each snapshot includes an image of the time graph and your search string.

You can add and remove snapshots from your collection. However, after adding a snapshot, you cannot modify the time graph within the container.

If you want to modify a snapshot in your collection:

1. In the Snapshot Container, click Restore search.

2. Modify your graph.

3. Click Snapshot.
Your modified graph has been added to your snapshot collection.

Event types

Splunk allows you to classify events that have common characteristics and save them as a value in the eventtype field. Examples of event types include all ssh logins and all sendmail syslog messages. Editing, tagging, and naming such event types helps the Splunk server learn and improve its performance.

In this tutorial, you will search for a specific event and save your results as an event type which you can later apply to a new search.

Find similar events

Search for all events in the sampledata index:

index=sampledata

Because the format of an event is often a powerful part of defining an event type, Splunk indexes the punctuation characters of events as a field called punct. This field, while it looks cryptic at first, is a powerful way of finding similar events quickly.

To activate the punct field:

1. Click the FIelds... menu.

2. From the list of fields, check the punct field.

3. Select Apply.
The punct filter menu now appears below the timeline.

4. Click on this menu to view a list of the 10 punct values that are most common in your results:

5. Add a filter to your search for the most common punct value in sampledata, which might be:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__

Save as event type

We want to save the last search as an event type. First, let's add "logout" to the search string:

index=sampledata punct="..._-_-_[//:::]_\"_//?=_/.\"__" logout

Now, the results are all sampledata events with the punctuation scheme that are logout actions.

To save your results as an event type:

1. Click the search bar menu.

2. Choose Save as event type...

3. In the Save Event Type dialog box, name your event type "trade_app_logouts".

4. Click Save.

Now you can include your new event type in a search.

Note: If your event type name contains spaces, the spaces will be replaced with underscores and the tags will not be saved. When naming the event type, do not include spaces.

Search for an event type

Let's search for all events in sampledata again.

1. After the search results load, activate the eventtype field filter.

2. From this field filter, choose trade_app_logouts.

Notice that eventtype=trade_app_logout appears as a field in your results, underneath each event:

3. Now, you can add the filter to your search:

• Click on Add filter to search, or
• In the events list, click on a highlighted instance of the field.

You search string updates to:

index=sampledata eventtype=trade_app_logouts

Note: Identifying and saving an event type means you can search for it directly. In fact, if you know the field name and value, you do not need to activate the field filter to add it to your search.

You can also click on the down-arrow to the right of the eventtype instance to perform the following options:

• Report on all event types
• Search for this event type
• Tag event type
• Rename event type
• Delete event type

Tag an event type

You can tag event types that have very different search terms and punctuation patterns with common words, then find all events of types that have any tag. This is a great way to create higher level classifications like "logouts" that cross different logout event types from different applications.

You can also tag hosts with one or more words describing their function or type, enabling users to easily search for all activity on a group of similar servers. Tagging hosts is useful for knowledge capture and sharing and for crafting more precise searches.

See the section in our Admin manual on Host tagging.

Automated event type discovery

Splunk automatically discovers event types based on seeing a large number of events sharing common characteristics. You can edit, delete, rename, and tag event types that Splunk discovers. You can also make your own event types by saving any search as an event type.

Learn how to use automated event type discovery in Splunk Web.

You can also change the settings that determine which keywords are considered in event type discovery in its eventdiscoverer.conf configuration file. Learn how to configure eventdiscoverer.conf.

Save options

You can save any of your searches, schedule your saved searches, and define alert conditions for your scheduled searches. For more information, refer to the User Manual topic about Save, schedule, and alert options.

Save a search

Search for the trade_app_logouts events in the sampledata:

index=sampledata eventtype=trade_app_logouts

To save a search:

1. Click on the search bar menu.

2. Select Save search... from the menu.
The Save Search dialog box opens.

3. In the "Search options" tab, name your search. (In 3.3, this is Search.)

4. Click Save.

Note: When saving your search, you can choose to add it to one or more dashboards.

Splunk lets you delete or modify your saved searches and add them to the dashboard. For more information on how to manage saved searches, refer to the User Manual's Find and manage saved searches page.

Schedule the search

From the search bar menu:

1. Choose Save search...

2. Click the Schedule & Alerts tab. (In 3.3, this is Schedule and Alert.)

3. Under Schedule, check "Run this search on a schedule".

Note: You can define the schedule frequency with the Basic or Cron options.

Schedule an alert

You can also trigger a shell script, such as a script to generate an SNMP trap or call an API to send the event to another system. If you need additional email options (like setting the From: address) see the Alerts page in the Admin manual.

Reports

Splunk allows you to summarize the results of any search as a report in a separate window.

You can access the reports window in three ways:

1. After running a search, click Report on results >> located below the search bar.

2. Select Report on this field >> from any interactive field filter menu.

3. Pipe your search results into a report command, such as stats, top, and rare.

We'll cover pipes and other commands in More searches.

Report on results

Let's build a report for all firewall deny events in sampledata:

1. Search for all firewall deny events in sampledata.

index=sampledata deny

2. After the results load, click Report on results >> above the timeline options. This takes you to a separate window where you can build your report.

Notice that:

• You can enter new search strings from the search bar at the top of the window.
• Splunk identifies fields from your search results and lists the field names in the Fields panel.

3. Select dst from the Fields list.

Splunk updates your search string to:

index=sampledata deny | top limit=100 dst

The report displays:

• A chart graphing the results (the top 100 values of dst).
• A summary of the count and events matching your search.

Notice that the options in the Series panel defines the data series for your chart. You can also choose a different chart to display your results.

Let's tune this search to report only the top 10 dst values of firewall deny events and display the series in a pie graph.

4. In the search bar, change the limit boundary to 10 and enter the search:

index=sampledata deny | top limit=10 src

5. In the series panel, select display as "pie graph".

5. When you mouseover each wedge of the pie graph, an information box appears.
The box lists the dst value and event count. If you click on the wedge, Splunk takes you back to the search results and updates your search string to include the specific field name and value you selected from the chart. Try it out!

Report on fields

Return to the search window and search for all firewall deny events in sampledata.

index=sampledata deny

To report on fields:

1. Click on the Fields... menu.

2. From the list, check and apply src.

3. From the src filter menu, choose Report on this field >>.

Splunk takes you to the report window and updates your search string:

index=sampledata deny | top limit=100 src

Now, you can modify your report settings.

Build new reports

From the reports window, you can also enter a new search and build new reports.

1. Search for all "access_common" data in sampledata.

index=sampledata sourcetype=access_common

2. From the resulting list of Fields, select bytes.

3. Under Series, define your data series to "show the sum of bytes vs. time split by action":

You can define a custom time range for your chart. Here, it's zoomed in to a day of data.

Note: The chart updates as you define your series.

Pick different charts

Change chart styles by selecting a type from the display as drop-down menu above the current chart. Choose from the following chart types:

• column
• line
• area
• scatter
• stacked column
• stacked area
• pie
• donut
• bubble
• heatmap

See a sample of these charts in the report gallery on our website.

Add a report to your dashboard

You can save a report just as you would any other search. When you save the search, add it to your default dashboard by checking the box at the bottom of the save dialog.

You'll see the report on the dashboard after clicking the logo to return to the home page. Dashboard searches are refreshed every tenth of the time interval (for example, a 4 hour search every 24 minutes) or every hour, whichever is shorter.

You can read more about saving searches to the dashboard in Manage saved searches.

Note: You won't see your report on your dashboard if you haven't loaded any data to your main index yet. As soon as you have data in your main index, the "getting started" links are replaced with a default dashboard including modules that are predefined in the product, plus additional searches and reports you've added.

More searches

A Splunk search consists of one or more data-generating commands and their arguments, which can include literal keywords, wildcards, Boolean expressions, modifier name and value pairs, and subsearches. The generated data (search results) can then be used as inputs into other search commands in a search pipeline.

Splunk search commands are categorized by the type of operations they perform. You've already seen some examples of data generating commands. There are also commands that allow you to:

• Evaluate each result.
• Filter and re-order your results.
• Summarize large result sets.
• Add or remove fields in your results.
• Save data in different output formats.
• Perform administrative functions.

The following examples will demonstrate some of these capabilities. Refer to Search Commands for the complete list.

Report

Report commands, such as timechart, stats, top, and rare, summarize your results in the report window.

timechart

timechart returns statistics bucketed by time and is good for driving line charts. Try these examples.

Count of deny events graphed by time.

index=sampledata deny | timechart count(_raw)

Sum of bytes for GET requests:

index=sampledata sourcetype="access_common" GET | timechart sum(bytes)

Average bytes by method:

index=sampledata sourcetype=access_common | timechart avg(bytes) by method

stats

stats provides summary calculations by any field.

Total bytes sent by destination.

index=sampledata sourcetype=syslog | stats sum(sent) by dst

top

Let's get the top denied source IP addresses. Try it with a column graph.

index=sampledata netscreen deny | top src

rare

You can also get the 10 least common source IPs (by using rare).

index=sampledata netscreen deny | rare limit=10 src

Transform

Transform commands, such as transaction and diff , allow you manipulate the fields and values in your search results.

transaction

This search takes events from the access logs and creates a transaction from events that share the same clientip value that occurred within 5 minutes of each other (within a 3 hour time span).

index=sampledata sourcetype=access_combined | transaction fields=clientip maxpause=5m maxspan=3h

diff

Search for errors in syslog and diff the first and third results.

index=sampledata error sourcetype=syslog | diff pos1=1 pos2=3

Compare the host field of the last search.

index="sampledata" error sourcetype="syslog" | diff pos1=1 pos2=3 attribute="host"

Re-order

You can modify the order of your results based on different fields.

sort

Use the sort command to re-order the top 100 src field values of netscreen deny events.

index="sampledata" netscreen deny | top limit=100 src | sort src

Filter

You can define constraints to modify your search results.

set

Return all URLs that have 404 errors but no 303 errors (using set).

index=sampledata | set diff [search 404 | fields url] [search 303 | fields url]

regex

Use the regex command to filter results out of your search results. Specify a regular expression in regex to remove results that do not match.

Note: if you want to use the "or" ("|") command in a regex argument, the whole regular expression must be surrounded by quotes (ie. regex "<expression>").

The following example gets sendmail events that contain IP addresses in the non-routable class A (10.0.0.0/8).

index=sampledata sendmail | regex _raw=(?<!\d)10.\d{1,3}\.\d{1,3}\.\d{1,3}(?!\d)

Note: The regex command supports inclusion of PCREs (Perl Compatible Regular Expressions).

Evaluate

You can perform operations directly on your data while searching.

fields

Use the fields command to specify the particular fields you want to see in your results. Here we will display only the src and dst fields of the sampledata netscreen deny events.

index="sampledata" netscreen deny | fields src, dst

Add a comparison

Let's go back to our top source IP addresses and filter for ones with more than 5 denies by using a logical comparison in the search command.

index=sampledata netscreen deny AND (count>5) | top limit=100 src

Refer to the Search Syntax for more information on Comparison Operators.

Use subsearches

Now we're going to put it all together by doing another search to find which of the actions with more than 2 500 http status codes also had 200 successes.

index=sampledata 200 [search index=sampledata 500 AND (count>2) | top action | fields + action]

Add inputs

About inputs

Splunk can access and process any format of IT data from different sources on your filesystem. Data sources include files, FIFO queues, network ports, databases, and scripts. You can add most of these input types to your index using Splunk Web's Data Inputs page.

This topic discusses the different input types you can add to Splunk's index using Splunk Web. For information about using other methods to define inputs (such as using inputs.conf), refer to the Admin manual's topic on data inputs.

Files and directories

When adding a new file or directory to your data inputs, you can monitor a directory, upload a local file, or index a file on the Splunk server. Use monitor to add continuous and non-destructive inputs. Upload or index files to add one-time and destructive inputs.

Monitor

Splunk's monitor command is similar to the UNIX tail -f command for file monitoring. When you monitor a directory, Splunk detects subdirectories and recursively examines them for new files. As new files are added to the directory, Splunk detects the changes and indexes any new data.

When you configure inputs via Splunk Web, Splunk modifies your inputs.conf file in $SPLUNK_HOME/etc/system/local to include a stanza that defines your new input.

For example, If you monitor /var/log, Splunk adds the following stanza to your local inputs.conf:

[monitor:///var/log]
disabled = false
host = <hostname>

Also, you can view and edit the input properties of your monitored directory from Admin > Data Inputs: Files & Directories in Splunk Web.

Upload

Browse for a local file and add it directly to your inputs. If you have a previous version of the file as an input, uploading a new file overwrites the existing version. Unlike a monitored file, the uploaded file does not continuously update. Therefore, use the upload option for one-time and destructive inputs.

Uploading a local file does not modify inputs.conf. Instead, it copies the specified file into $SPLUNK_HOME/var/run/splunk/upload/ and then moves it into $SPLUNK_HOME/var/spool/splunk/ for indexing. After indexing, Splunk deletes the file; hence, the indexed file does not show up as a new data input in Admin > Data Inputs: Files & Directories.

Note: When you upload a local file, if necessary, Splunk uncompresses the file before processing it.

Index

Indexing a file on the Splunk server copies the file directly into /var/spool/splunk, where it exists while Splunk processes the data. Similar to uploading a local file, this operation does not modify inputs.conf. After indexing, Splunk deletes the file; hence, the indexed file does not show up as a new data input in Admin > Data Inputs: Files & Directories.

FIFO queues

Caution: FIFOs are not recommended for application servers forwarding data to Splunk in a distributed setting. Due to their vulnerability, Splunk does not recommend that you use FIFOs. Monitor is a more reliable, stable method. Support FIFO inputs is deprecated and will be removed in a future release of Splunk.

Splunk accesses the data in a FIFO, or named pipe, queue as though it were a file. When defining a FIFO input in Splunk Web, provide the path that directs Splunk to the queue. FIFO access is very fast, but FIFOs are vulnerable when there are processing disruptions because the in-memory data may be lost.

Network ports

Splunk supports UDP and TCP connections. When configuring network ports, keep in mind that you cannot use ports lower than 1024 if you are not running Splunk as root.

UDP

UDP is a best effort protocol; you might not get messages if the network is clogged or has a hiccup. You also can't be absolutely sure the messages aren't spoofed or altered in transit. UDP should be reserved for logging implementations focused on day-to-day troubleshooting rather than compliance or security.

Splunk with an Enterprise license can read directly from the network on any UDP port. Use this configuration to make Splunk act directly as a syslog server by reading remote syslog events on UDP port 514. You can also send any other UDP source of logging data.

Like all network streaming approaches, direct UDP input is higher performance than reading files from disk.
Check the Spunk Wiki for information about the best practices for using UDP when configuring Syslog input.

TCP

TCP is a reliable, high-performance choice for most situations, since this protocol includes checks to ensure that data has arrived safely and intact. Splunk with an Enterprise license can receive data on any TCP port, allowing Splunk to receive remote data from syslog-ng and other syslog implementations that use TCP for security or reliability. TCP is the foundation of Splunk's distributed data access.

Note: If the sending process buffers data such that events are broken into multiple pieces, Splunk may interpret the parts as multiple events. This is more likely if events are being generated intermittently, as there may be long pauses (several seconds or longer) between blocks of buffered data. If you notice truncated events, try forcing the process to send events atomically.

Use Data Inputs page

This topic discusses how to use Splunk Web's Admin > Data Inputs page to add new inputs and edit existing inputs. These inputs include files, directories, FIFO queues, and network ports.

For more information about the different inputs you can add to Splunk, read About inputs.

Access Data Inputs page

In Splunk Web, you can add and manage all your data inputs from the Admin page:

2. From the lefthand navigation list, click Data Inputs.

This takes you to the Admin > Data Inputs: All page which tells you how many inputs you have in each category: Files & Directories, FIFO Queue, Network Ports, and Crawls.

You can add new inputs directly from this page by clicking Add input in the "Actions" column. If you want to view and edit the actual inputs, click on the input category.

Run crawls

Use the Data Inputs: Crawls page to run:

• New crawls that search for input sources to add.
• Saved crawls that update existing inputs.

Refer to Use crawl for more information on this search feature.

Add files and directories

Use the Data Inputs: Files & Directories page to view and edit properties for monitored directories and uploaded files. Configure new inputs by clicking New Inputs. Change existing inputs by clicking on the input's path in the File or Directory column.

To add a new input:

1. Click New Input.

2. Under Data access, choose one of the following options:

• Monitor a directory: to index a directory and continuously update when changes are made to that directory.
• Upload a local file: to upload a file from your machine and index it.
• Index a file on the Splunk server: to copy a file from the server directly into the your index.

3. Specify a pathname to the file or directory. If you choose to Upload a local file, you can browse for the source.

4. Under Host, select the host type under Set host and supply the required host value. Your host options depend on the data access method you selected in Step 2.

If you chose Monitor a directory, the Set host options include:

• Constant value: requires a fully qualified domain name or IP address.
• Regex on path: requires a regular expression for the host path.
• Sement in path: requires a segment number.

If you chose Upload a local file or Index a file on the Splunk server, you can only set Set host to Constant value. This requires a fully qualified domain name or IP address.

Note: Refer to the Admin manual for more information about assigning host values to an input.

5. Under Source Type, set the source type to:

• Automatic: to let Splunk define the sourcetype value.
• From list: to select from a drop-down list of existing sourcetype values.
• Manual: to supply a custom sourcetype value in the field provided.

6. Click Submit to save your new input.

Note: Refer to the Admin manual for more information about setting the source type for an input.

Add FIFO queues

Caution: FIFOs are not recommended for application servers forwarding data to Splunk in a distributed setting. Due to their vulnerability, Splunk does not recommend that you use FIFOs. Monitor is a more reliable, stable method. Support FIFO inputs is deprecated and will be removed in a future release of Splunk.

Use the Data Inputs: FIFO Queues page to view and edit properties of each FIFO processed by Splunk. Configure new inputs by clicking New Inputs. Change existing inputs by clicking on the input's path in the list.

To add a new input:

1. Click New Input.

2. Under Source, type in the path to the FIFO.

3. Under Host, select the host type under Set host and supply the required host value.
Note: You only have one host type option, Constant value, which requires a Fully qualified domain name or IP address.

4. Under Source Type, set the source type to:

• From list: to select from a drop-down list of existing sourcetype values.
• Manual: to supply a custom sourcetype value in the field provided.

Note: If you chose From list, the default Source type is access_combined.

5. Click Submit to save your new input.

Add network ports

Use the Data Inputs: Network Ports page to view and edit properties for UDP or TCP ports watched by Splunk. Configure new inputs by clicking New Inputs. Change existing inputs by clicking on the input's path in the list.

To add a new input:

1. Click New Input.

2. Under Source, select a Protocol and supply a Port number:

• If you select UDP, the Port defaults to 514.
• If you select TCP, the Port defaults to 9998.

Note: The default protocol is TCP.

3. Specify whether or not you want this port to accept connections from all host (Yes) or restrict to one host (No).

If you chose No, supply the IP address of the Host in the field provided.

4. Under Source Type, set the source type to:

• From list: to select from a drop-down list of existing sourcetype values.
• Manual: to supply a custom sourcetype value in the field provided.

Note: If you chose From list, the default Source type is syslog.

5. Click Submit to save your new input.

Use crawl

crawl searches your filesystem for new data sources to add to your index. Configure one or more types of crawlers in crawl.conf to define the type of data sources to include in or exclude from your results. Save this crawl search and schedule it to run regularly to update your indexes.

This topic explains how to use the crawl command and how to save and schedule a crawl search. Refer to the Admin manual for instructions to configure crawl. You can also watch this Splunk developer video about crawl.

Note: Splunk currently supports one type of crawler, labeled file_crawler. As yet, you cannot define a custom crawler.

Run a crawl

In Splunk Web, you can access and run the crawl command from the Splunk search bar and the Admin > Data Inputs: Crawls page.

The Splunk search bar
You can run the crawl command directly from the search bar:

| crawl

For example, you can tell Splunk to crawl specific directories when you include the root argument:

| crawl root=/private/var/log;/private/var/db

The Admin page
You can manage all your saved crawls from the Admin > Data Inputs: Crawls page. From this page, you can also run the default crawl search by clicking New Crawl:

| crawl | search NOT *personal*

Results of a crawl

For each item listed in your crawl results, Splunk displays whether or not it is a file, a timestamp indicating when it was last modified, its size, and its status (whether it is added or not added to your inputs). You can perform two actions on each data source: Add input and Preview file/directory.

Preview file or directory

To review the contents of the data source before adding it as an input, click Preview file or Preview directory.

A new window opens:

• If you click Preview file on a file, Splunk returns events from the file.
• If you click Preview directory on a directory, Splunk displays a list of the files in the directory and lets you drill-down further and preview each file.

Add input

To add the selected data source as an input, click Add input.

Now, when you go to the Admin page and select the Data Inputs tab, your selected data source is listed.

Note: Adding data inputs with crawl modifies your inputs.conf file to include a stanza describing the new source. For example, if crawl discovers /var/log, clicking Add input adds the following stanza to inputs.conf:

[monitor:///var/log]
disabled = false
index = main
class = crawl
generator = ui

Save a crawl

After you run a crawl search, save the search by clicking the Save this Crawl... link located above your search results. This action opens the Admin > Data Inputs: Crawls: Create Crawl page which prompts you to:

• Name your crawl search.
• If necessary, edit your search.
• If desired, elect to run your crawl on a schedule.
• Click Cancel to return to the Admin > Data Inputs.
• Click Save to save your crawl search.

Note: Your crawl won't save, if you don't provide a name.

Manage saved crawls

Manage your saved crawl searches from the Admin > Data Inputs: Crawls page. You can run a new crawl or select one or more saved crawls to:

• Run Now and update your indexes.
• Enable or Disable so that you can start or stop updating particular indexes.
• Delete to remove the search from your list.

Edit the search and schedule properties of an individual crawl by clicking on its Name.

Note: You can't change the name of your saved crawl.

Schedule saved crawls

When scheduling your saved crawls, you can define the type of schedule and how frequently to run it. You can also set alert options and define fields to include in summary indexes. These options are exactly the same as options provided for saving regular (non-crawl) searches.

Index

About indexes and indexing

We use the term "index" to refer to:

• How Splunk manipulates and prepares raw event data for searching.
• The act of processing raw event data as Splunk prepares it for searching.
• The directory where Splunk stores all the event data.

Splunk indexes data in real time. It accesses data using a variety of input methods, applies universal processing techniques to handle different formats of IT data, and persists the original raw data along with indexes and additional fields added during processing.

Note: Refer to the About inputs page for more information about input types and methods.

Note: Read about using Splunk Web to Manage you indexes and Create new indexes.

Events, segments, and fields

Events are a single record of activity or instance of data -- for example, a single log entry. Fields are attribute and value pairs that make up segments of events. As part of indexing, events are broken into segments; Splunk uses breaking characters and rules to define how events are divided.

Usually, Splunk can detect event boundaries for different data formats. However, if event boundary recognition is not working as desired, you can customize your rules in props.conf. Refer to the Admin Manual for how to configure event boundaries.

The are two types of segments: major and minor. Major segments are words, phrases, or terms in the data that are surrounded by breaking characters such as white space and newline characters. Minor segments are breaks within a major segment. For example, the IP address 192.168.1.254 may be indexed as a major segment and then separated into the following minor segment: 192, 192.168, and 192.168.1.

Edit your segment recognition rules in segmenters.conf and apply them to different fields via props.conf. Refer to the Admin Manual for how to configure segmentation.

Search and indexes

Splunk stores all processed data in a collection of database directories, also called an index. Each database directory is located in $SPLUNK_DB and named db_<starttime>_<endtime>_<seq_num>. $SPLUNK_DB defaults to $SPLUNK_HOME/var/lib/splunk. The following is a list of Splunk's preconfigured indexes and a brief description of what they store:

• history: search history.
• main: all processed data. Unless otherwise specified, this is the default database.
• sampledata: sample event data used for training.
• splunklogger: Splunk internal logs.
• summary: summary indexing searches.
• _audit: events from the file system change monitor and auditing.
• _blocksignature: event block signatures.
• _internal: metrics from Splunk's processors.
• _thefishbucket: internal information on file processing.

You can create new indexes, edit index properties, remove unwanted indexes, or relocate existing indexes. You can manage (create, view, and edit) indexes from Splunk Web. For more information, refer to the User Manual's topic on managing and creating indexes. You can only remove and relocate existing indexes via the CLI. For more information, refer to the Admin Manual's topic on index management.

Unless specified, Splunk automatically searches through the default index, main. You can restrict your search to another index by specifying it in the search bar. For example, to search for HTTP requests that occurred only in sampledata:

index=sampledata http

Manage your indexes

In previous Splunk releases, you used the command line interface (CLI) to manage your indexes. Now, you can view your indexes, edit their properties, and add new indexes from the Admin page of Splunk Web.

This topic discusses using Splunk Web to view and edit your indexes. To define a custom index, refer to the Create new index topic.

View and manage indexes

In Splunk Web, you can view and edit all your indexes from the Admin page:

2. From the lefthand navigation list, click Indexes.

This takes you to the Admin > Indexess: View/Manage Indexes page which displays a table of all your indexes and their properties, including:.

• The home path, or directory.
• The current size in MB.
• The maximum size in MB.
• A count of events.
• A timestamp for the latest event.
• A timestamp for the earliest event.

Edit index properties

From the Admin > Index: View/Manage Indexes page, click an index name to view and edit that index's properties. Properties that you cannot change are grayed out and include:

• The index's name.
• The path to the fields and hot/warm databases.
• The path to the cold databases.
• The path to the thawed databases.

Properties that you can redefine include:

• The maximum size (in MB) of the hot database.
• The maximum size (in MB) of an index.

After you make your changes, click Update. Then, restart Splunk to apply your changes.

Important: To apply any changes that you make to the indexes, such as editing properties or adding a new index, you must restart Splunk. In Splunk Web, you can restart the Splunk server from Admin > Server: Control Server. Just click Restart Now.

Create new index

Splunk ships with an index called main that, by default, holds all your events. Splunk with an Enterprise license lets you add an unlimited number of additional indexes. One of them serves as the default index for any input and search command that don't specify any index. You can add indexes via Splunk Web, Splunk's CLI, or indexes.conf.

via Splunk Web

1. The Admin > Indexes: Create Index page lets you define the properties for a new index. To create a new index, enter:

• A name for the index.
• The maximum size (in MB) of the hot database.
• The maximum size (in MB) of the index.

2. If you check Advanced settings, the list of properties expands. Advanced properties include:

• The maximum number of search results.
• The maximum number of warm database directories.
• The maximum number of cold databases open at any given time.
• The frequency that new hot database are to be created.
• The frequency that cold databases are to be frozen.
• The script and directory to archive the index's data.
• The number of concurrently running optimize processes.
• Whether to wait for optimize processes to finish or just kill them.
• The number of extra threads to use during indexing.
• The amount of memory (in MB) to allocate for indexing.
• The number of events to make up a block for block signatures.

3. After setting the index's properties, click Add.

4. Restart Splunk to save and apply your changes.

Important: To apply any changes that you make to the indexes, such as editing properties or adding a new index, you must restart Splunk. In Splunk Web, you can restart the Splunk server from Admin > Server: Control Server. Just click Restart Now.

Note: You can also use Splunk Web's Admin > Indexes page to edit an index at any time; just click on the index name to access the index's properties. Properties that you cannot change in Splunk Web are grayed out. For more information about configuring index properties, read indexes.conf.

via the CLI

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command. You can also add Splunk to your path and use the splunk command.

To add an index, first shutdown Splunk with splunk stop. Then navigate to Splunk's CLI. Then type:

# ./splunk add index [name] [directory (optional)]

Note: Do not use capital letters in your index name; this is a known problem that will be fixed.

The optional directory argument lets you set up an index outside of the default $SPLUNK_DB location.

The add index command brings you to a dialog session. Specify the configurations of your new index:

./splunk add index hatch

Hit enter to accept the default values in parenthesis, or enter your own values.

Delete an index

Use the CLI to delete an index from your Splunk instance:

# ./splunk remove index [name]

Search

Splunk search

Searching in Splunk is easy - type any term you'd expect to find in your data into the search box and hit Enter. A Splunk search lets you search indexed data in real-time, extract data from search results, and produce meaningful reports from the data you put into Splunk. For example, to search for all events containing a given IP address, type it into the box. Try it with usernames, error codes, transaction IDs, or whatever else you are looking for.

See the Search syntax page to learn about Splunk search syntax.

A search is pipeline of commands (similar to a Unix "|" pipeline) that starts with a command that gathers data (typically a search on data in a Splunk index), followed by data-processing commands that operate on the data to yield search results.

See the Search pipeline syntax page for details about the syntax of the search pipeline.

You can also watch this Splunk developer video about searching with Splunk.

Generate search results

Generate search results by using a data-generating command. Generate search results by using:

• search to get new search results from a Splunk index.
• savedsearch to execute a saved search.
• file to import previously gathered search results directly from a file.
• crawl to search your filesystem for new data sources to add to your index (returns them as search results).

Construct searches

Use the search command to construct simple keyword searches on data in your Splunk index (just like a Google search). Narrow your keyword searches with modifiers, fields, Boolean operators, and logical comparison operators.

You can also construct more powerful searches by using additional commands to extract data, perform statistical operations, and build reports. Learn about the search commands in the search command reference.

As you construct your search, Splunk's typeahead functionality will prompt you with predictive text based on the commands you are using and the contents of your data.
Note: Wildcards (*) are not supported. You can simply type and wait for the predictive text to complete your term.

Here are some valuable points to remember when constructing a search:

When generating data

To get more results:

• Increase the time range (search over all time to get the most number of results).
• Use wildcards and partial keywords instead of exact keywords.

If you want a faster search:

• Narrow the time range of your search.
• Reduce the number of fields that are extracted by un-selecting fields in the Fields picker menu.
• Reduce the segmentation by selecting inner, outer, or raw in the Preferences menu (reducing segmentation makes extracting and reporting more difficult).

When narrowing your search

• Filter results by event type if possible.
• Use the source and sourcetype fields to narrow a search to only a specified source.
• Use a combination of logical and Boolean expressions of keywords, modifiers, and fields.

Types of search

Form searches

Form searches are reusable searches that are pre-defined by a Splunk administrator. Form searches allow you to run complex searches by simply inputting variables in form fields. Learn more about Form searches.

Macro searches

Macro searches allow macro substitution of variables in saved searches. This allows you to run a complex search repeatedly with different variables. Learn more about Macro searches.

Transaction searches

Transactions let you search for groups of related events that are pre-defined as a transaction by your Splunk administrator. Use the transaction command to execute a transaction search. You can override specifications of a pre-defined transaction, or define a new transaction with the transaction command. Learn more about Transactions.

Live tail

Live tail allows you to see data as its being indexed into Splunk in real-time (similar to Unix's tail -f command). Live tail allows you to execute a simple search in its stand-alone window, and monitor events that match the search. Find out more about live tail.

Asynchronous searches

The Splunk CLI allows you to run multiple searches asynchronously. Use this if you have a search or report you want to run on a large amount of data where the search could take days and you still want to be able to run other searches with Splunk. Use the dispatch CLI command to execute asynchronous searches. Learn more about asynchronous searches.

CLI searches

Run searches in the CLI with the search CLI command. Searches in the CLI work the same way as searches in Splunk Web except there is no timeline rendered with the search results, and a time range isn't specified by default. Search for anything by including your search as the 'search string' argument of a CLI search command.

Learn more about CLI searching.

Save and schedule searches

After you save a search, you can set your searches to run regularly and schedule alert conditions. Read more about Save, schedule, and alert options.

Tune search performance

Splunk's searches are optimized for text-based searching of raw event data. Search speed is dependent on how your Splunk install is configured.

If you are searching for a term that occurs frequently in your data (or just for for *), you should expect results in 1-5 seconds. If Splunk doesn't return a full timeline, with all events rendered within 15-20 seconds, your index(es) or system configuration might be unhealthy. A good starting point is to read Understanding Buckets on the Splunk Wiki. If you're experiencing poor search performance and are unsure why, contact Splunk support.

You can improve the speed of your searches by editing configuration files, and by downloading various applications from SplunkBase. Read more about tuning search performance.

Form search

A form search is a saved search that has form fields that you must fill in before you run a search. Save any complicated search, and make it reusable as a form search (learn how to create form searches).

Form searches are saved searches that appear as forms when run. Save any search with form fields that a user running the search must fill out with parameters to run the search. You can create a sophisticated saved search and save it as a form with as many form fields as you like.

For example, you can define a search that returns all Web server errors for any username to be specified at search time:

503 OR 500 OR 404 sourcetype=access_common $user$

When run, this search appears as a form labeled user.

The search 503 OR 500 OR 404 sourcetype=access_common is still part of the search, but does not appear to the user.

Note: Form search works via text substitution, so the form fields can consist of anything, not just an indexed or an extracted field.

Run a form search

Form searches are saved searches. Run a form search by selecting it from the "Saved searches" menu in the search bar drop-down in Splunk Web.

If the saved search you select is a form search, then you'll be prompted with a form dialog like this:

Fill out the values in the form.

Note: You can substitute any text (not just a field) in a free-form text box in the form.

Refer to the Admin guide section on form searches to learn how to create form searches.

Transactions

Transactions enable you to search, and report on transactions in your data indexed in Splunk. A transaction is a grouping of events that contain related pieces of information. Transaction search is useful for a single observation of any physical event stretching over multiple logged events.

Here is an example of what can make up a transaction:

• An event in a Web access log
• An event in an application server log
• An event in an asynchronous fulfillment application that generates a message queue that subsequently identifies a business transaction

In this case, the Web access log might share a session ID with the event in the application server log; the application server log might contain the account ID, transaction ID, and product ID; the transaction ID may live in the message queue with a message ID, and the fulfillment application may l