Splunk User Manual (Splunk v2.0)
Getting Started
Login
First, make sure that Splunk is installed on a server somewhere, and that you have the correct hostname (or IP address) and port number for the server's splunkSearch Web interface. The default is port 8000.
Next, make sure you have a supported browser—Internet Explorer 6.0 or higher, Firefox or Mozilla. Use it to open port 8000 (or whichever port is assigned to Splunk) on your Splunk host.
http://splunkhost.mydomain.com:8000
If you have Splunk Professional you'll be prompted for a username and password by your browser. Enter the username and password your system administrator assigned to you. If you are the system administrator, try the default account below.
username: admin password: changeme
Splunk Server or Splunk Professional?
Check which version of Splunk you have, so you'll know which features in this guide are available.
The logo in the upper left corner of your Splunk server's home page will either say "Splunk Server" or "Splunk Professional."
Splunk Professional has six major features beyond the free Splunk Server, detailed in the Splunk Server vs Splunk Professional section of the Splunk Installation Manual.
Got Data?
If you haven't loaded data yet, follow the instructions on your Splunk server's home page.
Click the Add Data button in the upper right of the interface to see all input options. An easy way to load data is to use the Tail File input on your Splunk server host's main log directory, /var/log or whichever matches your host operating system.Once you have data indexed, try the 10-minute tutorial that begins on the next page.
Tutorial
Start Here
The following tutorial takes about ten minutes, and will familiarize you with Splunk's interface and search features.
Splunk Assistant
At the lower right of the interface is a button labeled Splunk Assistant. It pops up a window with context-specific advice and help links. You can leave Splunk Assistant open as you Splunk, or pop it up for context-specific suggestions. To close it, click the button again. Splunk Assistant doesn't behave like Clippy. It won't pop itself open and try to help when you didn't ask it to.
Splunk Assistant can tell that you have no results for a Splunk, or that you just opened the Events by Time histogram, or that you finally tried Ctrl-Alt-click (that's cmd-option-click on a Mac.) Splunk Assistant displays suggested next steps, and links to relevant online documentation.
Cancel or Reset Splunk
To cancel a running Splunk at any time, hit the Esc key on your keyboard.
To reset the interface controls (and cancel any running search), click the Splunk logo in the upper left corner of the page.
The Splunk Box
At the top of the Splunk interface is a box that looks like any Web search engine. We call it the Splunk box.
Use Splunk the same way you do Web search engines. Don't try to devise the perfect command or query the first time. Instead, start with your best instant guess, then refine your search.
Below is a simple example.
Below is an overly complicated example that shows off Splunk's search syntax.
Syntax
See the reference sections on Splunk box syntax and Splunk modifiers for complete lists of the supported syntax, operators and modifiers you can use to Splunk constructively.
Cheat Sheet
Click on "Cheat Sheet" in the upper right corner of Splunk's interface to pop up a one-page guide to Splunk's command syntax.
Typeahead
Type into the Splunk box and you'll see a drop-down list of typeahead options with the number of matches for each possible completion in the current index. It includes both terms in the index and Splunk modifiers such as source::. You can use the up and down arrows, Tab key, and Enter key to select entries from the typeahead panel.
Search Results—Everything's Clickable
Look at the components of each search result. The event itself is at the top. It can be a terse log file entry, or multi-line application stack trace. For each result, Splunk displays its event type, source type, host and source, as well as a timestamp. Splunk also adds links for Similar and Related, plus options to Show Source and Look up Event. All of these values and links are clickable. The downward-pointing arrows next to some values are buttons that activate popup menus.
Click combinations
Search for term: click
Restrict the current search further by the clicked term: Ctrl-click (On Macs, cmd-click)
Remove term from current search: Ctrl-click it again (On Macs, cmd-click)
Search for negative term (e.g. NOT apache): Alt-click (On Macs, option-click)
Add negative term (e.g. NOT apache) to search: Ctrl-alt-click (On Macs, cmd-option-click)
The fastest way to find obscure events is to start with a simple, broad search and then remove terms that don't match using Ctrl-Alt-click. (On Macs, cmd-option-click.) We call this "removing the noise."
Results Tabs
Splunk's results page has six tabs that present your search results grouped by Events, Event Types, Tags, Source Types, Hosts and Sources. Click each tab to see your results summarized differently.
Timerange Selector
Click the clock icon in the Splunk box—or click the timestamp of any event in your results—to pop open an interactive widget for setting start and end times for your search. You can use your keyboard, arrow keys or your mouse to adjust time values. Click the calendar icon to pop up a calendar widget to select dates.
Event Types
Event types make it easy to sift through a centralized log index. Splunk automatically assigns an event type value to each event it indexes. This lets you search and sort all events of the same type without needing to pattern-match them all manually.
Event types come in two forms:
- Local event types have a numeric value preceeded by ? to make it clear they have not been looked up at Splunk Base, the global repository of event types created by Splunk users.
- Global event types have an alphanumeric value like SP-CAAAE8K that shows they've been looked up and matched at Splunk Base. Every Splunk user around the world who encounters an event of this type will be able to refer to it by this label unambiguously when discussing it or sharing data samples with other Splunkers.
Example
SP-CAAADUP is a typical Splunk global event type. It happens to be an Apache error message.
A Splunk administrator who loads Apache error events might see them in Splunk results containing local event types—?12,?13,?14—until he looked each event up at Splunk Base. He could then download global assignments from Splunk Base. That would change local event types like ?14 to global event type names like SP-CAAAD7Y that match for all Splunk users everywhere.
Splunk Professional Users
Splunk Professional users without Power User or Admin status can view and search event types and look them up at Splunk Base. They can't download global event types to update the local Splunk index, though.
Tags
Tags are how Splunkers around the world share their knowledge about IT events.How Tags Work
It's just like tagging photos on Flickr.
- A single event type can have an unlimited number of tags.
- A single tag can be applied to an unlimited number of event types.
For example, there are nearly 400 event types at Splunk Base tagged Websphere, and 100-plus tagged Apache.
Example
Let's look at event type SP-CAAADUP again. It's been tagged with at least three different tags—Apache, listener, and network. Clicking each of those tags in Splunk Base shows a different slice of event types that match each tag, so you can see where the event type fits into the grand scheme of IT data.
Splunk Professional Users
Splunk Professional users without Power User or Admin status can view and search tags and look them up at \ Splunk Base. They can't download tags to update the local Splunk index, though.
Similar
Similar events are those whose event types have nearly identical structures. The examples below are two events that have different event types, but you can seethe obvious similarity between them.
Splunking for Similar events lets you find events that don't quite match the search terms you typed, but are what you're looking for. Or, you can exclude similar events to block not just one unwanted type of event, but anything near it in structure.
Related
Related events have matching rare values between them—not a timestamp, but a rare value such as an IP address, username, or status level. The examples below are two events that have different event types, but you can see that they're related by the IP address.
Splunking for Related events lets you find events that you might not have thought to look for if you needed to specify the shared values explicitly.
Events by Time
The Events by Time chart is a histogram plot of the events in your Splunk results, grouped into equal-size time slices according to their timestamps. Events by Time is interactive. You can zoom in and out by clicking the magnifying glass icons. You can click on one time bar to narrow your Splunk to that time range, and Shift-click on another to broaden the time range to the period between the first and second bars.
Splunk Base
Splunk Base is the Web hub where Splunkers share their knowledge. It's a user-editable repository of global event types, tags, and wiki entries for each. If you're not sure what an entry in your Splunk results is about, click Look up Event to see what info Splunk Base has about it.
Example entries
- Event type: SP-CAAADUP
- Tag: Apache
- Source type: apache_error
If you don't like what you see, edit it!
Saved Splunks
Both the free Splunk Server and Splunk Professional allow you to bookmark a specific set of search parameters as a Saved Splunk. When you select a Saved Splunk from the Saved Splunks menu, Splunk runs the same search again on the currently indexed set of events.
Live Splunks (Splunk Professional)
Splunk Professional lets you schedule Live Splunks. These are Saved Splunks that the server runs automatically at regular intervals. If a Live Splunk's results meet thresholds you've configured, such as a certain number of results or a change in that number, Splunk Professional will notify one or more users via email, RSS, or a shell script.
Splunk Professional also displays status and statistics for all Live Splunks on its home page.
Report Splunks
Splunk's report:: can run SQL select statements on your Splunk results to create tabular reports. You can then export the results as a text file or in comma-separated-values (CSV) format. Using report:: in a Live Splunk is a good way to set up management reports that arrive via email.
Demo Server
We've set up demo.splunk.com with a gigabyte of anonymized log data you can play with to learn Splunk.
Splunk Search Interface
When in Doubt, Click It
Nearly everything in Splunk's interface is clickable, especially inside your search results. Try clicking around the home and results pages to see where it takes you. Don't worry, there's no way to accidentally delete, modify or corrupt your data from the interface.
Clicking on Splunk Terms
You can perform these click actions on any part of an event in your search results— segments, meta data such as sourcetype::syslog, and links such as Similar.
Search for term: click
Restrict the current search further by the clicked term: Ctrl-click (On Macs, cmd-click)
Remove term from current search: Ctrl-click it again (On Macs, cmd-click)
Search for negative term (e.g. NOT apache): Alt-click (On Macs, option-click)
Add negative term (e.g. NOT apache) to search: Ctrl-alt-click (On Macs, cmd-option-click)
The fastest way to find obscure events is to start with a simple, broad search and then remove terms that don't match using Ctrl-Alt-click. (On Macs, cmd-option-click.) We call this "removing the noise."
Segment Selection
Roll your cursor over the different parts of an event in your search results. You'll see individual segments—character strings treated as single entities in the index—highlight as you pass over them. Matching segments in other events will also highlight. If you click a segment, it will submit a new search. See the section on Clicking for details.
You can change Splunk's handling of segment selection with the menu option Preferences -> Segment Selection above the Splunk box. There are five settings, described below.
Full
Splunk's default configuration treats segments separated by periods and other punctuation as minor segments and those separated by spaces as major segments. If you search for a term that appears as a minor segment, it will be highlighted on your results page. But when you roll over it to click it, the entire major segment it belongs to will highlight.
One example is worth a thousand words: Search for com and then roll your mouse over any Web domain names that appear in your results. See how you can add or remove whole domains from your search with one click. It's faster than typing into the box again and again, yet you can still do so whenever you prefer to.
To select multiple consecutive segments in an event, such as the hour and minute in a timestamp (17:30:01) or the subnet section of an IP address (18.7.1.151), place your mouse at the leftmost segment and mouse over the subsequent segments to the right. Each segement will highlight in yellow as you pass over it. To select the entire major segment, i.e. the entire address or timestamp, place your mouse at the rightmost end instead.
Outer
This setting forces Splunk to always highlight the longest possible segment, such as a complete email address. It's equivalent to mousing from the rightmost end in Full mode.
Inner
This setting forces Splunk to always highlight the shortest possible segment, such as .com in an email address. It's equivalent to mousing from the leftmost end in Full mode.
Raw
In this mode, Splunk does no segment selection. Clicking on an IP address will do nothing.
Full with Pyramids
Same as Full, but Splunk will draw grouping boxes around segments. The result looks like a topological map, with segments stacked in pyramid-like formations to show how they are grouped.
Event Meta Data
Splunk populates several meta data values for each event. You can click on them to add or remove them from your search just as you do with other parts of the event.
Event Type and Tags
An event's type sorts it into a bucket of events with the exact same format. For example, all J2EE exceptions of the same severity level that come from a specific application and have the same stack trace will be assigned the same event type.
Splunk users can tag event types. A tag can be applied to multiple event types, and an event type can have multiple tags. Splunk users can upload or download tags for specific event types at Splunk Base to share community knowledge about specific types of events in their results. Tags let Splunk users and admins group event types into ad-hoc collections that can be searched or filtered together.
Splunk Professional Users
Splunk Professional users without Power User or Admin status can view and search event types and tags, and look them up at Splunk Base. They can't download global event types or tags to update the local Splunk index, though.
Timestamp
Splunk creates a timestamp out of every single event, either by extracting it from the event or by assigning a value. Timestamps are adjusted to the same time zone and displayed in a standard format.
Click on a timestamp to see only those results whose timestamps are within the minute after it. This is a great way to troubleshoot. You can broaden your Splunk using the timerange selector. To plot events by their timestamps, click Show Events by Time at the upper right of the results page.
Source Type
Splunk determines a type not only for each event, but for each source—that is, each file or stream from which it loads events. It can automatically recognize about forty different source types including linux_syslog_message and weblogic_stderr. If the source type ends in -u1, -u2 or a higher value, it means Splunk considers it "unknown." This usually means the source contains some idiosyncratic event format Splunk doesn't recognize by name, although it can tell one unknown type from another.
You can rename unknown source types with more meaningful names like snort_native. This gives you another effective value on which to search. See the section on Renaming Source Types for instructions.
You can also train Splunk to recognize your local data by sourcetype. See the section on Training Source Types in the Splunk Admin Manual.
Splunk Professional Users
Splunk Professional users without Power User or Admin status cannot edit source type names.
Host
The host value of an event is the name of the originating host that spawned the event, as opposed to the central reporting host whose syslog file stored it.
Source
The name of the file or stream from which the data was indexed. As an administrator you'll know when to sort by sources, or when to search for or exclude a specific source from your results.
Event Links
Alongside the meta data in each event are two clickable links that perform event-specific actions.
Similar
There may be no matching values in two events yet they're obviously somewhat alike. Similar events are those whose signature data patterns resemble one another, such as different severity level warning messages from the same J2EE application server.
The similar link is useful for finding events whose values and event types you can't guess exactly but to which you can see close approximations in your index.
Related
Other events may have totally different formats yet share matching values, such as an IP address that appears in different parts of two totally dissimilar events. Splunk calls these related events.
Instead of searching again for every single value in a suspicious event, just hit the related link.
Show Source
Click this link to see the full text of an event in its original context, e.g. the syslog message stream from which it was indexed. The source pops up in a second browser window.
Look up Event
Not sure what a particular event is? You can look it up at Splunk Base, where a community of IT professionals have contributed knowledge about individual event types—what application or device created the event, what causes it, and how to fix it if it's broken. You can contribute your own knowledge to let other Splunkers around the world benefit from your experience. See the section on Look up Event for instructions.
Tabbed Results
The results page also includes tabs for each of the built-in meta data values. The tab labels display how many results each tabbed panel has for the current search.
Click on a tab to bring it to the front. You will see a plot of the individual results for that value, plus helpful links on some tabs such as a Similar link for event types.
Tabbed results. Here the Event Types tab displays 17 event types for the current search. Note the Similar link at right.
If you see fewer Event Types, Tags, Source Types, Hosts or Sources than you know are in your Splunk index, try adding a maxresults:: value to your search greater than the default value of 10,000, e.g. maxresults::200000. The tabbed results only display those event types, hosts, etc. found in the first N results, where N is the value of maxresults.
Set Timerange
Click the clock icon in the Splunk box—or click the timestamp of any event in your results—to pop open an interactive widget for setting start and end times for your search. You can see every event on your network that was logged just before or just after a trouble indicator.
Splunk can be configured to compensate for differing time zones and Daylight Saving Time between servers. See the Properties section of the Splunk Data Access Manual.
Show Events by Time
Click this link to display an interactive histogram plot of your splunk results.
Zoom In/Out
Click the magnifying glass-shaped + and - icons to zoom in or out from days to hours to minutes.
Select Range
Click on one bar in the histogram to narrow your search to that time range, or use Shift-click to select a range of several time buckets. The search results on the page will be filtered to match. This lets you quickly spot periods of unusual activity using the chart and then zoom in on the log data from that period.
Hoverplots
Mouse over individual meta data values—eventtype::, host::, etc.—on any of the tabbed results panels—Events, Event Types, Sources, etc.—to plot a different color histogram for only the events matching that value against the plot of all events. This is a good quick way to spot trends on your network.
Show Source
Click this link to see the full content of an event. Your browser will pop up a new window (be sure to disable popup blocking first) with the original raw data for the event. You can page through previous events from the same source.
The popup window has forward/backward arrows so you can step through the original source stream, e.g. a series of maillog events loaded from the same file. It also has an Export button that lets you save the event text to a file.
The Splunk Box
The Splunk box lets you type anything you can click, plus several other modifiers.
Splunk Box Commands
Any search you can create by clicking Splunk's graphical interface, you can also type into the Splunk box as a text-only search string. You've probably already noticed the box updates itself to match your point-and-click searches.
Nested Boolean logic, keyword operators, detailed parameter values—the Splunk box lets you use your command line skills to concoct advanced search strings. Here are a few quick examples.
Search terms are not case sensitive, except for the Boolean logic operators AND, OR and NOT. Splunk considers Foobar, foobar and fOoBaR identical.
Basic searches
You can type in any alphanumeric keyword. Watch the typeahead drop-down for hints as you type.
apache 404
To specify a Splunk meta data value such as eventtype, put two colons (::) between the meta data name and its value—metadata::value.
eventtype::imapd_login_failed
The same syntax applies to Splunk modifiers, for example daysago::3.
Search terms are additive. There is an implicit AND between any terms not separated by OR or NOT.
sourcetype::linux_syslog_message authentication failure
Wildcards
You can put wildcard * characters at the start or end of individual terms. We don't support regular expressions ... at least not yet.
apache 40*
month *ember 2005
You can wildcard the middle of a term by breaking it in two.
supercal* *alodocious
Quoted strings
Splunk does not yet support quoted strings such as "foo bar" yet. This search will return no results.
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; de) Opera 8.01"
This search will return lots of results.
Mozilla/4.0 compatible MSIE 6.0 Windows NT 5.1 de Opera 8.01
Boolean phrases
Boolean parentheses can be nested without limit. You need to put spaces between parentheses and other search terms—( foo AND bar ( NOT host::baz ) rather than (foo AND bar(NOT host::baz)).
host::trafficcop1 ( denied OR forbidden ) daysago::1
get NOT ( ( 404 or 20* ) AND host::testserver )
sourcetype::apache get /store NOT googlebot NOT ( 20* OR 404 )
source::/logs/08_19_05/voip/* OR ( sourcetype::asterisk_event_syslog NOT sip/4211office-testlab )
Show All Events
There are two ways to see every event in your index.
Splunk Box
Type this into the Splunk Box to search for every event in the default index.
meta::all
Saved Splunk
Select the menu option Splunks -> Saved Splunks -> all to search for every event in the default index.
Event Types and Tags
Event Types Explained
Splunk's automatic event typing lets you perform searches that would be impossible with grep, SQL, or shell scripts. By categorizing every event on your network Splunk lets you find or exclude the same type of results from hundreds of servers with one click.
An event type is a unique pattern seen in punctuation, alphanumeric patters, and certain keywords that make up an event. For example, Splunk would assign these events the same event type.
Apr 5 17:59:27 login user=joe Apr 5 17:58:33 login user=sue
Whereas Splunk would assign these three events three different event types.
Apr 5 17:59:27 login user=joe Apr 5 18:08:45 logout user=joe 17:59:27 05 Apr 2007 login user=joe
In short, Splunk attempts to replicate the mental categorization of events most administrators already perform in their heads.
Splunk doesn't try to assign meaningful event type values such as "imapd_login" because event patterns vary from one customer to another and change from day to day. Instead, it lets users look up event types in Splunk Base and sync tag assignments they deem correct.
Tag an Event Type
Users can assign one or more tags to every event type. When you set a tag it's assigned across every index on the same server.
To tag an event type, click the menu button next to the eventtype:: value in an event.
Splunk Professional Users
Splunk Professional users without Power User or Admin status can view and search tags and look them up at Splunk Base. They can't download tags to update the local Splunk index, though.
Tags can have any value composed of letters and numbers, plus the characters _ and -. We suggest short, separate words like netscreen syn_flood, netscreen configuration and snort syn_flood. This lets you create on-the fly groups by searching for (or excluding) eventtype::netscreen or eventtype::syn_flood. That's why tagging an event type as login failure is better than login_failure.
Click Save to assign your tag(s) to the event type.
Look up Event
You can see what tags other Splunk users have assigned to an event type by clicking the Look up Event link. You can upload your own tags for sharing, and discuss the circumstances and meaning of an event on Splunk's wiki.
Look up an Event Type at Splunk Base
Click the Look up Event link. A dialog box appears.
Anonymize Your Data
If you click Anonymize First any potentially identifying series of characters in the event will be replaced with semi-random text that doesn't change its event type pattern. Check your data carefully for unanonymized fields that shouldn't be submitted, as well as important segments (e.g. a severity level) that may have been over-zealously scrambled. Use the boxes labeled Restore these terms: and Anonymize these additional items: to refine the anonymization of the event. Then click Re-Anonymize to apply your changes.
Submit an Event to Splunk Base
If you've registered and logged in at www.splunk.com you'll see a page with information on the event type you submitted.
Download these tags
To apply tags from Splunk Base to your server, click the Download these tags button on the Splunk Base page.
Splunk Professional Users
Splunk Professional users without Power User or Admin status can view and search tags and look them up at Splunk Base. They can't download tags to update the local Splunk index, though.
Similar
Splunk defines similar events as those whose patterns are different, but not too different. Maybe one has an extra segment in it, or only two out of a dozen segments aren't of the same pattern. Different levels of log events from the same application are usually similar. They may have no segment values in common yet they're clearly worth looking at together if you're trying to find the source of a problem.
eventtype::?17-3
This means "events that are within 3 degrees of separation from event type 17." You can type this syntax into the Splunk box yourself, using values from 1 to 9.
Related
Splunk defines related events as those with matching rare values, such as an IP address or username. Events generated by the same user on different applications, or different applications on the same host, or events with matching error status labels are usually related.
Sources, Source Types and Hosts
Host
The host value of an event is the hostname of the server that created the file or stream from which Splunk loaded the event.
Source
A source is a file, stream, or other data input from which Splunk indexes events. The value of source is usually some combination of pathname, filename, and extension such as /archive/server1/var/log/ or /var/log/messages. Files uploaded through Splunk's browser interface get the pathname of the directory monitor's sinkhole directory, so they'll look something like /opt/splunk/var/spool/splunk/file.ext.
Source Type
The type of data in the source, as determined by Splunk or configured by an administrator. Typical examples are apache_combined_wcookie, linux_syslog_messages and mysqld_table_status. If no sourcetype is configured for a source Splunk will try to deduce it. If there's not enough data from the source for Splunk to deduce its type it sets the type to too_small.
Rename a Source Type
If you recognize a specific source type that Splunk has labeled as UNKNOWN—or even if you want to mark one as mysterious—click the menu button next to the sourcetype:: value. A dialog box labeled Rename this SourceType: will appear. Edit the sourcetype to be a string of alphanumeric characters (you can also use _ and -, but no spaces) and click Save to update your index. You can change it again later.
Splunk Professional Users
Splunk Professional users without Power User or Admin status cannot edit source type names.
Training Splunk
Source Types
Splunk can be trained to recognize the source types of your local data. You show it sample files and tell it what source type they are.
# splunk classify [filename] [sourcetype]
To see what source type Splunk thinks a data sample is, just omit the final parameter.
# splunk classify [filename]
Example
Splunk normally recognizes Linux syslog files, but what if it doesn't? In this example, we'll train it.
Splunk should recognize our syslog files, but an unusual event type or local customization confuses it. It decides the file type is unknown and assigns it messages-U8, which we can then find in unknown-props.xml. It gives us the cause too_small, saying it doesn't have enough events in the sample file to make a positive ID.
# splunk classify /var/log/messages.4
Using logging configuration at /opt/paul20/splunk/etc/log-cmdline.cfg.
WARN PropertiesMapConfig - File classifier settings file (/opt/paul20/splunk/etc/overlay-props.xml) cannot be read.
PROPERTIES OF /var/log/messages.4
Attr:AUTO_LINEMERGE True
Attr:AUTO_TAG False
Attr:BREAK_BEFORE_DATE True
Attr:BREAK_ONLY_BEFORE_DATE False
Attr:CLEANER_CONFIG /etc/myinstall/pluginConfs/cleaners.xml
Attr:DATETIME_CONFIG /etc/datetime.xml
Attr:MAX_EVENTS 256
Attr:MAX_TIMESTAMP_LOOKAHEAD 150
Attr:MUST_BREAK_AFTER
Attr:MUST_BREAK_BEFORE $\s*^
Attr:MUST_NOT_BREAK_AFTER
Attr:MUST_NOT_BREAK_BEFORE
Attr:SHOULD_LINEMERGE False
Attr:TYPING_CONFIG /etc/event-types/current/default.xml
Attr:file_type messages-U8
Attr:invalid_cause too_small
Attr:is_valid True
Attr:maxDist 300
To resolve the issue, we tell Splunk that this is definitely a linux_messages_syslog file.
# splunk classify /var/log/messages.4 linux_messages_syslog Using logging configuration at /opt/paul20/splunk/etc/log-cmdline.cfg. WARN PropertiesMapConfig - File classifier settings file (/opt/paul20/splunk/etc/overlay-props.xml) cannot be read. Training on /var/log/messages.4 of type linux_messages_syslog
From now on Splunk will know what our local syslog files look like.
# splunk classify /var/log/messages.4
Using logging configuration at /opt/paul20/splunk/etc/log-cmdline.cfg.
WARN PropertiesMapConfig - File classifier settings file (/opt/paul20/splunk/etc/overlay-props.xml) cannot be read.
PROPERTIES OF /var/log/messages.4
Attr:AUTO_LINEMERGE True
Attr:AUTO_TAG False
Attr:BREAK_BEFORE_DATE True
Attr:BREAK_ONLY_BEFORE_DATE False
Attr:CLEANER_CONFIG /etc/myinstall/pluginConfs/cleaners.xml
Attr:DATETIME_CONFIG /etc/datetime.xml
Attr:MAX_EVENTS 256
Attr:MAX_TIMESTAMP_LOOKAHEAD 150
Attr:MUST_BREAK_AFTER
Attr:MUST_BREAK_BEFORE $\s*^
Attr:MUST_NOT_BREAK_AFTER
Attr:MUST_NOT_BREAK_BEFORE
Attr:REGEXES syslog-host
Attr:SHOULD_LINEMERGE False
Attr:TYPING_CONFIG /etc/event-types/current/syslog.xml
Attr:file_type linux_messages_syslog
Attr:is_valid True
Attr:maxDist 100
Timestamps
If Splunk does not recognize the timestamps in a specific source, you can train it on a sample file of those events. Type this command.
# splunk learn dates
Splunk will prompt you for a file to use, and then prompt you to identify which fields in the file's events are timetamps.
A Sample Session
In the session below, we'll train Splunk to recognize timestamps in a WebLogic output log.
# splunk learn dates ------------------------------------------------------ What operation do you want to perform? (default=learn) ------------------------------------------------------ Enter choice: [Learn]/Test/Quit > l Enter full filename from which to learn dates > /home/joe/logs/weblogic.stdout.log --------------------------------------------------------------- If a line has no timestamp, hit enter; hit control-c when done. --------------------------------------------------------------- ####<Jun 3, 2005 5:37:53 PM MDT> <Notice> <WebLogicServer> <bea03> <asiAdminServer> <WrapperStartStopAppMain> <<WLS Ker nel>> <> <BEA-000327> <Starting WebLogic Admin Server "asiAdminServer" for domain "asiDomain"> -------------------------------------------------------------------------------- Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone. >
We type in the values in the event that correspond to the above fields.
> jun, 3, 2005, 5, 37, 53, pm, mdt Learned pattern.
Splunk will continue until it finds another event whose timestamp it cannot parse.
Learned pattern. . . . . . . . Validity: [From: Thu Mar 21 13:12:27 MST 2002, -------------------------------------------------------------------------------- Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone.
The timestamp does not contain an AM/PM field, so we leave that blank.
> mar, 21, 2002, 13, 12, 27,, mst Learned pattern. ...... SerialNumber: [ 33f10648 fcde0deb 4199921f d64537f4] -------------------------------------------------------------------------------- Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone. >
This line doesn't have a timestamp, so we'll just press Enter to continue.
> 0000: 9D 26 4C 29 C8 91 C3 A7 06 C3 24 6F AE B4 F8 82 .&L)......$o.... -------------------------------------------------------------------------------- Enter timestamp values as: month, day, year, hour, minute, second, ampm, timezone. >
We've trained Splunk enough, so we'll hit Ctrl-C to exit the training.
>^C Patterns Learned. Manually add these to '/opt/splunk/etc/datetime.xml' and add pattern names to timePatterns and datePatterns. -------------------------------------------------------------------------------- <define name="weblogic_1_date" extract="litmonth,day,year,"> <text><![CDATA[#+\<(\w+)\s(\d+),\s(\d+)]]></text> </define> <define name="weblogic_1_time" extract="hour,minute,second,ampm,zone,"> <text><![CDATA[,\s\d+\s(\d+):(\d+):(\d+)\s(\w+)\s(\w+)]]></text> </define> <define name="weblogic_2_date" extract="litmonth,day,year,"> <text><![CDATA[\w:\s\w+\s(\w+)\s(\d+)\s\d+:\d+:\d+\s\w+\s(\d+)]]></text> </define> <define name="weblogic_2_time" extract="hour,ampm,minute,second,zone,"> <text><![CDATA[\w+\s\d+\s(\d+)():(\d+):(\d+)\s(\w+)]]></text> </define> ------------------------------------------------------ What operation do you want to perform? (default=learn) ------------------------------------------------------ Enter choice: [Learn]/Test/Quit > q #
To put the training into effect, we'll need to add the above lines to our datetime.xml file, as shown below.
How to Add Trained Patterns
The training session creates one or more pairs of XML defintions. Each pair has one entry named ____date and one named ____time. The example below shows our WebLogic trained patterns above added corerectly to $SPLUNK_HOME/etc/datetime.xml.
- Insert all pattern names inside the datetime tags in the file.
- Insert an entry for each ____date pattern name inside the datePatterns tags.
- Insert an entry for each ____time pattern name inside the timePatterns tags.
<!-- datetime.xml -->
<!-- This file contains the general formulas for parsing date/time formats. -->
<datetime>
<define name="weblogic_1_date" extract="litmonth,day,year,">
<text><![CDATA[#+\<(\w+)\s(\d+),\s(\d+)]]></text>
</define>
<define name="weblogic_1_time" extract="hour,minute,second,ampm,zone,">
<text><![CDATA[,\s\d+\s(\d+):(\d+):(\d+)\s(\w+)\s(\w+)]]></text>
</define>
<define name="weblogic_2_date" extract="litmonth,day,year,">
<text><![CDATA[\w:\s\w+\s(\w+)\s(\d+)\s\d+:\d+:\d+\s\w+\s(\d+)]]></text>
</define>
<define name="weblogic_2_time" extract="hour,ampm,minute,second,zone,">
<text><![CDATA[\w+\s\d+\s(\d+)():(\d+):(\d+)\s(\w+)]]></text>
</define>
<...>
<timePatterns>
<...>
<use name="weblogic_1_time:"/>
<use name="weblogic_2_time:"/>
</timePatterns>
<datePatterns>
<...>
<use name="weblogic_1_time:"/>
<use name="weblogic_2_time:"/>
</datePatterns>
</datetime>
When you have finished, restart the server and check your results as described in the Test Configuration Changes section of the Admin Manual.
Report Fields
To train Splunk on which data segments in your events should be treated as fields for Report Splunks, use this command on a sample file of events.
# splunk learn fields
A Sample Session
The easiest way to explain learn fields is with a sample session, below. We'll create an _action field that can contain the values "Installed" and "Updated," and a _program field that contains the names of the programs handled and logged by yum.
[root@support01]# splunk learn fields
------------------------------------------------------
What operation do you want to perform? (default=learn)
------------------------------------------------------
Enter answer: ['learn', 'test', 'quit'] > learn
---------------------------------------------------------------
Please specify the full names of the files from which to learn.
---------------------------------------------------------------
Enter new values on separate lines. Enter a blank line when done. Enter '??' for more options.
Enter filename > /var/log/yum.log
Enter filename >
Splunk will now grab some sample lines from the file, and ask us to define a field to learn against them.
Sample lines from /var/log/yum.log
------------------------------------------------------------------------
Mar 28 14:21:57 Installed: Xaw3d.i386 1.5E-6.2.2
Apr 03 14:55:54 Updated: libstdc++.i386 4.1.0-4
--------------------------
Specify this field's name.
--------------------------
Enter fieldname > action
---------------------------------------------
Please specify examples of values to extract.
---------------------------------------------
Enter new values on separate lines. Enter a blank line when done. Enter '??' for more options.
Enter good value > Installed
Enter good value > Updated
Enter good value >
-------------------------------------------------
If there are any bad terms extracted, enter them.
-------------------------------------------------
Enter new values on separate lines. Enter a blank line when done. Enter '??' for more options.
Enter bad value >
The "bad value" prompt shouldn't appear at this point—it's sort of a misfeature. Press Enter to continue. We'll talk about bad values later.
Enter bad value >
Learning...
1 rules
Terms Learned: ['Installed', 'Updated']
------------------------------------
Are the terms extracted good enough?
------------------------------------
Enter [Y]es, [N]o, or [C]ancel > y
If Splunk had actually extracted a bad value, this would be the time to type "n" and enter the bad values from the "Terms Learned" list. This is a rare occurence, though. Instead, we'll press "y" to continue.
Enter [Y]es, [N]o, or [C]ancel > y
Using values:
Fieldname: action
Files: ['/var/log/yum.log']
GoodTerms: ['Installed', 'Updated']
BadTerms: []
------------------------------------
Learn more fields for this filetype?
------------------------------------
Enter [Y]es or [N]o >
Next, we'll train Splunk on a second field, which we'll call "program."
Enter [Y]es or [N]o > y
Sample lines from /var/log/yum.log
------------------------------------------------------------------------
Mar 28 14:21:54 Installed: emacs-common.i386 21.4-14
Apr 03 14:55:53 Updated: libgcc.i386 4.1.0-4
--------------------------
Specify this field's name.
--------------------------
Enter fieldname > program
---------------------------------------------
Please specify examples of values to extract.
---------------------------------------------
Enter new values on separate lines. Enter a blank line when done. Enter '??' for more options.
Enter good value > emacs-common.i386
Enter good value > libgcc.i386
Enter good value >
-------------------------------------------------
If there are any bad terms extracted, enter them.
-------------------------------------------------
Enter new values on separate lines. Enter a blank line when done. Enter '??' for more options.
Enter bad value >
Learning...
Learning...
2 rules
Terms Learned: ['dmidecode.i386', 'emacs-common.i386', 'emacs.i386', 'libgcc.i386', 'strace.i386', 'syslog-ng.i386']
------------------------------------
Are the terms extracted good enough?
------------------------------------
Enter [Y]es, [N]o, or [C]ancel > Y
Using values:
Fieldname: program
Files: ['/var/log/yum.log']
GoodTerms: ['emacs-common.i386', 'libgcc.i386', 'strace.i386', 'dmidecode.i386', 'emacs.i386', 'syslog-ng.i386']
BadTerms: []
------------------------------------
Learn more fields for this filetype?
------------------------------------
Enter [Y]es or [N]o > n
--------------------------------------
Learn fields for additional filetypes?
--------------------------------------
Enter [Y]es or [N]o > n
-------------------
Save rules learned?
-------------------
Enter [Y]es or [N]o > y
-----------------------------------------------------------------------------------------------------------
Runtime system uses the rules file '/home/test//splunk/etc/report.xml'.
do you wish to save to this file?
-----------------------------------------------------------------------------------------------------------
Enter [Y]es or [N]o > y
------------------------------------------------------
What operation do you want to perform? (default=learn)
------------------------------------------------------
Enter answer: ['learn', 'test', 'quit'] > quit
How to Use Trained Fields
To make use of the trained fields _action and _program from the above example, we need to restart the Splunk Server as as described in the Test Configuration Changes section of the Admin Manual.
We can then use the extracted fields in reports such as the example below.
meta::all report::[select _action, _program from resultstable where _program like %.386]
Saved Splunks and Live Splunks
Save a Splunk
Once you've performed a search whose parameters you want to save, click the This Splunk menu at the upper right of the interface and choose Save Splunk As.... A popup dialog will ask you to name the splunk. Enter a name and then click Save.
If you have Splunk Professional and are an Admin or Power User, you can make your Splunk a Shared Splunk that appears on all users' Saved Splunks menus.
Set up a Live Splunk (Splunk Professional)
Once you've saved a splunk, if you've got Splunk Professional you can set it up to run every ten minutes, every two weeks, or whatever, and to alert you if the results are unusually low, high, or drastically changed in number. You'll need to have Admin or Power User privilege set on your account, or you can ask your administrator to set it up for you from a permalink.
To make a Saved Splunk into a Live Splunk, click the Make Live link that appears when you save it. Or you can make it live later by clicking the Admin link at the upper right of the interface and then clicking the Splunks tab.
Alerts (Splunk Professional)
If a Live Splunk's results trigger an alert there are four ways it can alert users: the Splunk Server home page, email, RSS and a shell script.
Home page
Whenever you hit your Splunk server's home page, it will display any Saved Splunks that have met their alert settings.
If your Splunk Server host has outbound email enabled, Splunk can email an alert that looks like this.
From: livesplunk@splunk.enet.interfoo.net (Joe Admin) Date: November 16, 2005 3:19:23 PM PST To: admin-list@interfoo.net Subject: alert if fewer than 10. Live Splunk http://splunk.enet.interop.net:8000/?events?q=7%318989%32500%20m%69%6e%75t%65sago %3a%3a%310%20doma%69%6e%3a%3ad%65fa%75%6ct%20 triggered with the result : The Number of Events (0) was Less Than 10. Splunk Name : calls home last 10 minutes Query Terms : 7189892500 minutesago::10 index::default Auto-generated by Splunk Professional
RSS
An RSS alert from a Live Splunk looks like this. The link value is a permalink URL to run the Live Splunk now on the server.
<?xml version="1.0"?>
<rss version="2.0">
<channel>
<title>test</title>
<link>http://qa-fc4:8000/?events?q=meta%3a%3aall%20</link>
<description>Live Splunk Feed for live splunk test</description>
<item>
<title>Query run from 0 To 1134079839</title>
<link>http://qa-fc4:8000/?events?q=meta%3a%3aall%20%20starttimeu%3a%3a0%20endtimeu%3a%3a1134079839</link>
<description> The Number of Events (1000) was Greater Than 1.</description>
<pubDate>1134079840</pubDate>
</item>
</channel>
</rss>
The on-screen appearance of the alert will vary depending on your RSS reader, but generally will include the name of the Live Splunk, the time it was run, the rule which caused it to send an alert, and a link to run the splunk yourself.
You can also attach the results of the Live Splunk to the notice. If you use the report:: feature to create a report table, it will attach the report.
Shell Script
A Live Splunk can call an alert shell script that you specify in the interface. Splunk will pass five arguments to your script:
- $1—A results summary in XML.
- $2—The search terms for the Live Splunk.
- $3—The fully qualified query string for the Live Splunk.
- $4—The name of the Live Splunk.
- $5—The reason the Live Splunk fired.
Search History
Global History
Both the free Splunk Server and Splunk Professional maintain a full history of searches run on them by users. You can search the history by choosing history from the index menu or by typing index::history into the Splunk box.
To run any search in the history again, click Run This Splunk Again at the lower right corner of the result.
Personal Histories (Splunk Professional)
Splunk Professional enables any user's personal history to be searched by specifying the user:: value. The special value user::current_user means whichever user is logged into the interface.
Command Line Search Tool
Search Examples
The Splunk Server lets you run searches from a shell on the host running your Splunk Server setup. You can Splunk from your shell prompt or from a shell script.
# splunk search deny inbound hoursago::1 maxresults::5
You should get results that look something like this.
Dec 15 12:42:12 stage-test Jan 01 2004 12:48:07: %PIX-2-106006: Deny inbound UDP from 123.67.53.22/12345 to 10.0.253.252/1433 on intthee Dec 15 12:42:12 stage-test Jan 01 2004 12:48:06: %PIX-2-106006: Deny inbound UDP from 144.1.10.222/12345 to 89.89.154.72/6161 on inte Dec 15 12:42:12 stage-test Jan 01 2004 12:48:05: %PIX-2-106006: Deny inbound UDP from 45.67.123.44/12345 to 148.14.67.34/3244 on inte Dec 15 12:42:12 stage-test Jan 01 2004 12:47:38: %PIX-2-106006: Deny inbound UDP from 45.67.123.44/12345 to 89.89.154.72/6161 on inte Dec 15 12:42:12 stage-test Jan 01 2004 12:47:37: %PIX-2-106006: Deny inbound UDP from 123.67.53.22/12345 to 148.14.67.34/3244 on inte
The command supports the entire Splunk box search command syntax.
Differences from Splunk Box
Command line searches have a default maxresults:: value of 100 rather than 10,000.
Report Splunks
Create
The report:: modifier is a way to create SQL reports from your search results. If you add report:: to a search, Splunk doesn't deliver results in its usual format. Instead it creates a temporary SQL table, resultstable, based on the results set for the search, executes the value of report:: as a SQL statement, and outputs the results of the statement as a table instead of Splunk's usual results page.
resultstable contains one row for each event, and one column for each identifiable field that Splunk was able to extract from the results of the search. If Splunk cannot extract any fields from an event, it leaves that event out of resultstable so that the report doesn't contain hundreds of blank lines.
The example Splunk below first creates a results table of all events that match eventtype::logon, then selects the pre-trained address fields _ip and _url from each. _ip contains a comma-separated list of all IP addresses for each event; _url contains URLs in the event.
eventtype::logon report::[select _ip, _url from resultstable]
The above Report Splunk would produce a table with two columns, _ip and _url. If an event did not contain a value for _ip or _url, Splunk would exclude its row from the report rather than include an empty row.
Syntax
- All keywords must be typed in lowercase, contrary to SQL convention. A fix for this is pending.
- select is the only supported SQL statement.
- Report Splunks are built on SQLite. See SQLite syntax reference for select for specifics. For example, SELECT does not support TOP and BOTTOM.
- resultstable is the only table available to be queried. It does not persist after the Report Splunk completes.
- select statements may be nested.
Shortcuts
Splunk includes several shortcuts to SQL's select syntax for quick splunking.
- report::[*]
Shortcut for report::[select * from resultstable]
- report::[field]
Shortcut for report::[select field from resultstable]
- report::[top field]
Shortcut for report::[select field, count(*) from resultstable group by field order by count(*) desc]
- report::[rare field]
Shortcut for report::[select field, count(*) from resultstable group by field order by count(*) asc]
Export...
To save your Report Splunk as a CSV or text file, use the Export function on the Splunks menu.
How Splunk Recognizes Fields
There are three ways Splunk populates fields with values from events.
- It looks for segments that match the acceptable formats and values for IP addresses and URLs. It uses these to populate the fields _ip and _url.
- It looks for segments that are proably name-value pairs, such as user=jsmith or level:3. These are used to create fields such as _user and _level on the fly.
- You can train Splunk to recognize patterns in your data and use them to populate fields with specific names. The training will work on events already in the index, as well as new events indexed after training.
report:: and maxresults::
Adding report:: to a Splunk changes the default value of maxresults:: to 500 instead of 10,000. This is to prevent browsers from choking on large reports, but you can use higher values. A pending fix will eliminate the need to worry about browser performance.
The number of rows in a report will usually be less than the value of maxresults::, because the report won't include rows that don't contain the fields specified by the select statement given as the value of the report:: modifier.
No Results?
If you expect results in a report:: but don't get any, it means there were no matching fields in the first maxresults:: results for your Splunk. You can increase the value of maxresults::, but it's more effective to sharpen your Splunk so that relevant values appear in the first 1000 results. Set a time range or specify your event type or source type to remove events that aren't necessary for the report.
Examples
meta::all report::[select * from resultstable where _ip like '%445%'] host::gwrk1 eventtype::?9034 report::[select _ip, count(*) from resultstable group by _ip order by count(*) desc]
See Also
External Links
- SELECT statement (SQLite)
Export
To save a report to file, go to the Splunks menu at the top of the results page and choose Export.... You can choose either text or comma-separated value (CSV) format.
Please be patient while your browser saves the file. It may seem to hang for a few seconds.
Learn Fields
If Splunk does not create the fields you expect from your results, you can train it to recognize fields using the shell command splunk learn fields on the Splunk server host.
# splunk learn fields
You will then be able to search for those fields using report::. For example, if you train Splunk to recognize the field _OriginatingIP, you can then run the following Splunk:
report::[select _OriginatingIP from resultstable]
No database involved
To be clear, Splunk has no relational database to drag on its performance. Instead, whenever a Splunk search contains report:: the Splunk server creates a short-lived resultstable at run time. The report:: operator can then make a SELECT statement on resultstable.
See Also
Faster Splunking Tricks
Ready, Fire! Aim.
The biggest speed boost you can give yourself is to splunk first and ask questions later. Start with a broad search for the first term that comes to mind. Then use Ctrl-Alt-click (on Macs, cmd-option-click) to filter out results you don't want. Use the same approach you do when Googling or when piping a file through longer and longer grep commands to filter it down to what you're looking for.
Use maxresults::
To control the length of a search (and hence its speed) add the maxresults:: modifer to specify the number of results after which it should finish.
Set a Timerange
Splunk partitions its indexes by timestamp both in memory and on disk. So the smaller the range between start and end times in a splunk of the same index, the smaller the amount of RAM or disk the server will need to read, and the faster it will finish. The minutesago::, hoursago:: and daysago:: modifiers are quick ways to reduce the length of a search.
Hide Events by Time
The Events by Time histogram takes extra time to load data from the server, and then more time to render it in your browser. Click Hide Events by Time before running a search to shorten the time it will take to appear.
Use Related and Unexpected Carefully
Most modifiers don't affect the time it takes to return results. But related:: and unexpected:: require the server to examine more complex data structures. These can slow down a search.
Turn off Full Segments
If you're Splunking long event lines, your browser may slow as you mouse over event segments because it can't keep up with highlighting requests. To speed up highlighting go to the Preferences menu at the top of the interface and choose Segment Selection -> Outer or Segment Selection -> Inner.
Reference Chapters
Search Command Syntax
System administrators can reconfigure which characters Splunk treats as breaking characters described below, and which breaking characters separate major or minor segments. See the Splunk Admin Manual for configuration instructions.
Keywords
Splunk assumes keywords are separated by spaces, and are composed of letters and numbers. They are separated by breaking characters into major segments that may contain two or more minor segments.
refer google com
Case Sensitivity
Splunk is case-insensitive except for the Boolean operators AND, OR and NOT described below.
These searches are identical:
foobar
Foobar
FoObaR
Breaking Characters
By default Splunk is configured to treat characters such as &, %, . and , in data it loads as breaking characters that separate searchable segments. You can't search for breaking characters. Parentheses have a special function described below.
Major and Minor Segments
Splunk's default configuration treats segments separated by periods and other punctuation marks as minor segments, and those separated by spaces as major segments. If you search for a term that appears as a minor segment, it will be highlighted on your results page. But when you roll over it to click it, the entire major segment it belongs to will highlight.
Left-to-Right Matching
When searching the index, Splunk tries to match minor segments inside a major segment by starting from the leftmost segment and working to the right.
For example, let's say we have events with URLs of the form http://amazon.com/exec/obidos/12345 in them. This query would return results because each term is a minor segment that can be matched inside URL strings in the index.
amazon com exec obidos
This query would also return matches because the minor segments in the query match URLs in the index starting from left to right.
http://amazon.com/exec
This query would return no matches because it would not match the first minor segment in the search term ("amazon") with the first minor segments in the indexed data ("http").
amazon.com/exec/obidos
Modifiers and Meta Data
The format for specifying both modifiers and meta data value in the Splunk box is name::value, for example host::web1.splunk.com.
host::jupiter sourcetype::linux_messages_syslog fatal
See the Search Modifiers section for a complete list of built-in meta datanames and search modifiers. Your administrator may also configure other meta data values that you can use. Unfortunately there's no way yet to get a complete list of those available.
Boolean logic—AND, OR, NOT, ()
You can perform nested Boolean searches on any index. Use parentheses separated by whitespace to mix uppercase AND, OR and NOT operators. NOT is a Boolean operator; Not, not and nOT are case-insensitive search keywords.
( foo NOT ( bar OR baz ) )
AND is implied by default between search terms, but you can plug it in anyway. These Splunks are equivalent.
foo bar NOT baz
foo AND bar NOT baz
Precedence
The order of precedence among operators is:
( )
OR
AND, NOT
When in doubt, use parentheses.
( Parentheses )
Splunk supports parentheses, but you must put space around parentheses to make clear they are Boolean operators and not part of search terms. This is so Splunk can be configured to treat ) and ( as searchable characters.
Wildcards *
You can place wildcards at the start or end of keywords.
*inux
linu*
You can place wildcards at the end of meta data terms, but nowhere else in the term.
host::webserver*
You can effectively wildcard the middle of a keyword like this, although it may return a few other results that match by containing two separate terms.
supercali* *expialidocious
You can't wildcard both ends. This search, sadly, won't return anything.
*upercalifragilist*
+Literals
To explicitly search for a term with punctuation marks in it, prefix the term with +.
This example would search for the string "index::foo" in your index, rather than specifying an alternate index to search.+index:foo
You can search for terms that begins with "+" using the same syntax. This example would search for the literal string "+800".
++800
"Quoted strings"
Splunk does not currently support the use of quotes to combine segments separated by breaking characters. This search would return no results.
"four score and seven years ago"
This search will find results, but possibly many you're not looking for. You may have to Ctrl-Alt-click unwanted matches away. (On Macs, cmd-option-click.)
four score and seven years ago
Search Modifiers
Modifiers take the format name::value.
Modifiers may appear anywhere in a splunk command—before, after, or in between regular keywords and Boolean operators.
Some, as noted, are only allowed one value per search. The last one will be evaluated and the rest ignored. A splunk for maxresults::10 apache maxresults::5000 will return at most ten results.
Your Splunk administrator may configure additional meta data values besides those listed here. Unfortunately there is currently no way to obtain a list of all meta data defined in the server's configuration or present in an index. Check with an administrator or power user.
maxresults::
The maximum number of results to return. The default is 10,000. Only the last declaration of maxresults will be evaluated.
foo maxresults::20
If there are more results than you specified the tab summaries will add a > in front of their summary counts, e.g. Events (>10,000).
daysago::
Events within the last N days. Only the first declaration of daysago will be evaluated.
daysago::1
index::
The index to search—default, history, _internal or, on Splunk Professional, another index defined by the administrator. Only the first declaration of index will be evaluated. Its default value is index::default. On the free Splunk Server this resolves to index::main. On Splunk Professional it resolves to whichever index is currently set as the default.
index::_internal finished
There are several indexes built into Splunk by default.
- main
The main data index.
- history
An index of all searches performed on the server (Splunk Server) or by the current account (Splunk Professional.)
- _internal
The index of Splunk server events.
hoursago::
Results within the last N hours. Only the first declaration of hoursago will be evaluated.
greylisting hoursago::4
eventtype::
Events with an event type or tag that matches the specified value.
Local event types have this format.
eventtype::?17
Globally defined event types downloaded from Splunk Base have this format.
eventtype::SP-CAAAAD
For backwards compatibility eventtype:: will also match tags.
eventtype::apache
host::
Results generated by the specified hostname, or at least whose host:: value was set to that hostname during indexing.
host::webserv* 404 daysago::1
minutesago::
Events within the last N minutes. Only the first declaration of minutesago will be evaluated.
minutesago::10 login
related::
Events with segment values (e.g. 404 or joeuser) matching one or more in the current event. You cannot manually enter related searches; you need to click the Related link on a specific event. The splunkSearch interface will insert a keyword of the form related::4:156287821 into the Splunk box. The number is a hash value that only makes sense to the server.
related::4:156287821
Unlike all other current Splunk searches, related results are sorted by relevance rather than by time.
report:: (Splunk Professional)
This modifier creates a Report Splunk from your Splunk results. You can then choose Export Splunk As... to save the report to a file. See the section on Report Splunks for details and examples.
savedsplunk::
Use this modifier to include the terms of a Saved Splunk inside another Splunk, further refining it.
savedsplunk::logins minutesago::40
You cannot include a savedsplunk:: parameter inside a Saved Splunk—the feature is not recursive.
similar
Events whose event type signature is somewhat like that of the current event. There is no similar modifier; it's a special syntax version of eventtype. If you click on the Similar button for an event you'll get a splunk that looks like this.
eventtype::?91-3
The above means "events whose type is within 3 degrees of separation from type 91." You can manually enter values from 0 (identical) to 9 (not similar at all.)
eventtype::?91-1
sourcetype::
Events whose source type matches the specified value.
404 sourcetype::php_*
tag::
Events with a tag that matches the specified value.
tag::apache
unexpected::
Results that lie outside observed patterns in the index by the specified value of 0 (expected) to 9 (most unexpected).
unexpected::9 sourcetype::php_*
You can specify a single value, or a range between two values.
unexpected::4-9 host::webdev
Home Page Controls
The home page contains the controls listed below.
My Account (Splunk Professional)
Click here to go to the Admin page to edit your own account.
Log Out
This one is probably self-explanatory.
Help
Connect to Splunk's live documentation at www.splunk.com
Admin
This link opens a tabbed page to manage accounts and see statistics.
Splunk Watch (or Splunk Professional Watch)
A status and statistics panel with tables of sources, events, terms, and data volume indexed by the server, plus current processes running.
Splunks
The list of Saved Splunks for the server (Splunk Server), or Saved Splunks and Live Splunks for your account (Splunk Professional.) This is where you create and edit Live Splunks. See the how-to section on Saved Splunks and Live Splunks for instructions.
Users (Splunk Professional)
A control panel through which admins can add, delete, or edit user accounts.
License (Splunk Professional)
License key information for the server. You can click to renew a license that has expired or will expire soon.
Splunk Box
The text entry area for search terms. See the Search Command Syntax for full specs.
Search button
Click this button to submit the contents of the Splunk box as a search.
Saved Splunks
This is your list of Saved Splunks (and for Splunk Professional users, Live Splunks) that you've created and saved, or that your administrator has saved for all users.
Save Splunk As...
Pops up a dialog so you can give your current search terms a name and add it to your Saved Splunks list
Manage My Splunks
Opens the Admin page to the Splunks tab.
Set/Clear Timerange
If you click the click icon in the search box, you'll find input boxes for start and end times for your search. You can adjust them with your mouse, your keyboard's arrow keys, by clicking on an event's timestamp, or by typing explicit values into them. Click one of the calendar icons to pop up an interactive calendar.
Look for the menu buttons next to START and END. These let you clear the setting or quickly select from a list of popular time settings such as 1 hour ago.
Splunks
Pops up a menu of options for saving or manipulating your current search.
Save...
Identical to the option on the Saved Splunks menu, it pops up a dialog to let you name your current set of search terms and add it to your list of Saved Splunks.
Pops up a dialog to save your current search results to a file.
Export...
Pops up a dialog that lets you open your current results in an editor or save them to your computer desktop as a file. The default filename will contain identifying terms from the search in it, e.g. splunk_access.2.log_get_xjjk_.log. Subsequent saves will have -1, -2, etc. appended to their filenames rather than overwriting existing files.
Permalink
Replaces the URL in your browser's URL window with a long, specific URL that recreates your exact search settings. You can bookmark the URL or send it to another user.
Turn Word Wrap Off/On
If your results include multi-line events with columnar data, their format may be disrupted by HTML formatting to fit your browser window. Click this link to display events without wrapping.
Index Menu (index::)
Lets you select the Splunk index to search. The free Splunk Server is limited to the three created at installation time—main, history and _internal. Splunk Professional allows an unlimited number of user indexes. See the Splunk Admin Manual section on Creating Additional Indexes for instructions to create and manage indexes.
This option corresponds to the index:: modifier in the Splunk box.
default
The main index of your data.
history
A complete list of past Splunks by all users of the server.
_internal
The Splunk Server's own logs.
Sources Indexed
A table of data about the currently selected index.
Source Types
Source types are meta data Splunk sets on individual sources—input files or streams—to identify the class of data in them. The Splunk Server and Splunk Professional are trained to recognize about forty common source types such as apache_access, linux_syslog_message, and weblogic_stdout. If Splunk loads events from a source but doesn't recognize the source type as one it knows, it sets the sourcetype parameter for those events other either too_small or UNKNOWN-1578458965 with a unique 10-digit number.
too_small
Splunk sets the sourcetype too_small on files without enough events (typically < 100) for it to definitively determine a value.
UNKNOWN-2569854367
If Splunk has gathered enough events from one source to uniquely identify its type but can't match it to a known source type, it assigns a serial 10-digit number as the sourcetype value. (The number is a hash value with no particular meaning.)
You can edit the value to anything you want by clicking the menu button next to the sourcetype:: value in the event. Your edit will apply to all events with the same sourcetype value. See the section on Renaming Source Types.
Sources
A list of all files, pipes, and any other inputs that have been loaded for the current index since it was created.
Index a file now
(This feature is not available to regular Splunk Professional users. You need an Admin account.)
Click this button to upload a file through your browser into the Splunk index.
More
This link appears when there are more Source Types or Sources than would fit on a comfortably-sized page. Click it to show the full list.
Live Splunks (Splunk Professional)
A table of currently configured Live Splunks for your account.
Help
Connect to Splunk's online documentation at www.splunk.com
File a Bug
Connect to Splunk's online support site to report a problem. We hate bugs, but we love our customers who report them!
Results Page Controls
Many results page controls match those on the home page. These are described above.
The rest of this section describes controls unique to the results page.
Search Results—Everything's Clickable
Each search result is a unique event indexed from source files or streams. Splunk extracts several fields and adds several additional links for each result. Mouse over it to highlight individual segments. Matching values in other segments will highlight in sync.
Remember: Everything's clickable.
Rank
The result's order, from 1 through N. Results are sorted in reverse chronological order for all searches except those that specify a related:: modifier.
Event
The text of the event itself.
Event Type (Tags)
The event's signature pattern of segments. A value of the form ?34 is assigned by the server while loading the data. Users may replace these values with one or more tags entered locally or found through the Look up Event feature. See the section on Event Types and Tags for details and instructions.
Look Up Event
Click this link to look up the current event's event type in Splunk's online community database of documented event types. See the section on Event Types and Tags for details and instructions.
Timestamp
Splunk extracts a timestamp from every event it indexes, or else assigns it one based on its load time. Splunk timestamps are normalized to a standard format and the Splunk host's time zone. You'll still be able to see the original timestamp in your event as well, but Splunk timestamps make searching and sorting easy.
Similar
Click this link to add, remove, or search for events whose event type is somewhat like the current event. Note that Splunk adds a specially formatted eventtype meta data value to your search.
You can edit the final integer to be anywhere from 1 (very similar) to 4 (remotely similar.)
Related
Click this link to add, remove, or search for events with one or more segment values (e.g. 404 or joeuser) that match values in the current event. Note that Splunk adds a specially formatted related modifier to your search.
You cannot modify this term.
Source Type
In the same way it assigns event types to individual events, the server also assigns unique values to each different type of input stream. These values are local to the index. You can rename them to be meaningful, e.g. "cisco_syslog".
Host
The host that generated the event. To prevent errors Splunk doesn't try to deduce host automatically as it does timestamps. The value must be explicitly configured by the system administrator. See the Splunk Admin Manual section on How to Load Your Data for instructions and examples of setting the host value.
Source
The name of the file or stream from which the event was loaded.
Show Source
Click this link if you need the full content of an event. Your browser will pop up a new window (be sure you haven't blocked it) with the original raw data for the event. The popup window has forward/backward arrows so you can step through the original source stream, e.g. a series of maillog events loaded from the same file.
Tabs
Splunk's tabbed results group the results of the current search by the meta data present in each result, sorted in descending order of frequency.
Events
The list of individual events sorted by timestamp, most recent first. Click on any part of the event as described in Search Results above to add, remove, or search for specific values displayed in the results.
Event Types
The list of unique event types found in the search results, sorted in descending order of frequency. Click on an eventtype to add, remove, or search for it.
Tags
The list of unique tags found in the search results, sorted in descending order of frequency. Click on a tag to add, remove, or search for it.
Source Types
The list of recognized, unrecognized, and user-edited source types in descending order of frequency. Click on a source type to search for it.
Hosts
The list of unique hosts found in the search results, sorted in descending order of frequency. Click on a host value to add, remove, or search for it.
Sources
The list of unique sources found in the search results, sorted in descending order of frequency. Click on a source value to add, remove, or search for it.
Showing __ per page
You can set the interface to show 5, 10, 20, or 50 results on each page.