Distributed Search

In this post I will be talking about a feature of Splunk that got turbo charged for 4.0 : Distributed Search.

Splunk is a great tool when it’s just running on a single system but distributed search has some great advantages.

  • Provides completely different views into the same data by having different apps on different systems.
  • Allow leveraging of map reduce architecture to run complex queries.
  • Linearly scale Splunk indexing by simply adding more servers.

Terminology Used:

  • Search Head : The splunk instance that the user logs into and distributes searches from.
  • Search Peer : A splunk instance that receives search requests from the search head.
  • $SPLUNK_HOME : The root of your splunk install, this environment variable will be automatically set if you source $SPLUNK_HOME/bin/setSplunkEnv on unix.

Note this post will be written with *nix in mind but it is applicable to Splunk on windows as well.
For a basic primer and a nice diagram you can check out

Setting up Distributed Search:

Access to Distributed Search in 4.0 is based on a public private key architecture similar to ssh-keys. We provide secure access to a search peer by installing a public key on that search peer that corresponds to a private key on the search head.

On the search head these keys can be found in $SPLUNK_HOME/etc/auth/distServerKeys private.pem is the private key and trusted.pem is the public key. These keys are generated the first time you run splunk so if you haven’t run splunk yet you should start it now. If you are adding the search peers via the cli or by editing the distsearch.conf file you will need to install the public key from the head onto your peers if you add via the ui you can skip this step by providing user credentials on the search peer and splunk will install the key for you.

To install the public key on the search peer you must create a sub directory on the search peer in $SPLUNK_HOME/etc/auth/distServerKeys with the same name as the SPLUNK SERVER NAME of the search head ( you can find this by doing a “splunk show servername” from the ui or in Server Settings in the manager UI ) so if you are trying to distributed to a search peer named FOO from a search head named BAR you must create a directory $SPLUNK_HOME/etc/auth/distServerKeys/BAR on FOO. Copy the trusted.pem from BAR into this new directory. Now you can add FOO as a peer on BAR by doing splunk add peer : in the cli on BAR.

Note that as your keys are stored by splunk server name if you change this on the search head you will need to update your search peers.

Using Distributed Search to provide different views of your data:

One of the best features of distributed search in 4.0 is that the search on the search peer runs as if that search was running on the search head. This means that only those eventtypes, tags and search time extractions that are defined on the search head will be visible when you search. This means that you can create different search heads that each define a different way of looking at the data and you only have to give users access to the view they need to see.

Due to the certificate nature of distributed search you do NOT need to replicate your users across all your distributed peers, any user that can log into your search head can search any search peer that has that head’s public key installed.

Map Reduce Distributed Search:

Map reduce is an increasing common buzz word these days, basically what it means for Splunk is that when you are doing a distributed report as much work as possible will be off loaded to the search peers. So you can search and report over billions of events in a reasonable amount of time if you have your events distributed across many servers.

No special knowledge is needed as all of Splunk’s internal reporting commands are already set up to work in this way! Neither the Splunk administrator or the Splunk users ever need to think about it. If you have 10 search peers each with 100 million access log events it will take the same amount of time to report on the top ips across all billion events as it would to preform the same search on just one of the search peers directly.

Index Scaling:

I’m not going to get into distributing the indexing load across indexers in this post as I want to focus on search but there are some great docs available at :

Debugging Distributed Errors:

Also new to splunk 4.0 is the enhanced UI management of peers accessible from Manager -> Distributed Search -> Distributed Peers. Here you will see a list of all the peers you have access to as well as their statuses. Here is a break down of what each possible status means :

  • Up : The peer is functioning correctly and the search head can run searches against it.
  • Down : The peer is unreachable.
  • Not a splunk server : This peer does not appear to be a splunk 4.0 server, commonly the web port was entered instead of the splunkd management port.
  • Blacklisted : The peer is blacklisted in distsearch.conf.
  • Version Mismatch : This peer is running the free version of the product and so cannot participate in distributed search.
  • Certificate Mismatch : This peer does not have the search head’s public key installed.
  • Duplicate License : This peer has the same license as another peer or the search head.
  • Duplicate Servername : This peer’s servername collides with another peer or the search head.

Brian Murphy
Senior Software Splunker
Splunk Inc.

Posted by


Join the Discussion