Deploy Splunk Enterprise on Kubernetes: Splunk Connect for Kubernetes and Splunk Insights for Containers (BETA) - Part 1

Since leaving KubeCon 2017, we've been hard at work exploring Kubernetes and container technologies with our customers, partners and the greater CNCF community. Together, we’re all examining how this amazing new technology will change the way we work and the new opportunities it can provide us in running our applications.

At KubeCon 2018, we were excited to share the learnings we’ve gained thus far, and enjoyed seeing you in Seattle to talk more about what you want to achieve this year with Splunk and the Kubernetes ecosystem.

In the following walkthrough, we’ll share and invite you all to contribute to test scenarios that deploy Splunk’s supported docker image on Kubernetes. We’ll also integrate Splunk Connect for Kubernetes to monitor our Splunk deployment and the rest of our Kubernetes resources, and cover how our new Splunk SmartStore feature can be configured to take advantage of Splunk’s ability to decouple Compute and Storage. This also marks the beta availability of Splunk Insights for Containers (BETA).


Access to a functioning Kubernetes environment, some options include:

We’ll be using the Heptio AWS Quickstart in this walkthrough, but most any Kubernetes option should do.

On your local machine, you’ll need:

  • Docker and Kubectl installed
  • The Splunk Docker image (available on dockerhub)
  • Access to an S3 storage endpoint if you’d like to deploy SmartStore (optional)
  • Any other requirements your cloud provider of choice needs. (This could include AWS cli tools, gcloud cli tools, etc. Consult relevant documentation as necessary).

To get started, pull down the Splunk image to your local machine.

We’ll use it to generate some configs to push to Kubernetes with kubectl.

docker pull splunk/splunk:latest

Review the deployment strategies we’re exploring, inspired by Splunk Validated Architectures. (We say “inspired by” because the examples included are meant to explore Splunk’s existing architectures and what they may evolve into in Kubernetes, not to serve as an official deployment guidance or as replacement for existing Splunk Architecture guidance and support. *takes off lawyer hat*)

We recommend most users start with our 3idx1sh1cm deployment, as it provides a great starter bootstrap environment that focuses on deploying a three-member Indexer Cluster, with the Master acting as your “Admin Search Head,” and a standalone Search Head you can provide to users.

The 3idxc3shc1cm1lm1dep is for those looking to explore larger Splunk deployment concepts at scale, and is a “Lift and Shift” of the traditional Splunk environment you might see in the field. We chose to stick with the architecture concepts we already know—that way, we can focus on learning the key Kubernetes concepts Splunk can use and how they can help us today, without drastic changes to the topologies our admins know.

Having said that, the inclusion of NGINX as a means to serve files centrally is a great example of how the traditional roles in a Splunk deployment may evolve with the power of Kubernetes, its ecosystem, and the orchestration tools you use.

The test architectures include both non-persistent and persistent variants, so whether you’re simply learning and practicing your Splunk-Fu, developing against Splunk and Kubernetes, or considering running in Production and want to explore persisting data and state in k8s, there’s a good starter kit here for you to explore and start to chart your Kubernetes course.

Deploy Splunk Enterprise on Kubernetes

Clone the docker-splunk repo

Git clone

navigate to the nginx-data-www folder

cd docker-splunk/test_scenarios/kubernetes/nginx/nginx-data-www

use the following command to generate a sample default.yml

docker run splunk/splunk:latest create-defaults > ./default.yml

This will yield a great starting point for configuring your cluster, where we’ll enable indexing clustering to true, everything else will remain the default.

retry_num: 50
    default: /opt/splunk/etc/apps
    httpinput: /opt/splunk/etc/apps/splunk_httpinput
    idxc: /opt/splunk/etc/master-apps
    shc: /opt/splunk/etc/shcluster/apps
  exec: /opt/splunk/bin/splunk
  group: splunk
  hec_disabled: 0
  hec_enableSSL: 1
  hec_port: 8088
  hec_token: 1b327f13-7e51-44a3-9439-f615e9ff6d0f
  home: /opt/splunk
  http_port: 8000
    enable: true
    label: idxc_label
    replication_factor: 3
    replication_port: 4001
    search_factor: 3
    secret: wl/NAKo0Nzaopd6CaOb1dY2H6LvmTkLg
  opt: /opt
  password: 4m5iSmk4oJbaeq7c6KtEmjuqLVaKt73C
  pid: /opt/splunk/var/run/splunk/
  s2s_port: 9997
    enable: false
    label: shc_label
    replication_factor: 3
    replication_port: 4001
    secret: /UiXWBqmRGc6JhDnhYELcXHhis76SAIt
  svc_port: 8089
  user: splunk

Once you've generated your default.yaml, you can insert valid license XML into the mySplunkLicense.lic file (optional, will run without a license under the Enterprise trial), when complete, it should look something like this:

cd nginx-data-www/
ls -la
total 16
drwxr-xr-x  4 mmodestino  staff 128 Sep 26 23:50 .
drwxr-xr-x  5 mmodestino  staff 160 Sep 26 22:35 ..
-rw-r--r--  1 mmodestino  staff 1443 Sep 26 18:18 mySplunkLicense.lic
-rw-r--r--  1 mmodestino  staff 856 Sep 26 23:50 default.yml

Now, let’s create a Kubernetes Namespace called “Splunk” to work in:

kubectl create ns splunk

Then, step back into the nginx directory create your nginx configmaps.

Note: While we could have just created these configmaps as part of the deployment yamls we’ll use shortly, we wanted to demonstrate the kubectl capability to create configmaps from a directory of config files. This is a capability that lines up nicely with existing Splunk App structures, and makes turning a Splunk app into a Kubernetes configmap, real easy.

cd ..
kubectl -n splunk create configmap nginx-data-www --from-file=nginx-data-www

Then create one for the sample nginx conf file:

kubectl -n splunk create configmap nginx-config --from-file=nginx-static.conf

The deploy the nginx manifests:

kubectl -n splunk apply -f manifests

If deployed successfully, you’ll have an nginx pod running:

kubectl -n splunk get pods
NAME                              READY STATUS  RESTARTS AGE
splunk-defaults-cff5cb574-kzf9x   1/1   Running 0        8s

You should also be able to reach the nginx page serving the license and the defaults file from your browser, with kubectl -n splunk port-forward.

For example:

kubectl -n splunk port-forward splunk-defaults-686b5885f6-k7crh 9999:80

Point your browser at localhost:9999 or curl localhost:9999 and you should see your files being served.

Now that we have our license and default configurations in place, let’s use them in our pods!

Return to the ../test_scenarios/kubernetes folder and run the manifests.

cd ..
kubectl -n splunk apply -f 3idx1sh1cm

This folder of yamls deploys a Cluster Master, three clustered Indexers and a single Search Head.

kubectl -n splunk get pods
NAME                               READY STATUS  RESTARTS AGE
indexer-0                          1/1   Running 0        26s
indexer-1                          1/1   Running 0        12s
indexer-2                          1/1   Running 0        10s
master-6d7b98f8f5-tb7sh            1/1   Running 0        26s
search-5944fc8696-w4z9n            1/1   Running 0        26s
splunk-defaults-686b5885f6-k7crh   1/1   Running 0        8m

As the pods spin up, you can monitor their progress with the kubectl logs command.

For example:

Kubectl -n splunk logs -f master-55c7bcf764-8z5cj

This will allow you to watch as the Ansible playbooks that orchestrate the environment complete.

Once the play summary is complete, and as long as it’s free of errors, you should be ready to jump into any of the pods to ensure they have come up as expected:

PLAY RECAP *********************************************************************
localhost                  : ok=27 changed=12 unreachable=0    failed=0  

Thursday 06 December 2018  14:51:45 +0000 (0:00:00.027)       0:01:20.770 ***** 
splunk_common : Install Splunk ----------------------------------------- 36.94s
splunk_common : Start Splunk ------------------------------------------- 15.04s
splunk_common : Restart the splunkd service ---------------------------- 12.57s
splunk_common : Enable the Splunk-to-Splunk port ------------------------ 3.88s
Gathering Facts --------------------------------------------------------- 1.89s
splunk_common : Check if Splunk is running for the first time ----------- 1.86s
splunk_cluster_master : Set the current node as a Splunk indexer cluster master --- 1.74s
splunk_cluster_master : Apply the cluster bundle to the Splunk cluster master --- 1.03s
splunk_cluster_master : Set indexer discovery --------------------------- 0.99s
splunk_common : Generate user-seed.conf --------------------------------- 0.56s
splunk_cluster_master : Setup tcpout:group1 stanza in outputs.conf for clustered deployments --- 0.45s
splunk_common : Remove installer ---------------------------------------- 0.43s
splunk_common : Remove user-seed.conf ----------------------------------- 0.34s
splunk_cluster_master : Setup tcpout stanza in outputs.conf for clustered deployments --- 0.27s
splunk_cluster_master : Setup indexer_discovery:splunk-indexer stanza in outputs.conf --- 0.27s
Provision role ---------------------------------------------------------- 0.17s
splunk_cluster_master : Lower indexer search/replication factor --------- 0.14s
splunk_cluster_master : Get default replication factor ------------------ 0.14s
splunk_cluster_master : Get indexer count ------------------------------- 0.13s
splunk_common : include_tasks ------------------------------------------- 0.13s
Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log

Let’s jump into the master and make sure the cluster has formed:

kubectl -n splunk port-forward master-6d7b98f8f5-tb7sh 9999:8000

Log into the Splunk instance with the default credentials admin/helloworld. Navigate to Settings > Indexer Clustering.

Looking good!

I can see my Indexers and Search Heads have joined the cluster.

While what we’ve just accomplished was incredibly easy and fast, I’d be remiss if I didn’t pause here to re-enforce what a game changer this is. Being able to instantiate fast, repeatable deployments is a major benefit that orchestration and containerization of Splunk brings to us. What an exciting glimpse into the future of making Splunk easier and more accessible to everyone!

Tune in for the following parts of this series to learn:

  1. How to push some configs out to our indexer cluster to enable some indexes to send data to, as well as configure SmartStore
  2. How to use Splunk Connect for Kubernetes to monitor your cluster and forward data to your Splunk cluster
  3. How to clean up all your resources

Thanks for checking out our test scenarios, and for more on monitoring your Kubernetes stack learn more with our Beginner’s Guide to Kubernetes Monitoring. And if you’re interested in gaining immediate insight into your Kubernetes stack including performance metrics for your clusters, pods, containers, and namespaces, as well as log, metric, event, and metadata, sign up for Splunk Insights for Containers (BETA) to test this out.

Matthew Modestino (aka Matty Mo) is an IT Operations Analytics Practitioner at Splunk and an honorary member of the 2018 SplunkTrust! Prior to joining the Splunk IT Markets Product organization, he served as a Customer Success Manager, working closely with customers to accelerate the value of Splunk in their organizations. Prior to joining Splunk in early 2016, Matthew spent 10 years at TELUS, a Canadian Telecom, specializing in network assurance and root cause analysis of complex network issues. 

Join the Discussion