Deploy Splunk Enterprise on Kubernetes: Splunk Connect for Kubernetes and Splunk Insights for Containers (BETA) - Part 3

Welcome back to the finale of our blog series exploring Splunk on Kubernetes!

In this final post, we’ll deploy Splunk Connect for Kubernetes, giving us full visibility of our Kubernetes logs, metadata and metrics. We’ll install some apps, including a sneak peek into the newly announced Splunk for Container Monitoring (BETA). Once complete, we’ll help you clean up all the resources created in this walkthrough!

If you have followed Part 1 & Part 2 in this series, your Splunk namespace should be looking like this:

Let’s continue!

Deploy Helm & Tiller

We’re going to deploy using Helm, so let’s get Tiller deployed to our Splunk namespace.

If you do not use Helm, you can deploy manually using the manifests here (we will not cover the customizations needed in the manifests in this blog, though we recommend to check out Helm template command as a way to render the templates locally).

To make things easy in dev, I’ve provided yaml to create a service account for Tiller:


apiVersion: v1
kind: ServiceAccount
  name: tiller
  namespace: splunk
kind: ClusterRoleBinding
  name: tiller
  namespace: splunk
  kind: ClusterRole
  name: cluster-admin
  - kind: ServiceAccount
    name: tiller
    namespace: splunk

This will give Tiller cluster-admin rights and allow it to deploy apps to our cluster. While the topic of Tiller permissions is one you’ll need to balance with security in real-world deploys, for the scope of this demo and in labs this should be fine. See Helm Docs for more!

kubectl -n splunk apply -f tiller-rbac-config.yaml

Now that we have a service account for Tiller to use, we’ll install Helm on our local machine as outlined here.

Then run the following command, making sure you have access to your Kubernetes cluster from the console session you’re using:

helm init --service-account tiller --tiller-namespace splunk

If deployed successfully, you’ll see a Tiller pod in your namespace:

kubectl -n splunk get pods
NAME                               READY STATUS RESTARTS AGE
indexer-0                          1/1 Running 0 2h
indexer-1                          1/1 Running 0 2h
indexer-2                          1/1 Running 0 2h
master-6d7b98f8f5-nnl9n            1/1 Running 0 2h
search-5944fc8696-hpj5m            1/1 Running 0 2h
splunk-defaults-686b5885f6-k7crh   1/1 Running 0 5h
tiller-deploy-79f76f8b55-96tq4     1/1 Running 0 8s

Now copy this sample values.yaml to your local machine, and ensure the HEC token and indexes match the ones from the ta_containers in Part 2 of this walkthrough!


#global settings
  logLevel: info 
      protocol: https
      insecureSSL: true
      host: hec
      token: 00000000-0000-0000-0000-000000000000

#local config for logging chart
  journalLogPath: /run/log/journal
      indexName: cm_events 

#local config for objects chart      
    create: true
    create: true
    name: splunk-kubernetes-objects
    insecureSSL: true
        - name: pods
          interval: 30s
        - name: namespaces
          interval: 30s
        - name: nodes
          interval: 30s
        - name: services
          interval: 30s
        - name: config_maps
          interval: 30s
        - name: persistent_volumes
          interval: 30s
        - name: service_accounts
          interval: 30s
        - name: persistent_volume_claims
          interval: 30s
        - name: resource_quotas
          interval: 30s
        - name: component_statuses
    interval: 30s
        - name: events
          mode: watch
        - name: deployments
          interval: 30s
        - name: daemon_sets
          interval: 30s
        - name: replica_sets
          interval: 30s
        - name: stateful_sets
          interval: 30s
      indexName: cm_meta    
#local config for metrics chart
    create: true
    create: true
    name: splunk-kubernetes-metrics
      indexName: cm_metrics

Deploy Splunk Connect for Kubernetes with the following command, making sure to point to the values.yaml we just created.

helm install --name kubecon-demo-2018 --tiller-namespace splunk --namespace splunk -f values.yaml

If successful, you should now have your Splunk Connect for Kubernetes pods running in our Splunk namespace:

kubectl -n splunk get pods
NAME                                                           READY STATUS RESTARTS AGE
indexer-0                                                      1/1 Running 0 2h
indexer-1                                                      1/1 Running 0 2h
indexer-2                                                      1/1 Running 0 2h
kubecon-demo-2018-splunk-kubernetes-logging-f6n2z              1/1 Running 0 17s
kubecon-demo-2018-splunk-kubernetes-logging-hqrlb              1/1 Running 0 17s
kubecon-demo-2018-splunk-kubernetes-logging-rv78f              1/1 Running 0 17s
kubecon-demo-2018-splunk-kubernetes-metrics-5695b46d58-6fgb4   2/2 Running 0 17s
kubecon-demo-2018-splunk-kubernetes-objects-7f7f47f6f-qkkzg    1/1 Running 0 17s
master-6d7b98f8f5-nnl9n                                        1/1 Running 0 2h
search-5944fc8696-hpj5m                                        1/1 Running 0 2h
splunk-defaults-686b5885f6-k7crh                               1/1 Running 0 6h
tiller-deploy-79f76f8b55-96tq4                                 1/1 Running 0 11m

You can also check the pod logs to ensure there are no errors:

kubectl -n splunk logs -f kubecon-demo-2018-splunk-kubernetes-logging-hnghs
kubectl -n splunk logs -f kubecon-demo-2018-splunk-kubernetes-objects-7f7f47f6f-qkkzg
kubectl -n splunk logs -f kubecon-demo-2018-splunk-kubernetes-metrics-5695b46d58-6fgb4 -c splunk-heapster

If you run into any errors, let us know on the github repo or come join us in the #kubernetes room on Slack ( Especially in newer Kubernetes deploys, you may need to patch the metrics deployment and clusterRoleBinding. See this github issue for a workaround.

Otherwise, assuming all went well, we should be able to go back into the master and check that we see our new indexes:

kubectl -n splunk port-forward master-6d7b98f8f5-nnl9n 9999:8000


Next, let’s port-forward into our search pod and make sure we can search our Kubernetes data. If you have

kubectl -n splunk port-forward search-5944fc8696-hpj5m 9999:8000

Run the following searches and ensure you see results. If you can’t see logs, metrics or metadata, check your configs or come join us in the #kubernetes room on Slack.

Search 1: index=cm_events
Search 2: index=cm_meta
Search 3: | mcatalog values(metric_name) WHERE index=cm_metrics by host

It’s alive!

Now you can install your favorite apps on the standalone Search pod, right through the GUI. From the Splunk Enterprise homepage, press the gear icon in the Apps bar:

Then select Install app from file:

We’ll download Splunk Metrics Workspace and install using the method above to enhance your ability to search the Kubernetes metrics you’re getting from your cluster.

As long as your metrics are flowing in, you should see Splunk has discovered your metrics and you should be able to begin exploring!

For those that have registered for the Insights for Containers Splunk for Container Monitoring (BETA) and received the app, we’ll install it via the Search Pod’s GUI.

Navigating to the App for Containers BETA after install, use the Investigate tab to view and filter down dimensions, i.e. namespace and entity_type and create a group called splunk-pods.

Now, anytime I want to investigate the pods in our splunk deployment from Part 1 & 2 of this series, I can view them as a group, with all metrics and logs and alerts available in one place.

Quite a powerful way to correlate resources across datasets and get started with Kubernetes data! From metrics, to application logging to API auditing and more, there are many insights to be gleaned when your data is all in one place.

Time to take a step back and look at what we’ve built!

kubectl -n splunk get pods 
NAME                                                          READY STATUS RESTARTS AGE
indexer-0                                                     1/1 Running 0 3h
indexer-1                                                     1/1 Running 0 3h
indexer-2                                                     1/1 Running 0 3h
kubecon-demo-2018-splunk-kubernetes-logging-hnghs             1/1 Running 0 6m
kubecon-demo-2018-splunk-kubernetes-logging-p5s8q             1/1 Running 0 6m
kubecon-demo-2018-splunk-kubernetes-logging-xkg57             1/1 Running 0 6m
kubecon-demo-2018-splunk-kubernetes-metrics-dfd4c76c9-946r5   2/2 Running 0 6m
kubecon-demo-2018-splunk-kubernetes-objects-67d8dc7c7-92rgt   1/1 Running 0 6m
master-6d7b98f8f5-nnl9n                                       1/1 Running 0 3h
search-5944fc8696-hpj5m                                       1/1 Running 0 3h
splunk-defaults-686b5885f6-k7crh                              1/1 Running 0 6h
tiller-deploy-79f76f8b55-96tq4                                1/1 Running 0 28m

You’re now the captain of a fully self-contained Splunk demo environment, consisting Splunk Indexing Cluster deployed via Kubernetes, indexing all your Splunk Connect for Kubernetes logs, metadata and metrics in Splunk SmartStore enabled indexes, all being searched in one place in some of our newest Splunk apps!

Awesome, right?

We’re so excited to see where our passionate user community will go with Splunk & Kubernetes!

Be sure to join the conversation and share your journey on our Slack chat by registering at in #kubernetes

Contribute with us on github!

Thanks again for checking out our test scenarios, and for more on monitoring your Kubernetes stack learn more with our Beginner’s Guide to Kubernetes Monitoring.

And if you’re interested in gaining immediate insight into your Kubernetes stack including performance metrics for your clusters, pods, containers, and namespaces, as well as log, metric, event, and metadata, sign up for Splunk Insights for Containers (BETA) to test this out. Please inquire if you might be a good candidate for this beta program. You must have a Kubernetes deployment. Existing and non-existing Splunk customers are welcome!

If you are ready to clean it all up, see the following steps to tear down your resources.

Clean Up

Once you’re done exploring, you can clean up all your resources with the following commands:

Remove the Splunk Connect for Kubernetes helm deploy

helm delete kubecon-demo-2018 --tiller-namespace splunk --purge

Remove Tiller

helm reset --tiller-namespace splunk

Delete the HEC service:

kubectl -n splunk delete service hec

Delete the Splunk Cluster

Navigate to the docker-splunk/test_scenarios/kubernetes directory and run:

kubectl -n splunk delete -f 3idx1sh1cm

Delete nginx

Navigate to docker-splunk/test_scenarios/kubernetes/nginx directory and run:

kubectl -n splunk delete -f manifests

Delete your configmaps:

kubectl -n splunk delete cm --all

Be sure to return to AWS/your storage provider and delete your S3 bucket when complete!

Matthew Modestino

Posted by


Show All Tags
Show Less Tags