Announcing Splunk on Docker

I am happy to announce the official support for the Splunk Docker image for Splunk Enterprise 7.2 on DockerHub. Since 2016 when we posted our first community-supported Docker image, we've seen an overwhelming response from the Splunk community to deploy Splunk on containerized infrastructures. Today, we are taking this a step further by providing an officially supported Docker image.

What Are Containers and Why is the Industry So Excited About It?

A container is a lightweight software package that bundles application code along with the runtime, tools, system libraries and environment settings required to execute it. This enables an application to be abstracted from the environment that they are executed in and allows it to be isolated from other applications. Containers are extremely portable and can run on a variety of operating systems, hypervisors and bare metal at massive scale. Containers can also be spun up and down quickly.

With applications deployed in containers, IT operations teams are more agile, have more flexibility in their deployments and can potentially reduce their total cost of ownership. As opposed to hypervisors that abstract the hardware but still require the virtual machines to have their own guest operating system, the container runtime enables applications to share the same kernel.

^{Source: https://cloud.google.com/containers/}

Docker is an open-source container runtime and packaging standard that has been widely adopted by the industry. See the figure below for details.

^{Source: https://blog.docker.com/2018/03/5-years-later-docker-journey/}

The container ecosystem has realized the following benefits of containers:

• Low TCO - Well-designed containerized deployments leads to a smaller operating system, hypervisor and hardware footprint. Containerized applications provides better system utilization as multiple applications can be deployed on a single host.
• Faster time to value - Containerized deployments are order of magnitude faster to deploy than hypervisor-based deployments. Applications can be deployed and upgraded quickly which enables end users to get value out of their applications faster.
• Increased agility and flexibility - With containerized applications, enterprises can quickly expand the footprint of Splunk and onboard new teams. Enterprises will gain more flexibility because containers can be easily brought up and down based on their needs. Containers are well-aligned with the paradigm of microservices based software development where a complex application is decomposed into loosely coupled micro-services that can be developed and deployed independently, leading to greater agility in the enterprise.

To gauge the interest level on containers in the Splunk community we posted the first community supported Splunk image on DockerHub in 2016. As of the writing of this blog post, we have seen more than 2 million downloads of the Splunk Enterprise image and more than 5 million downloads of the Splunk Universal Forwarder image. This overwhelming response from the Splunk community prompted us to move forward with officially supporting the Splunk Docker image for Splunk Enterprise 7.2 and provide best practices for deploying Splunk on containerized infrastructures.

What is Container Orchestration?

Container orchestration refers to the automated deployment, scaling and management of containerized applications. Real-world deployments of containerized applications are managed through a container orchestration platform like Kubernetes, Docker Swarm, Mesos and Rancher.

The Splunk Docker image is orchestrator agnostic. Our customers are currently testing out deployments with all four of the major container orchestration platforms. The orchestration platform selected by the customer needs to support Docker Engine 17.06.02 or later.

To help advanced customers plan their production deployments, we are also providing examples targeted for Kubernetes users in the Splunk Github repository.

What Can Customers Do With the Officially Supported Splunk Docker Image?

Depending on the customer expertise with container technologies and Splunk, there are a range of options that customers can explore with the Splunk Docker image that is aligned with their organizations needs. Qualitative guidance based on our experience interviewing a wide range of customers—including a handful who are running production Splunk deployments on containers—is shown below.

Customers with experience managing production deployments with containerized applications and an advanced knowledge of Splunk can plan out production deployments with the Splunk Docker image. Customers who have advanced knowledge of containers as well as Splunk can start with building out Splunk in a dev/test environment. For other customers who are new to Splunk and the world of containers, we recommend the path of prototyping.

While planning any deployment of Splunk, we recommend using the Splunk Validated Architectures; these deployment blueprints guide customers on selecting the right deployment architecture for Splunk. To start with, we will officially support the S1 (single server) architecture. Examples and guidance are provided for the D1/C1/C3 architectures. We have internally validated these architectures but official support will be provided in a follow-on release.

Datasheet

A summary of what the Splunk Docker image supports today is shown below.

Getting Started

Once you have installed the Docker Engine, a single instance of Splunk can be run using the following steps.

To get the latest docker image from DockerHub, enter the following on the command line:

docker pull splunk/splunk:latest

To start a single instance of Splunk, enter:

docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=<password>' splunk/splunk:latest

The above command publishes the container's port 8000 on which the Splunk web service runs to port 8000 of the localhost.

To check the status of the container, enter:

docker ps
CONTAINER ID        IMAGE COMMAND                          CREATED       STATUS                 PORTS                                                                                     NAMES
d1fecab7b9a1        splunk:latest "/sbin/entrypoint.sh…"   2 minutes ago Up 2 minutes (healthy) 4001/tcp, 8065/tcp, 8088-8089/tcp, 8191/tcp, 9887/tcp, 9997/tcp, 0.0.0.0:8000->8000/tcp   stoic_sammet

To access Splunk web enter http://localhost:8000/ in your browser and login to Splunk with the password that was passed at the command line.

To stop the instance of the container, enter:

docker stop <CONTAINER ID>

For Advanced Users

The Splunk Github repo has some examples of deploying a Splunk cluster on Kubernetes. To build out a Splunk cluster with 3 indexers, 1 search head, and 1 cluster master, enter:

kubectl -n splunk apply -f 3idx1sh1cm

To get a status of your Kubernetes pods, enter:

kubectl -n splunk get pods
NAME                      READY STATUS  RESTARTS AGE
indexer-0                 1/1   Running 0        2m
indexer-1                 1/1   Running 0        2m
indexer-2                 1/1   Running 0        2m
master-99cd7fdd9-d5vlc    1/1   Running 0        2m
search-6cb9945dbf-8h66m   1/1   Running 0        2m

The status above indicates that all the cluster components have been started and are active.

To access a particular component of the cluster from your web browser, you can use the Kubernetes port forwarding command.

kubectl -n splunk port-forward master-99cd7fdd9-d5vlc 9999:8000

Now you should be able access the cluster master by entering http://localhost:9999/ in your browser window by using the username admin and the password helloworld.

To scale the cluster to 4 indexers, enter:

kubectl -n splunk scale statefulsets indexer --replicas=4

To see the status of the cluster, enter:

kubectl -n splunk get pods
NAME                      READY STATUS  RESTARTS   AGE
indexer-0                 1/1   Running 0          13m
indexer-1                 1/1   Running 0          13m
indexer-2                 1/1   Running 0          13m
indexer-3                 1/1   Running 0          43s
master-99cd7fdd9-d5vlc    1/1   Running 0          13m
search-6cb9945dbf-8h66m   1/1   Running 0          13m

You can see now that an additional indexer has been added to the cluster. The view in the Splunk UI for the cluster master node changes accordingly:

Splunk on Docker Resources

• Splunk Docker Image on DockerHub
• Repository for building docker images + K8S examples
• Splunk Ansible Repository
• Slack Channel (Channel: #docker)
• Splunk Answers

.conf18 Sessions

You can watch replays of the sessions at .conf18 where we dove into the exciting world of Splunk on Docker below:

• FN1035 - How to Containerize Your Application - An Example of the Splunk Docker Image
  
• FN1089 - Dockerizing Splunk at Scale 2: The Container Strikes Back
  
• IT1647 - A Container Adventure: Scaling and Monitoring Kubernetes Logging Infrastructure
  

A warm welcome to the world of Splunk on Containers! In follow-up blogs, we will get into more detail on the possibilities of using Splunk on Docker.

Happy Splunking!

----------------------------------------------------
Thanks!
Amit Mookerjee