Deploying Splunk Securely with Ansible Config Management – Part 1

Johnny Automation

Automation Johnny


More times than not I have seen corporations struggle with config management and it is key for concise mitigation and remediation plan. Interfacing with a variety of Splunk customers the corporations whom do implement a config management system usually have a different tactic on how to manage Splunk while doing it in a secure fashion. In this series of blog posts which will hopefully walk you through a simple deployment of Ansible all the way to the most complex use-cases I have seen. I will first be covering how Ansible can be leverage to manage a simple Splunk deployment on your own hosts. Part 2 we will cover how this can be done in a larger scale with EC2 utilizing dynamically changing inventory of hosts for deployments whom need to scale in a cloud environment. Finally part 3 will cover how to manage a giant deployment of Splunk with multi-tenecy requirements where there are various “customers” or business units with different Splunk config needs. The idea is to embark the necessary knowledge to not only deploy Splunk but anything else using Ansible as your config management system.

Why Ansible?

There are a few config management system. The most common ones I have seen deployed are Chef and Puppet. In my previous job we evaluated a few of them, including Puppet and Chef and we ended up choosing Ansible. Here are the reasons why I have seen organizations make that choice:

  • No Agent Required – This is awesome! Personally, I think the less agents a system has the less moving parts it has and the smaller your exploit landscape is. Ansible also has the ability to deploy an agent specially on a scenario where your host are pulling configs instead of a master server pushing them.
  • Using SSH as Transport – No need to deal with custom communication protocols, everything is encrypted natively. Keys are used instead of passwords. Also SSH is pretty ubiquitous across the varied linux distributions. Most times than not you are already talking to your servers via SSH.
  • Ease to pickup – I found that the learning curve in Ansible is tremendously smaller any otherconfig management system. This is primarily due to the fact that play books read easily. They are YAML base and the project has great documentation
  • Low overhead and scales to huge deployments – There is no need to run a dedicated Ansible master server; the application has very low resource requirements. Ansible is also shown to scale, I have seen deployments as large as 100,000+ endpoints, twitter has a great video on how they use it to manage their server fleet.
  • Python Base – Do not like something, want to integrate with something else? Ansible is python based and very easily extendable. If you can code in python this is the CMDB for you, they speak your language!
    • ansible-playbook: Ansible executable which runs the playbooks etc..
    • hosts: INI file which contains the role/group and host mapping
    • playbooks: Ties in Roles, host groups and task together to create orchestrated actions on target hosts
    • roles: contains the actions each group will complete (this is where the deployment logic lives).
    • Ansible Installed
    • splunk-admin user updated with your keys under /playbooks/splunk_creds/
    • Root password of hosts to run Ansible in
    • Make sure you have ssh keys generated for root
    • Hosts inventory updated
  • Ansible is not for organizations whom their large server base is windows. Although they are working on Windows clients.

    Installing Ansible

    I suggest you grab Ansible from their stable repo in github instead of your distributions repository. The stable version in github has welcomed updates and additions like ansible-vault which we will cover later in further detail.

    cd /opt/
    git clone

    Now lets install the playbooks and configuration files we will use for deployment.

    cd /etc/
    git clone
    mv ansible-splunk-simple /etc/ansible

    Now you have a copy of the Ansible application under /opt/ansible and a collection of configuration files under /etc/ansible. Below is a logical diagram which represents a high level Ansible application structure.

    Ansible Structure

    diagram Ansible structure
    Explanation of what the different pieces are:

    Roles in details

    Lets walk through the structure of a role in details. I will start with the common role. The common role should be ran no matter what kind of role the host has as it performs common functions that you would want on every host. If we look at the main.yml under tasks for this role we can see all the tasks it performs.

    ansible$cat roles/common/tasks/main.yml
    # This playbook contains common tasks in this role
    - include: apt.yml
    - include: users.yml
    - include: files.yml
    - include: cron.yml
    - include: time.yml

    The role contains 5 play books. Lets walk though one of them and what does it do. Looking at apt.yml in detail:

    ansible$cat roles/common/tasks/apt.yml
    # This playbook install the apps required in a server
    - name: install security controls
    - configuration
    - security
    apt: name={{ item }} state=present
    - chkrootkit
    - rkhunter
    - clamav
    - fail2ban
    - name: install basic utilities
    - configuration
    apt: name={{ item }} state=present
    - vim
    - screen
    - iotop
    - htop
    - ioping
    - ntp

    The description of this is at the top as a comment. Using the “apt” Ansible module (follow the link for information on modules) we install a variety of software on the server. The first batch of software is tagged as “configuration” and “security” and the last are just configurations. The first batch install chkrootkit, rkhunter, clamav, fail2ban in the system and the second batch installs a set of configuration utilities (vim,screen, etc..). This is the typical structure of an Ansible play, which coordinates what happens on the host system. Plays are powered by different kind of Ansible modules, the one used above is apt module. You can see a full list of supported modules in Ansible docs.

    Running Ansible for the First Time


    Before running Ansible make sure that your environment is set correctly. Run. /opt/ansible/hacking/env-setup in order to have the necessary paths and environment activated for Ansible to run. Verify it worked by running
    which ansible-playbook
    This can also be added to bash profile as well.

    To build a splunk server from scratch just run:
    /etc/ansible#ansible-playbook /etc/ansible/playbook/search_heads.yml

    Make sure that you have hosts defined under hosts
    If the host you are running is virgin (nothing has been done to it) you will want to run Ansible with -k so it prompts for the password during the first run. You will only need this once as Ansible has a task under roles common to copy the ssh key of your user to that host.

    You should see a similar output to this:

    /etc/ansible#ansible-playbook playbooks/search_heads.yml
    PLAY [apply common configuration to all nodes] ********************************
    GATHERING FACTS ***************************************************************
    ok: []
    TASK: [common | install security controls] ************************************
    ok: [] => (item=chkrootkit,rkhunter,clamav,fail2ban)
    TASK: [common | install basic utilities] **************************************
    ok: [] => (item=vim,screen,iotop,htop,ioping,ntp)
    TASK: [common | create splunk-admin] ******************************************
    ok: []
    TASK: [common | copy splunk-admin bash_profile] *******************************
    ok: []

    Closing Thoughts

    This should have armed you with the basic tools to do config management and orchestration on Splunk hosts as well as the rest of your infrastructure.

    I encourage you to walk through each roles task to verify what it is doing. Also the github read me will have more information as to how Splunk is configured using the shipped playbooks. Feel free to drop me a line with any questions or if something is not working at jose at splunk.

    Thank you Mike Regan, Mike, Baldwin and the rest of the Splunk Cloud team for your contributions and working with me on the Playbooks.

Jose Hernandez

Posted by