DEVOPS

Manage Splunk On-Call Using Terraform

HashiCorp’s Terraform has emerged as a powerful tool for managing infrastructure as code. Teams can fully describe an application’s infrastructure needs such as physical machines, VMs, containers and more using configuration files. This allows the application infrastructure to be version controlled, reducing human errors during deployments.

Splunk On-Call and Terraform

Terraform modules and providers are extensions that allow configuring applications that a service depends on. For example monitoring tools, on-call, communications etc.

Splunk On-Call (formerly known as VictorOps) empower teams by routing alerts to the right people for fast collaboration and issue resolution.

Using the HashiCorp verified Splunk On-Call Terraform provider, teams can fully automate the Splunk On-Call setup steps associated with an application. A common scenario our customers run into when deploying a new service is to also ensure Splunk On-Call is configured to page appropriate users with alerts related to the service. With Terraform, this can be fully automated in a declarative way, ensuring that the new service is properly monitored with Splunk On-Call from day 1.

In this post, we will use the Splunk On-Call Terraform provider to create the following resources, without logging into Splunk On-Call:

  1. User
  2. Team
  3. Assign User to Team
  4. Escalation Policy
  5. Routing Key
     

Note that you can also manage existing Splunk On-Call resources or destroy them with this provider, but we will focus on resource creation in this post.

Steps to Get Started

If you don’t already have Terraform installed, go through the instructions here.

Once Terraform is installed, verify you are running the latest version by entering the following command in the terminal.

$ terraform version
Terraform v0.13.2

Next, create a new file named splunk_on_call.tf and paste the following in the file: 

# Install VictorOps Terraform Provider
terraform {
 required_providers {
 victorops = {
  source = "splunk/victorops"
  version = "0.1.1"
 }
 }
}

Save the file and type following in the terminal.

$ terraform init

This should download and install the Splunk On-Call Terraform provider automatically, from the Terraform Registry.

The provider communicates with Splunk On-Call using the REST API,so we need to provide it with an API key. Here are the instructions to find your organization’s API key.

Copy the API Id and API Key from Splunk On-Call and paste it in the following format.

provider "victorops" {
  api_id  = "your api id goes here"
  api_key = "your api key goes here"
}

Next, we will start creating Splunk On-Call resources. Paste the following into the file.

# Create a team within victorops
resource "victorops_team" "team" {
    name = "DevOps-Test"
}
# Create a user within the victorops organization
resource "victorops_user" "user1" {
  first_name       = "John"
  last_name        = "Doe"
  user_name        = "john.doe-test"
  email            = "john.doe@example.org"
  is_admin         = "false"
}





# Assign user to a team
resource "victorops_team_membership" "test-membership" {
  team_id          = victorops_team.team.id
  user_name        = victorops_user.user1.user_name
}





# Create email contact method for a user
resource "victorops_contact" "contact_email" {
  user_name    = victorops_user.user1.user_name
  type         = "email"
  value        = "john.doe2@example.org"
  label        = "test email"
}





# Create phone number contact method for a user
resource "victorops_contact" "contact_phone" {
  user_name    = victorops_user.user1.user_name
  type         = "phone"
  value        = "+12345678900"
  label        = "test phone"
}





# Create an Escalation Policy for the team
resource "victorops_escalation_policy" "devops_high_severity" {
  name    = "High Severity"
  team_id = victorops_team.team.id
  step {
    timeout = 0
    entries = [
      {
        type = "user"
        username = "john.doe-test"
      }
    ]
  }
}





# Create a Routing Key that uses the above Escalation Policy
resource "victorops_routing_key" "infrastructure_high_severity" {
  name = "infrastructure-high-severity"
  targets = [victorops_escalation_policy.devops_high_severity.id]
}

The comments within the code above describe what each block does. For more detailed documentation for each of the resources being used, visit our Provider documentation

Save the file and type the following command in the terminal.

$ terraform plan

This should result in a summary of the resources that Terraform will create. 

Next, to execute creation of the resources in Splunk On-Call, type the following:

$ terraform apply

You should see an output similar to terraform plan, this time with a prompt to confirm execution. Do you want to perform these actions?

  Terraform will perform the actions described above.

  Only 'yes' will be accepted to approve.

  Enter a value: 

Type yes to approve and you should see output as follows:

victorops_team.team: Creating...
victorops_user.user1: Creating...
victorops_team.team: Creation complete after 1s [id=team-0mnx4iUkiP2OkqCt]
victorops_escalation_policy.devops_high_severity: Creating...
victorops_user.user1: Creation complete after 2s [id=john.doe-test]
victorops_contact.contact_email: Creating...
victorops_team_membership.test-membership: Creating...
victorops_contact.contact_phone: Creating...
victorops_escalation_policy.devops_high_severity: Creation complete after 1s [id=pol-tRi4Mn8fyGoN6p8b]
victorops_routing_key.infrastructure_high_severity: Creating...
victorops_contact.contact_phone: Creation complete after 1s [id=554c80cf-b6b7-465d-ab9f-0884b41a98fc]
victorops_contact.contact_email: Creation complete after 1s [id=a56f04be-04fa-4edb-a349-1705e1ac5a1c]
victorops_routing_key.infrastructure_high_severity: Creation complete after 1s [id=infrastructure-high-severity]
victorops_team_membership.test-membership: Creation complete after 2s [id=team-0mnx4iUkiP2OkqCt_john.doe-test]





Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Login to Splunk On-Call and click on the Teams tab. You should see a new Team created with the name “DevOps-Test” (or the name you provided for the team in config file). Within the team you should see the User and Escalation Policy as defined in the config file and a Routing Key under Settings --> Routing Key tab.

This was a basic demonstration of what you can do with the new Splunk On-Call Terraform Provider. Check out more detailed documentation here.

To use the Splunk On-Call provider with your team, sign up for Terraform Cloud.

Interested in learning more about Splunk On-Call? Start here.

Feedback

If you have feedback please visit https://ideas.splunk.com/ where you can submit and vote on enhancements to any Splunk product!

Pavan Tummala
Posted by

Pavan Tummala

Pavan is a Product Manager for VictorOps and strives to improve on-call and incident response experience for customers. Prior to Splunk, Pavan was Product Manager for Big Data and Machine Learning based products including IBM Watson.

TAGS

Manage Splunk On-Call Using Terraform

Show All Tags
Show Less Tags

Join the Discussion