Securely Storing & Accessing Passwords For Alert Action Scripts

Updated Oct. 22, 2021: Please note that this blog post is deprecated. For an updated blog post on secret storage, see Store Encrypted Secrets in a Splunk App.

I recently helped a customer securely store and access credentials for an alert action script in Splunk Cloud and wanted to share the details. Ledion Bitincka wrote a great article about storing encrypted credentials using the storage/passwords REST endpoint and accessing them in scripted inputs. This tactic is just a slight tweak on the same foundation.


This example gives you a base template to use within a shell script. You can easily adapt the methods to the language of your choice. Ledion actually gives some sample code for accessing and using the stored credentials using Python in his article.

Create Bare Bones App

Create a barebones app from the UI for this to live in. For this example we’ll call the app ‘cloud_alert_auth’. By default, app.conf will have ‘is_configured=0’ set so this will force your users to add a password the first time they use the app.

Create setup.xml

Copy the following contents into $SPLUNK_HOME/etc/apps/cloud_alert_auth/default/setup.xml

Create Alert Script

1) Create the bin/scripts directory within your app.

mkdir -p $SPLUNK_HOME/etc/apps/cloud_alert_auth/bin/scripts

2) Create an alert script in the above directory and use the following code as the base for your alert action script.

The key is to read the sessionKey from STDIN, url decode it using Splunk’s Python, call the REST storage/passwords endpoint for the CREDENTIAL_USER and then parse the clear_password. This will give you the variable $clear_password to use in the subsequent commands that require authentication within your script.

** Note that you do not need to define a ‘realm’ when creating the password, but if it is specified, the CREDENTIAL_REALM variable will need to be set accordingly.

3) Create your search/alert within the app context pointing to this script. You’ll need to create the search as a user with admin privileges or the admin_all_objects RBAC capability set, otherwise you won’t be able to access the storage/passwords REST endpoint. Make sure the search runs as ‘owner’ (it does by default).

If you need to access multiple credentials within the script you can modify it to process an array of users and then loop through them using standard shell scripting techniques.

Password Management

Passwords are stored within the app in local/passwords.conf. Here’s an example.

password = $1$+g7Chwf7xgyt7w==

If you just need to change a password just delete the entry in local/passwords.conf and he can re-enter without a restart.

Additional Considerations

If you have multiple search heads you’ll need to create the credentials on every search head since the encryption key differs on each of them.

Download The Sample App

Download a sample app from GitHub that has all the code needed and a sample alert that writes your password to /tmp/password.txt

Scott Haskell

Posted by