

Cryptography Bob
Intro
Very often we get questions about generating stronger Cert/Keys for Splunk. Specifically from users who run a vulnerability scanner against their Splunk instance. By default, Splunk ships with a 1024 bit strength RSA Cert. This is the same certificate authority cert across all Splunk binaries. Splunk recommends customers use their own CA and cert/key pair in the securing Splunk documentation. It is good practice to sign certs by your own CA.
I recommend an EC (Elliptical Curve) key/pair. For more about EC have a look here. Here are a few pros and cons of using EC certs:
Pros
- Perfect Forwarding Secrecy (PFS) support
- Shorter Keys which are as Strong as RSA key but are easier on the CPU
- Lower Memory Usage
Cons
- Compatibility – older openssl libraries do not support EC ciphers
If you are running a fairly modern infrastructure and your clients have modern openssl libraries you should not have a problem with compatibility. To check if your client supports EC ciphers:
openssl ciphers -v | grep EC
Generate EC Certs and Keys
Change directory to Splunk home and create a cert folder (I am assuming $SPLUNK_HOME is in /opt/splunk
cd /opt/splunk
mkdir cert
Generate a New Elliptical Curve CA key and Cert
openssl ecparam -out ca-key.pem -genkey -name prime256v1
openssl req -x509 -new -key ca-key.pem -out ca-cert.pem
We will use the CA in a future step to sign Certs and Keys for your splunkd process as well as the web server (splunkweb).
Next we generate a CSR (Certificate Signing Request) in order to sign the CERT/KEYs
Generate Servers Private key and CSR
openssl ecparam -out server-key.pem -genkey -name prime256v1 -noout
openssl req -new -key server-key.pem -out server-csr.pem
Finally using our CSR we generate a Cert. Here we use the CA we previously generated (prior to the CSR):
Generate Public Certificates:
openssl x509 -req -days 365 -in server-csr.pem -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
The result from the generation command is similar to:
Signature ok
subject=/C=US/ST=California/L=San Fran/O=Internet Widgits Pty Ltd
Getting CA Private Key
Convert cert and key to PEM format
cat server-cert.pem server-key.pem > server.pem
Now you have the following files under /opt/splunk/cert/
/opt/splunk/cert#ll
total 32
drwxr-xr-x 2 root root 4096 Jun 3 14:52 .
drwx------ 10 splunk splunk 4096 Jun 3 14:03 ..
-rw-r--r-- 1 root root 741 Jun 3 14:08 ca-cert.pem
-rw-r--r-- 1 root root 302 Jun 3 14:07 ca-key.pem
-rw-r--r-- 1 root root 587 Jun 3 14:52 server-cert.pem
-rw-r--r-- 1 root root 456 Jun 3 14:52 server-csr.pem
-rw-r--r-- 1 root root 227 Jun 3 14:52 server-key.pem
-rw-r--r-- 1 root root 814 Jun 3 14:53 server.pem
Configs
Below are some example configs with the Certs and Keys that we generated above:
Inputs.conf on the indexer:
/opt/splunk#cat etc/system/local/inputs.conf
[default]
host = acme
[splunktcp-ssl:9997]
sourcetype = authentication
[SSL]
password = $1$cmlMTz8Xe+U=
rootCA = $SPLUNK_HOME/etc/certs/ca-cert.pem
serverCert = $SPLUNK_HOME/etc/certs/server.pem
Example of outputs.conf in a Universal Forwarder:
/opt/splunkforwarder#cat etc/system/local/outputs.conf
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = acme.myindexer.com:9997
sslCertPath = $SPLUNK_HOME/etc/certs/server.pem
sslPassword = $1$whFOKEQgOes=
sslRootCAPath = $SPLUNK_HOME/etc/certs/ca-cert.pem
Testing
To verify the use of EC ciphers:
On the Universal Forwarder:
/opt/splunkforwarder/bin#./splunk cmd openssl s_client -connect acme.myindexer.com:9997
Look for a line with New, TLSv1/SSLv3, Cipher is ECDH-ECDSA-AES256-GCM-SHA384
Troubleshooting
Are your certs correct?
If you want to test your keys. Start a web server with the following openssl command.
openssl s_server -www -key server-key.pem -cert server-cert.pem -CAfile ca-cert.pem -state
Then, try to connect to
openssl s_client -connect 127.0.0.1:4433
From a Universal Forward test that you can connect to the indexer
/opt/splunkforwarder#bin/splunk cmd openssl s_client -connect acme.myindexer.com:9997
Look for connected message in splunkd.log in the universal forwarder:
/opt/splunk/bin$sudo ./splunk search "index=_internal connected"
You now have the steps to deploy, test and troubleshoot your own EC key pair deployment for Splunk. This provides PFS on inter Splunk communication as well as less overhead than the default key pair.
Notes on the “Dual_EC_DRBG”
There has been concerns for using Elliptical Curve certificates due to mentions of it random number generator being backdoor. To put these concerns to rest, openssl actively blocks the use of Dual_EC_DRBG in the “FIPS capable” module. The openssl library we ship with Splunk has FIPs mode set via its config files. Please refer to this article for more details.
Thank you Monzy for helping me release this in a legible state :-). Also thank you Ariel Velasco for bringing some use cases to my attention and Alex Eisen for clarifying the Dual_EC_DRBG concerns.
----------------------------------------------------
Thanks!
Jose Hernandez