SECURITY

Generating Elliptical Curve Certs for Splunk

Splunk By Splunk June 03, 2014

Intro

Very often we get questions about generating stronger Cert/Keys for Splunk. Specifically from users who run a vulnerability scanner against their Splunk instance. By default, Splunk ships with a 1024 bit strength RSA Cert. This is the same certificate authority cert across all Splunk binaries. Splunk recommends customers use their own CA and cert/key pair in the securing Splunk documentation. It is good practice to sign certs by your own CA.

I recommend an EC (Elliptical Curve) key/pair. For more about EC have a look here. Here are a few pros and cons of using EC certs:

Pros

  • Perfect Forwarding Secrecy (PFS) support
  • Shorter Keys which are as Strong as RSA key but are easier on the CPU
  • Lower Memory Usage

Cons

  • Compatibility – older openssl libraries do not support EC ciphers

If you are running a fairly modern infrastructure and your clients have modern openssl libraries you should not have a problem with compatibility. To check if your client supports EC ciphers:
openssl ciphers -v | grep EC

Generate EC Certs and Keys

Change directory to Splunk home and create a cert folder (I am assuming $SPLUNK_HOME is in /opt/splunk

cd /opt/splunk
mkdir cert

Generate a New Elliptical Curve CA key and Cert

openssl ecparam -out ca-key.pem -genkey -name prime256v1
openssl req -x509 -new -key ca-key.pem -out ca-cert.pem

We will use the CA in a future step to sign Certs and Keys for your splunkd process as well as the web server (splunkweb).

Next we generate a CSR (Certificate Signing Request) in order to sign the CERT/KEYs
Generate Servers Private key and CSR

openssl ecparam -out server-key.pem -genkey -name prime256v1 -noout
openssl req -new -key server-key.pem -out server-csr.pem

Finally using our CSR we generate a Cert. Here we use the CA we previously generated (prior to the CSR):
Generate Public Certificates:

openssl x509 -req -days 365 -in server-csr.pem -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem

The result from the generation command is similar to:

Signature ok
subject=/C=US/ST=California/L=San Fran/O=Internet Widgits Pty Ltd
Getting CA Private Key

Convert cert and key to PEM format

cat server-cert.pem server-key.pem > server.pem

Now you have the following files under /opt/splunk/cert/

/opt/splunk/cert#ll
total 32
drwxr-xr-x 2 root root 4096 Jun 3 14:52 .
drwx------ 10 splunk splunk 4096 Jun 3 14:03 ..
-rw-r--r-- 1 root root 741 Jun 3 14:08 ca-cert.pem
-rw-r--r-- 1 root root 302 Jun 3 14:07 ca-key.pem
-rw-r--r-- 1 root root 587 Jun 3 14:52 server-cert.pem
-rw-r--r-- 1 root root 456 Jun 3 14:52 server-csr.pem
-rw-r--r-- 1 root root 227 Jun 3 14:52 server-key.pem
-rw-r--r-- 1 root root 814 Jun 3 14:53 server.pem

Configs

Below are some example configs with the Certs and Keys that we generated above:

Inputs.conf on the indexer:

/opt/splunk#cat etc/system/local/inputs.conf
[default]
host = acme

[splunktcp-ssl:9997]
sourcetype = authentication

[SSL]
password = $1$cmlMTz8Xe+U=
rootCA = $SPLUNK_HOME/etc/certs/ca-cert.pem
serverCert = $SPLUNK_HOME/etc/certs/server.pem

Example of outputs.conf in a Universal Forwarder:

/opt/splunkforwarder#cat etc/system/local/outputs.conf

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = acme.myindexer.com:9997
sslCertPath = $SPLUNK_HOME/etc/certs/server.pem
sslPassword = $1$whFOKEQgOes=
sslRootCAPath = $SPLUNK_HOME/etc/certs/ca-cert.pem

Testing

To verify the use of EC ciphers:
On the Universal Forwarder:

/opt/splunkforwarder/bin#./splunk cmd openssl s_client -connect acme.myindexer.com:9997
Look for a line with New, TLSv1/SSLv3, Cipher is ECDH-ECDSA-AES256-GCM-SHA384

Troubleshooting

Are your certs correct?
If you want to test your keys. Start a web server with the following openssl command.
openssl s_server -www -key server-key.pem -cert server-cert.pem -CAfile ca-cert.pem -state
Then, try to connect to
openssl s_client -connect 127.0.0.1:4433

From a Universal Forward test that you can connect to the indexer
/opt/splunkforwarder#bin/splunk cmd openssl s_client -connect acme.myindexer.com:9997
Look for connected message in splunkd.log in the universal forwarder:
/opt/splunk/bin$sudo ./splunk search "index=_internal connected"

You now have the steps to deploy, test and troubleshoot your own EC key pair deployment for Splunk. This provides PFS on inter Splunk communication as well as less overhead than the default key pair.

Notes on the “Dual_EC_DRBG”

There has been concerns for using Elliptical Curve certificates due to mentions of it random number generator being backdoor. To put these concerns to rest, openssl actively blocks the use of Dual_EC_DRBG in the “FIPS capable” module. The openssl library we ship with Splunk has FIPs mode set via its config files. Please refer to this article for more details.

Thank you Monzy for helping me release this in a legible state :-). Also thank you Ariel Velasco for bringing some use cases to my attention and Alex Eisen for clarifying the Dual_EC_DRBG concerns.

----------------------------------------------------
Thanks!
Jose Hernandez

Splunk
Posted by

Splunk