By Splunk June 04, 2015
As a customer of Splunk I used HAProxy as a software load balancer to distribute users amongst my search heads. I was using the old search head pooling technology at the time, but the same principal holds true for our search head clustering feature; both require a load balancer to distribute users to your search heads. At the time, I couldn’t quite get HAProxy configured to allow use of the REST API. I now believe that was a function of the fact that I was on the 1.4.x branch which didn’t support SSL proxying.
Late last year I had a customer who used our professional services to help with a project. It revolved around using our SDK’s and REST API to surface some data to their customers. The customer was also my previous employer and they were still using the HA Proxy server that I had originally configured. Since the project required proxying of REST API and SDK calls, an upgrade and re-config of HAProxy was in order. Luckily, HAProxy had released the 1.5.x branch which supports SSL proxying.
Here are the steps for configuring HAProxy to proxy users to the default web port 8000 and SSL REST API/SDK requests to the default management port 8089.
1. Downloaded HAProxy 1.5.12 (latest version as of this post)
2. Verify that gcc, openssl-dev, zlib-dev and pcre-dev are installed with yum/dpkg or whatever your Linux distro package manager is.
3. Build and install HAProxy
$ tar -zxf haproxy-1.5.12.tar.gz $ cd haproxy-1.5.12 $ make TARGET=linux26 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 $ sudo make install $ ln -s /usr/local/sbin/haproxy /usr/sbin/
4. Install attached init script to /etc/init.d/haproxy (RHEL/CentOS compatible)
5. Create directory (if it doesn’t already exist) and create blank config file.
# mkdir /etc/haproxy # touch /etc/haproxy/haproxy.cfg
6. Copy your Splunk SSL certificate to the HAProxy config dir. If you’re using the default certificate that ships with splunk then you’d copy $SPLUNK_HOME/etc/auth/server.pem and place that on your HAProxy server in /etc/haproxy/cert.pem. I’d highly encourage you to generate your own SSL certificate and use this in place of the default certificate. Here are the docs for configuring Splunk to use your own SSL certicicate.
7. Modify the following config by substituting your Splunk search head server names for the highlighted servers. Paste the contents in /etc/haproxy/haproxy.cfg. Feel free to tune any of the timeouts or maxconn in the global and defaults section. I’m by no means an HAProxy expert. To add more servers update the frontend and backend sections adding your search heads and giving them a unique name; e.g. ‘server s3’ for the third search head.
global
maxconn 2048
tune.ssl.default-dh-param 2048
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:8000
reqadd X-Forwarded-Proto:\ http
default_backend backend-splunk-http
frontend https-in
bind *:8089 ssl crt ./cert.pem
reqadd X-Forwarded-Proto:\ https
default_backend backend-splunk-https
backend backend-splunk-https
# Use load balancer session cookie persistence
balance roundrobin
cookie SERVERID insert indirect nocache
server s1 splunk-server-1:8089 ssl verify none check cookie s1
server s2 splunk-server-2:8089 ssl verify none check cookie s2
backend backend-splunk-http
# Use load balancer session cookie persistence
balance roundrobin
cookie SERVERID insert indirect nocache
server s1 splunk-server-1:8000 check cookie s1
server s2 splunk-server-2:8000 check cookie s2
8. Start HAProxy and supply the certificate password of ‘password’ if you are using the default SSL certificate that ships with Splunk.
# /etc/init.d/haproxy start
9. Test your setup. Substitute your HAProxy server in the highlighted section
HTTP
http://haproxy-server:8000
REST API/SDK
curl -k -u admin:changeme https://haproxy-server:8089/services/licenser/slaves
You should now have a working HAProxy software load balancer that is compatible with the REST API and SDK’s.
----------------------------------------------------
Thanks!
Scott Haskell