How the deployment server works

Please note: If you want an overview of deployment, please read the configuring a Splunk deployment section first.

A deployment server is an instance of Splunk that acts as a centralized configuration manager, allowing any number of Splunk servers to be grouped together and managed collectively. Any Splunk instance can be a deployment server, including those that are also indexing data. There can be multiple deployment servers servicing hundreds or thousands of client servers.

Each deployment server must be configured to handle one or more unique server classes. These classes share common configuration definitions and can be managed as a unit. A Splunk instance may be a member of multiple server classes at once.

Deployment servers can deploy bundles as well, as long as the deployment client can handle dynamic configuration when the bundle it is interested in changes. This could include LDAP mapping information, user access controls, etc.

Please note: currently, scripted inputs do not get bundled in the deployment server. In the future, Splunk will support this behavior. For now, please use your preferred configuration automation tool to push your script directory to your server classes.

Communication between deployment server and client

Splunk servers that are remotely configured by deployment servers are called deployment clients. A Splunk instance can be both a deployment server and client at the same time. In general, the system implements the pull model of deployment, where each deployment client pulls its configuration from a deployment server when it senses that there is a difference between the configuration that it currently has and the one on the deployment server.

There are two different ways clients can sense configuration changes.

UDP Multicast. The deployment server can be configured to send multicasts of a set of doubles containing the server class name and a configuration bundle's CRC for each server class that it is in charge of deploying. Deployment clients can be configured to listen to the IP and port that the deployment server is multicasting on. They can also specify the interface if needed, otherwise the kernel will select one as the default.
Deployment Client Polling. The deployment client can be configured to periodically poll the deployment server, asking it for the server class/CRC pairs that the client is a member of.

Deployment clients that are configured to use multicast do not need any additional configuration parameters to handle multiple deployment servers, but deployment clients that use polling need to be configured with the URI of each deployment server.

Detecting configuration changes

When a deployment client is first run, it looks in its configuration directory and calculates a CRC for each server class configuration bundle that it finds. This pair of server class name and CRC is stored in memory for purposes of later comparison.

Every deployment client automatically becomes a member of 2 server classes, default and its hostname. In the case of 'default', this allows the administrator to target all deployment clients without having to specifically set a name for them. The 'hostname' server class is determined at deployment client startup time by asking for the fully qualified host name of the machine from the DNS server. If the word 'localhost' appears anywhere in the name, the host name class is not set for this server. This server class allows for deploying bundles to individual machines if desired.

If the deployment client senses a configuration change for a server class that it's a member of, it sleeps a random number of seconds between 0 and 30 and then asks the deployment server for a tarball copy of the configuration bundle (using SSL/SOAP on the deployment server's management port). Note that random sleeping is used so that the deployment server doesn't get overwhelmed with too many concurrent requests when a large deployment change has taken place.

In version 3.0 Beta, once the tarball is received, the client moves the previous bundle into a sub directory and then restarts the server.

In 3.0GA and beyond, once the bundle tarball is received, the client moves the previous bundle into a sub directory and then kicks its Splunk instance. This action causes all processors and modules which support the bundle to reload their configuration information without restarting splunkd. If the configuration change requires a system restart, the system will be restarted. If the configuration fetch and kick procedure completed without issue and a restart did not happen, an event is generated to that effect. This event will be indexed by whatever indexer the deployment client is sending data to. In this way, a system administrator can use Splunk to see if all deployments worked properly at any point in time. If the configuration fetch and kick procedure did not work properly, the previous configuration bundle(s) are restored and an event detailing this is sent to the indexer.

How the deployment server keeps track of its clients

The deployment server keeps track of which bundle configuration each client is currently running and which server classes each client is a member of. Because the deployment server uses the pull model, each client must tell the server which version it's using.

Here is a step-by-step description of the process:

The deployment server boots up:
- it sets a 'phone home' flag in the multicast packet for 10 multicast sessions (multiple times because UDP multicast is a non-reliable transport).
Any client that receives the packet containing the flag sends back to the deployment server:
- the timestamp of each class bundle.
- a list of server classes that it's a member of.

A client that just sent its info back to the server will not respond to any more of these packets unless the server explicitly sets the flag again, the server is restarted, or the client is restarted. If a client is not listening to multicast because it's not configured to do so, every poll sent to the deployment server contains the same information. If a client comes up and the multicast packet doesn't include the 'phone home' flag, the very first time it hears any multicast packet, it sends its info back to the server. This behavior ensures that newly joined clients are kept track of as well.

If the deployment server has been configured to assign classes to a given host or IP address specifier, the response to the 'phone home' call will include this list of classes. Upon receiving this list, the client will restart itself with the new classes.

Configuration files for deployment server

Both deployment servers and clients are configured in deployment.conf.

Previous: Exclude specific Splunk servers from distributed searches | Next: Configure a Splunk Deployment Server

Comments

No comments have been submitted.