By Splunk May 26, 2010
How to configure Splunk to meet your information assurance requirements
In my post the other week, I tried to answer some of our customer’s most common questions about Splunk IT data being used as evidence in a court of law. This week, I hope to tackle some similar issues with regard to Splunk, namely the utility of using Splunk for information assurance.
Information assurance can be defined as the practice of measuring and managing the risks of storing, processing, and transiting digital information. Although forensic and legal requirements must be considered as part of the information assurance process, they are just a component part of the overall practice. Some people think that instead of a “practice”, information assurance might be a form of paranoia or sleep-loss aid, but I’m sticking with “practice” until the results of the studies are conclusive.
The aim of practicing information assurance is to guarantee the following:
- Confidentiality – That unauthorized actors cannot access data.
- Integrity – That data will not be modified or destroyed by unauthorized actors.
- Availability – That access to data and data services will be timely and reliable for authorized actors.
- Authentication – That validity checks will be performed against all actors in order to determine proper authorization.
- Non-repudiation – That the sender of the data is provided with the transmitted data in a way that prevents both the sender and consumer from denying that the transaction occurred.
These aspects are often referred to as the “tenants” or “pillars” of information assurance. To achieve these guarantees, traditional information assurance has prescribed the following cycle:
- Enumeration of assets
- Risk assessment
- Formulation of risk management plan
- Implementation of risk management plan
- Audit and testing of implementation
- Back to step 1…
In a future post, I intend to dive deeper in to the information assurance cycle mentioned above and talk about how Splunk can help with each step in the process. In this post, however, I intend to demonstrate how Splunk can help information assurance teams guarantee the confidentiality, integrity, availability, authentication, and non-repudiation of IT data.
Confidentiality
Strong Encryption
All communication with Splunk Web (TCP port 8000), splunkd (TCP port 8089), and splunkTCP (port 9997) can be encrypted using SSLv3. Additionally, you can specify your required cipher suite for splunkd and splunkTCP connections. Keep in mind that you are going to need to alter Splunk’s default settings to attain this level of confidentiality.
A common and easily correctable mistake that both I and my colleague J-Rod see our customers make far too often is failing to use a different certificate authority and/or root certificate than those which ship with Splunk.
For the Splunk root certificate:
- Follow these instructions to create a unique root certificate.
For the Splunk server certificate:
- Follow these instructions to create a new Splunk server certificate and key pair.
- Follow these instructions to generate a Splunk server certificate signing request (CSR) and submit it to your organization’s certificate authority.
For the Splunk web certificate:
- Follow these instructions to use an alternate certificate for Splunk web.
- Follow these instructions to create a new Splunk web certificate signed by the Splunk server’s root certificate.
Role-Based Access Control
Splunk administrators can create custom roles to control access to Splunk data and features. Access to data can be controlled by “search filters“, while the most sensitive IT data can be routed into discrete indexes based on retention policy, business unit, or clearance. Whether through the use of discrete indexes, search filters, or a combination of the two, a user’s access to the data is dictated by what roles she is placed into by the Splunk administrator. If a given Splunk feature does not provide sufficient confidentiality for an organization, it can be disabled on a per-role basis.
While I am on the subject, please note that there are at least two Splunk features, enabled by default, that may leak data in such a way that complete confidentiality in Splunk might not be attainable if they are not disabled.
Predictive text (get_typeahead capability in authorize.conf) should almost certainly be disabled in the majority of cases where complete confidentiality is required.
Also, the “metadata” command (get_metadata capability in authorize.conf) can leak host, source, and sourcetype information in cases where search filters are used to control data access.
Integrity
SSL
Per the section “Confidentiality” section above, all Splunk communications can be encrypted using SSLv3. In addition to providing confidentiality, SSL authentication provides message integrity. By creating a SHA-1 message digest on the sender and recipient and comparing them, SSL communication has a fairly strong guarantee of integrity.
File System Change Monitoring
Splunk ships with a feature called “File System Change Monitor” (aka fschange), which allows you to audit changes the file systems on your forwarders, indexers, and search heads. This is a useful feature for determining when configuration files or binaries have been changed, which might constitute a violation of integrity.
Before you start using fschange, you should first generate audit keys and configure audit.conf to make Splunk use these keys for cryptographically sign your audit events. Then, you must set signedaudit=true in any fschange stanzas you choose to define in inputs.conf.
IT Data Signing
Splunk can perform block signing on any index. The process works as follows for any index where blockSignSize has been set to a value greater than “0” in indexes.conf:
- A hash is generated for each individual event.
- The events are grouped into blocks of a size you specify via blockSignSize.
- A digital signature is generated and applied to each block of events.
Thus, if an attacker were to add/modify/delete event delete some events from the original log file, the event hashes and block signature can be compared to the hashes and signatures that Splunk created when it initially indexed the data. Also, if IT data in one of your signed indexes is tampered with, Splunk web will indicate that the block signature discrepancy indicates gaps or tampering.
Often, people question the utility of block signing. If an attacker has write access to the Splunk’s indexes, the game is already over. Any tampering to one of the signed indexes could be covered up by tampering with the _blocksigning index. Nonetheless, the feature is still useful for detecting alterations to the data at the original source or repository.
Availability
Data Cloning
Splunk provides the ability to send data from a Splunk forwarder to more than one Splunk indexer in order to provide data availability should one Splunk indexer become unavailable. Data cloning ensures that one or more Splunk or operating system failures will not prevent you from searching your Splunk data – that is the time when you really need Splunk!
Authentication
Splunk Native
Splunk Native is the built-in authentication method provided by Splunk. Similar to native Unix authentication, Splunk Native provides basic user management via the $SPLUNK_HOME/etc/passwd file.
In general, Splunk Native authentication is not a wise choice for use in environments with strict information assurance requirements. Splunk Native authentication does not provide standard security features such as:
- Account lock/unlock thresholds for failed password attempts
- Password age requirements
- Password complexity requirements
As such, if you plan on using Splunk Native authentication, make sure that you are carefully monitoring the Splunk audit log for brute force attacks:
05-26-2010 08:44:32.275 INFO AuditLogger - Audit:[timestamp=05-26-2010 08:44:32.275, user=admin, action=login attempt, info=failed][n/a] 05-26-2010 08:45:45.848 INFO AuditLogger - Audit:[timestamp=05-26-2010 08:45:45.848, user=admin, action=login attempt, info=failed][n/a] 05-26-2010 08:45:50.061 INFO AuditLogger - Audit:[timestamp=05-26-2010 08:45:50.061, user=admin, action=login attempt, info=failed][n/a] 05-26-2010 08:45:57.150 INFO AuditLogger - Audit:[timestamp=05-26-2010 08:45:57.150, user=admin, action=login attempt, info=failed][n/a] 05-26-2010 08:46:05.853 INFO AuditLogger - Audit:[timestamp=05-26-2010 08:46:05.853, user=admin, action=login attempt, info=failed][n/a]
A saved search such as the following will find any account that has incurred at least 5 failed logins during the last 15 minute interval:
index="_audit" action="login attempt" info=failed earliest=-15m | stats count by user | search count>4
Splunk LDAP
In most cases, Splunk LDAP authentication is a better choice than Splunk Native authentication. Splunk LDAP allows you to tie Splunk authentication to your existing LDAP implementation. The advantages of this approach are as follows:
- Centralizes user and group management
- Provides secure authentication features that are absent from Splunk Native authentication
- Provides more robust auditing of authentication activity
There are a few important things to consider if you decide to use Splunk LDAP authentication:
- As of Splunk 4.1, Splunk Native and Splunk LDAP operate side-by-side
- Thus, you should closely monitor the $SPLUNK_HOME/etc/passwd file for unauthorized changes
- By default, Splunk LDAP uses an unencrypted data channel
- Thus, you should strongly consider configuring Splunk LDAP to use Secure LDAP (LDAP-s)
- Splunk LDAP does not offer fail-over between LDAP servers
- Thus, you should utilize load balancers or other third-party solutions to ensure that Splunk LDAP authentication is not interrupted by one server’s lack of availability
- Splunk LDAP does not support referral chasing or chaining
Splunk Scripted
Splunk also provides a scripted authentication method that allows Splunk administrators to tie Splunk authentication to a system other than Native or LDAP. This method, called appropriately enough scripted authentication, is most commonly used by customers that desire to wire Splunk authentication to Pluggable Authentication Module (PAM), Network Information Service (NIS), and/or Remote Authentication Dial In User Service (RADIUS).
In general, this should be a method of last resort, and given that any authentication script will be accepting potentially tainted user input, customers are urged to take due care in designing and implementing Splunk scripted authentication.
Certificate-Based Authentication
Communication between Splunk forwarders and indexers can be configured to use SSL authentication. Similarly, distributed search communication depends on certificate-based authentication in order to function. In the future, the use of certificate-based authentication will be extended to other Splunk communication channels.
NB: The default Splunk root certificate (which can be found in $SPLUNK_HOME/etc/auth/ca.pem) uses a private key that every other user of Splunk in the world has access to. Possession of a certificate authority’s private key will allow attackers to generate certificates that are signed by the trusted authority, which would defeat attempts to control authentication via PKI. See the above section “Confidientiality” for information on replacing the default certificates.
Non-Repudiation
Certificate-Based Authentication
As mentioned above, certificate-based authentication is used between forwarders and indexers as well as between distributed search peers.
These methods, in addition to providing some guarantee of confidentiality, integrity, and authentication, also provide reasonable confidence of non-repudiation with regard to proof-of-origin. Specifically, the messages cannot be received at the recipient side without the inherent proof-of-origin provided by certificate-based authentication.
With regard to non-repudiation with proof-of-receipt, this problem is unfortunately not solved by cryptography. The use of a non-repudiation service would be required to provide fair proof-of-receipt.
Conclusion
I hope that I have been helpful in identifying the features of Splunk that pertain to your information assurance requirements as well as how to configure most of them. I also hope that I have helped to identify areas where Splunk could stand to be improved so that you can implement the necessary mitigations.
On that note, please keep in mind that we want need to hear from you about what features we need to add or how current features could be improved to meet your standards for confidentiality, integrity, availability, authentication, and non-repudiation.
If you have a good idea for a security feature that a future version of Splunk just has to have, please comment on this post, or contact our support team via our support portal (choose “Enhancement requests” under Priority). Please give us your feedback so that we can make sure we build in the security and assurance features that you want!
----------------------------------------------------
Thanks!
Alex Raitz