By Splunk October 21, 2012
I work a lot with the various people who plan, deploy and support the Splunk App for Active Directory. Some issues come up quite frequently and I thought it would be a good idea to give you a roadmap of things to check as you deploy your environment. I’ll go through the issue and how to check for it so that you can make your roll-out as smooth as possible.
10. Audit Logging is not turned on
A lot of the apps distributed through Splunk Base have no external configuration – just install them on a Splunk instance where the data is being produced and you are done. Splunk App for Active Directory is not one of those. In particular, Domain Controllers don’t produce audit logs by default – you need to turn the audit policies on. If you have installed the technology add-ons (Splunk_TA_windows and TA-DomainController-NT*) on a Splunk Universal Forwarder, installed on each Domain Controller, but you still are not seeing data in the security and change audit dashboards, then this is likely the reason. You can check it by checking that the windows security event logs are producing appropriate events using the search:
eventtype=msad-successful-user-logons
Turning on audit is fairly simple – you need to create a Group Policy Object (GPO) on each domain that configures the audit policies, then apply that GPO to the domain controllers. This activity needs to be done within each and every domain that you run. You can read about this process in our documentation. A new GPO will need to be pushed out using the GPUPDATE command. Once the Group Policy is applied, you will see the data flowing into the Splunk Indexer and this will drive the dashboard creation.
9. Powershell is not enabled
The other data input that requires a little bit of configuration is the health scripts. Active Directory stores a lot of the health information in data structures stored in memory rather than in the directory. It requires that we access .NET libraries to retrieve the information. If you are not seeing the domain selector working, then it is likely that you are not receiving this health data. You can further check this by executing the following search:
eventtype=msad-dc-health
If you log on to a domain controller, you can run the health script manually with the following command:
CD C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\bin
C:\Program Files\SplunkUniversalForwarder\bin\splunk cmd runpowershell.cmd ad-health.ps1
If Powershell is turned off, the error message will tell you that scripts are disabled on this host. You can repair this situation by turning on Powershell within the same GPO you use to alter the audit settings, or you can create a new GPO for this purpose. As with the audit settings GPO, it needs to be attached to the domain controllers on each domain. As with the audit settings, you can read about this process in our documentation.
8. Lookup Tables are not Created
Once you have the data for health flowing into indexer, you need to generate the default domain lookup tables. There are two of them that drive the drop downs and domain selectors that drive the dashboards. These dashboards get generated each night, but you can generate them yourself to expedite the information. If you have information when you do the following search:
eventtype=msad-dc-health
then you have this issue. To generate the lookup tables, you need to run the following searches over that last 24 hours:
`domain-list`|dedup host|outputlookup DomainList.csv
`domain-selector-search`|outputlookup DomainSelector.csv
Once these searches finish (and they should not take a lot of time), your dashboard selectors will start working.
7. Group Information is Not Available in the Security Reports
Active Directory associates groups with the lists of users by listing the users by Distinguished Name (DN). When we retrieve these groups from Active Directory and expand the membership, we need to associate the DN with a Domain. That activity uses something called the RootDSE within Active Directory. The RootDSE is stored in the Global Catalog on the Domain Naming FSMO role, and replicated to all Global Catalogs within the Forest root domain. That’s a mouthful. If you only have one domain, then it’s a Global Catalog on any one of your domain controllers. If you have more than one domain in your forest, then you need to be a little more careful with picking it. You need to configure the chosen server in the ldap.conf file within the SA-ldapsearch app on your search heads. If you don’t, then group expansion does not happen and the dashboards that deal with group expansion (including all the Group reports within the Security Reports) will not bring up data.
To configure the RootDSE server, add a default stanza to the ldap.conf
[default]
server=dc1.domain.com
You don’t need to restart the server when changing the ldap.conf file – changes take effect immediately.
6. Domains Have Three Names
Active Directory provides a storage container, called a domain, for users, groups, computers, and other objects. Some domain names are easy to find – the DNS domain name, for example, is configured when you create the domain and is well-understood. Many security events include the NetBIOS name – a legacy name from the NT5 days but still in use today. However, there is a third name – the distinguished name – which specifies the location within the Active Directory tree that the data for the domain is stored. When Windows produces event logs for the domain, it can use any of the three depending on what it is logging, so we need to know about all three domains. Specify the DNS domain name with all the details of the domain in the ldap.conf file:
[domain.com]
server=dc1.domain.com;dc2.domain.com
port=389
ssl=false
basedn=DC=domain,DC=com
binddn=CN=Splunk,OU=Managed Service Accounts,DC=domain,DC=com
password=changeme
Once you have set up the DNS domain name reference you can specify the other two domain references as aliases:
[DOMAIN]
alias=domain.com
[DC=domain,DC=com]
alias=domain.com
5. Attribute Names in the Base DN are Not Capitalized
We mentioned the chained lookups we do for group expansion earlier. That’s not the only place we use chained lookups – in fact, you can try chained lookups yourself by using the new custom LDAP commands we provide with the SA-ldapsearch app. However, sometimes you need to specify the right capitalization. You will note in the previous issue that we capitalized the Domain Component (DC) attribute – that was very deliberate.
You can check the capitalization by looking at the distinguishedName of a record within the domain. Open up Active Directory Users & Computers and expand the target domain until you get to a user or computer record. Enabled the Advanced View, then right click on the record and select Properties. There will be an Attribute Editor tab – within that tab, find the distinguishedName attribute and take a look. It will tell you what the correct capitalization will be.
4. Using bindas instead of binddn
In the very first version of the Splunk App for Active Directory, we used a file called activedirectory.conf to configure the Active Directory connectivity. This very first version only supported one domain, but in that file you configured a single user for searching with the bindas parameter. When customers upgraded to the multi-domain version (the very next release), we switched to the more normal binddn parameter. Some people cut-and-paste the old value into the new file without changing the parameter name.
This will manifest itself (as with all the top 4 elements) with LDAP searches not working. You can execute the following search:
|ldapsearch domain=DOMAIN search="(cn=Administrator)"
You should get a red bar across the search app that tells you binddn is not set. This is a clear indication that you have not configured the ldap.conf domain record properly.
3. SSL is not enabled within Active Directory
Microsoft can sometimes be a little sneaky, and this is one of those times. In the default configuration of Active Directory, the SSL port (TCP port 636) is enabled, but the server does not accept authentication through this channel until you install a certificate and enable authentication over SSL. Microsoft has a support article (KB 321051) on this very topic.
If you have enabled SSL support within the ldap.conf file and you receive red bars indicating that the system could not establish a secure connection or authentication over the secure channel was not accepted, then try disabling SSL within ldap.conf to see if that fixes the issue.
2. Java SE 1.7 is not installed
Do you have a red bar when running LDAP-related dashboards that tells you an Invalid Version of Java is installed?
Many of our customers run Splunk on a Linux system from one of the major distributions like Redhat, CentOS or Ubuntu. All of these Linux distributions are currently shipping with Java 1.6 installed. Even our Windows customers may have a down-revved Java installation. The custom LDAP commands distributed with the SA-ldapsearch app (and that the Splunk App for Active Directory relies on) require newer features of Java 1.7. As a result, Java 1.7 is a requirement to run the SA-ldapsearch commands. You can easily test what version of Java you have installed by opening up a command prompt are running ‘java -version’. This works on both Windows and Linux, although you may have to specify the path to Java for Windows.
$ java -version
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
If you see 1.7.0 in the first line, then you are good to go. If you have multiple versions of java installed on your system and you need to select the right one, then you may have to alter the scripts that load java, located in the Splunk_for_ActiveDirectory/bin directory. However, most of the time this will not be necessary.
1. The Wrong Password
Yes, the #1 issue for Splunk for Active Directory is the age-old problem of the correct password. Honestly, it’s remarkable the number of people who swear the password is right, then reset the password to something simple for a test and the whole system works. If you have seen a red bar stating a DSID error code of 525 or 52E, then this is your problem. A good test is to just reset the password, store it in clear text in the ldap.conf file and use cut-and-paste to set the password on the user being used.
The Support Process
So how do I go about supporting the Splunk app for Active Directory with individual customers? Once I see which dashboard is not working, that generally directs me to a line of investigation. A lot of the issues are manifest by the red error bars that direct me to a line of inquiry. I start at the top and take Splunk out of the equation. Using standard tools like ldapsearch (on Linux) and LDAPBrowser (on Windows), I go through each of these ten elements until I find the one that is at the root of a support call. Hopefully, knowing these issues will help you as you are deploying the Splunk App for Active Directory. If not, our Support group is always standing by, ready to assist.