Universal Forwarders and the Splunk App for Active Directory

About once a week I respond to a call or online question asking about the Splunk App for Active Directory.   Specifically, these questions ask one of two things.  The first is “can I collect the Active Directory data remotely?,” and the second is “What user shall I run the Universal Forwarder as?”  The cliff notes version is that you should not collect Active Directory data remotely, and you should install the Universal Forwarder as the system local user.  If you want more information, read on.

Let’s start with the first question – can you collect the Active Directory data remotely?  Technically, the answer is yes, but reality is the answer is that it is ill advised from a security point of view and difficult to do, requiring many changes that are not supported.  Let’s look a bit at the pieces that you may want to collect:

  • Windows Security Logs
  • DNS Debug Logs
  • Performance Counters for Active Directory and DNS Operations
  • PowerShell results for host information
  • PowerShell results for DNS information
  • PowerShell results for Domain Services Replication information

You could set up a log subscription for the Windows Security logs and collect those logs on a remote system.  This accounts for about 90% of all the log gathering that the Splunk App for Active Directory does, so it’s a great option.  You will have to add a transform on the collecting host to ensure the host field is set appropriately – if you don’t, the events will appear to be from the collecting host instead of the generating host.  Do not use syslog, snare or WMI to collect the Windows Event Logs – the format is different and you would have to do an extraordinary amount of work to get the same field extractions that we rely on to produce the dashboards.  The DNS Debug Logs are generated within the SYSTEM32 directory and are locked when they are being written.  You cannot collect these logs remotely as locked files cannot be opened remotely.  Locally, you can use the MonitorNoHandle method of reading the file.  Fortunately, you only need them for producing certain reports within the app.  If you can avoid those reports, you can also decide to forgo the collection of the DNS Debug data.  The performance counters can be collected via WMI.  However, the names of the counters are different, so the dashboards will remain unpopulated without some work.  Since most organizations use something else (for example, Systems Center Operations Manager) for performance management, this may not be a significant loss.

This leaves us with the PowerShell scripts.  As many of you will know, PowerShell includes the ability to run commands remotely.  However, the scripts are not designed to run remotely.  When you run a script remotely, it serializes the results and sends them back to you.  Unfortunately, when deserialization occurs on the collection end, the data is not in exactly the same format.  In addition, you need to have domain or enterprise administrator privileges to run many of the functions that touch the Active Directory Domain Controller functionality – things like replication information and the contents of the DNS Server.  A good PowerShell expert might be able to alter the scripts, but how are you going to get around the requirements for the wide-ranging security requirements that remote execution requires?

This brings us to the next question – who should I run the Universal Forwarder as?  As you might have guessed from the above comments, you have only two choices, and one of those is really not a choice.  Your two choices are the system local user and a domain administrator.  If you choose the domain administrator, then the Splunk Administrator can execute any PowerShell script anywhere in the domain.  This is bad – really bad.   I strongly discourage you from even thinking this is a good idea.  It isn’t.  Worse – if you are running the Splunk Universal Forwarder on a forest root node, the user needs to be an Enterprise Administrator.  This is even worse than the domain administrator, giving the Splunk Administrator the ability to do anything on any computer within the enterprise.  Don’t do it.

That leaves us with the system local user.  Choosing this option will allow the universal forwarder to access all the information it needs, but only on the local system.  It can’t access that same information across the network.  This is perfect – you isolate the capabilities to just the system you are monitoring.  Of course, this requires that you have a Universal Forwarder on the domain controller, but that’s better all round for a security setting.

This begs the question – what is the impact of the Universal Forwarder on my domain controller?  There are two phases of collection – baselining and ongoing service.  During baselining, you can expect a memory utilization about the same size as your directory and a high CPU load.  If your directory is large, then this can take some time and be a concern.  Once the baselining is complete, you can expect a 40-80Mb RSS on the memory side and 3-5% CPU utilization, depending on how busy your domain controller is.  I always recommend the following actions:

  1. Turn off baselining
  2. Turn off the admon data input on all but 1-2 systems (depending on redundancy needs) per domain.

You can read all about these recommendations in more detail in another blog post.  If you follow these rules, your Splunk App for Active Directory installation will go smoothly.


Posted by