Splunk Stream in VMware Environments

Splunk Stream is an amazing wire data collection and analytics solution from Splunk. With Splunk Stream, you can capture many different types of protocols, encrypted or not. Complimentary to Splunk Stream is the brand new Splunk Essentials for Wire Data app created by my colleague here in Australia, Simon O’Brien. Aside from having really cool icons, this new Splunk Essentials app showcases the many use cases for Stream data complete with SPL examples and instructions on how to get amazing data from Stream.

Here are the categories that the Splunk Essentials for Wire Data offers:

A particular example that Simon focuses on is using Splunk Stream for DNS data. Have a read through his blog post, "Splunking DNS Using Splunk Stream – AKA, The Easy Way," to go through the configuration and benefits of doing this (and getting what he refers to as “lovely, lovely, DNS data").

So if we focus on this very same DNS use case but in a VMware virtualised environment, how do we actually get the wire data into Splunk Stream? The first method is to use the streamfwd binary supplied with Stream and install in-guest in the virtual machine hosting the DNS server. As Windows is a very common solution for DNS (since it’s tightly coupled to Windows AD), this can become a challenge. That’s because Stream uses WinPcap drivers, which unfortunately allow all local users the ability to use its packet sniffing capability. It also requires the NIC to be put in promiscuous mode. How many Windows sysadmins want their domain controller with these kinds of capabilities? Remember that we're now talking about a VMware ESX environment with all its goodness, so why install a binary in guest machine?

Luckily, VMware provides some really easy solutions to get this lovely data into Stream. Most VMware customers will be using VDS (vSphere Distributed Switch) within their ESX environment, so the solution is easy! Within each VDS, you can setup port mirroring; with this setup, the traffic from the Windows DNS host can be mirrored to the Splunk Stream forwarder with no intrusive configurations in the Windows DNS host. 

Before I show the setup of this, there's a really important item to note. First, a VDS is really not the same as a physical switch, for example, across all ESX hosts. It appears as a single switch but has some limitations. Most important to note, port mirroring does not work between ports when the endpoints are on different ESX hosts. I’m advised this may be different for VMware NSX environments, but for non-NSX, this is the case. This means that your Windows DNS host VM and your Splunk Stream VM should always be co-hosted on the same VMware ESX host. This can be easily accomplished by setting up VM/host rules, found under the configuration of your HA cluster. Setting a simple rule forces the VMs to follow each other around during HA/DRS events.

In terms of setup, it’s really quite easy. In your vSphere client, navigate to the networking menu, select configure, and find the port mirroring item under settings.

Select “new” under port mirroring and select “distributed port mirroring.” This will do a port mirror from two specific ports on a VDS. Note that you can also mirror from a specific VLAN to a destination port (remote mirroring destination), a specific port to an uplink(s) (remote mirroring source), and specific port to an IP address (encapsulated remote mirroring).

Give your session a name, and make sure to enable it!

Select the port of your source Windows DNS host and make sure you choose ingress/egress.

Select the port of the Splunk Stream forwarder.

And that’s it on the VMware side; now to the Splunk Stream side. On your Splunk heavy forwarder with Splunk Stream installed, go into the Splunk Stream app and into the configuration menu. Here, scroll down to DNS and edit it.

All you need to do is ensure it’s enabled and that you’ve chosen the appropriate index (“mirror” in this case)—that’s about it. Note: If you’re concerned about the amount of traffic, you can put the metadata stream into estimate mode which will show you beforehand how much data you will ingest.

Head on over to search and have a look at what you have coming in. In my case, I would search for index=mirror sourcetype=stream:dns 

What comes back is some lovely DNS metadata all structured quite nicely in JSON format. In this case, I issued a ping splunk.com from splunk-fwd01 (10.1.11.37) which uses my windows DNS (10.1.11.50) for name resolution.

Here we see src_ip as my originating requestor (10.1.11.37) and dest_ip as the Windows nameserver (10.1.11.50) and the name of request, splunk.com. There is a whole host of other useful fields that can be leveraged.

Lastly, have a look at the scope of what you will be collecting and think of using a dedicated vNIC in the Splunk Stream forwarder, as well considering the overall network impact; the Stream estimate facility should help you with this. Also, don’t forget to set up VM rules as stated previously or else you may not get data!

I’ve covered just one possible use case, but there are numerous other ones like DHCP, SMB, NFS, etc. Also be aware that Stream can ingest Netflow data, of which the VMware VDS can export as well. The uses cases for Netflow revolve around capturing specific metadata on network flow, e.g. the rate of network flow.

I encourage to you to have a play if you’re a VMware customer and see how easy it is to leverage Splunk Stream in virtualised envirnments.

----------------------------------------------------
Thanks!
Dean Jackson