Splunk at the Wall for DEF CON 23 – Part II

­­­­Splunk at the Wall for DEF CON 23 – Part II

Hello again. Since the initial post, we’ve released the app developed for the Wall of Sheep. I’m going to go over the functionality here.

To review, the WoS app is meant to be a proof of concept that shows the type of data that traverses the wire, in the clear. Some of the data is innocuous, but we try to highlight the data that could be used by adversaries targeting your data. In fact, you may not even know that you have software using insecure protocols, so it pays to dig in and find out.

Before we go through the various dashboards, I want to comment on the data sources that we us in the WoS app. I’ve been asked by a few people, “why did we choose the specific data sources that we used (Splunk Stream, Bro, and Snort), when we could have realistically stuck with just Stream or just Bro?”. When it comes down to it, we could have utilized just one data source, but since this is a teaching tool and part of which is to show some of the various ways to gather data, we decided to showcase a few options.



This was an interesting dataset. To monitor chats/IM, we pulled data sent via the XMPP (Extensible Messaging and Presence protocol, formerly known as Jabber). We used the Splunk Stream application to capture this one. For those that don’t know, this protocol is used for a number of different applications, including text chat, gaming, VoIP, IoT, file transfer, and more. When not encrypting the data with TLS, the data is sent in the clear. We primarily saw XMPP being used for gaming on the DEF CON network, for both game control, along with chat amongst the players.

Splunk Wall Sheep chat/IM



OK, you may be thinking, “who the hell uses FTP these days?”. We sorta thought the same thing. However, it turns out that old habits die hard, and, maybe more importantly, old (insecure) tools die hard. Especially when users are satisfied that they “just work”. We used Splunk Stream to capture the FTP protocol data on the network, and while we only displayed the login IDs being used, passwords are also easy to pull with Stream. With other, readily available tools, you can easily pull whole files off of the wire or from full packet capture. When transmitting files across a network, consider using scp, sftp, or ftps … for all of us baby.




This particular dashboard, displayed as a word cloud, garnered a lot of attention when attendees (particularly managers) saw their company’s domain name on the big screen. We gathered this data using Bro, monitoring DNS traffic. This is a great example of unintended info making it onto the wire. A user simply needs to connect their corporate computer to a network and it will automatically try connecting to its normal LDAP servers, utilizing DNS and thus giving up info about the owner of the device. This, in and of itself, is not terribly damaging but can identify you as a target. Be careful of the networks that you connect your company laptop to.

Splunk Wall Sheep LDAP dashboard



We took the original Wall of Sheep concept and rather than display partial passwords, used the opportunity to show statistics about the passwords gathered from various insecure protocols. We looked at length of passwords along with the count of upper-case and lower-case letters in the passwords. In addition, we utilized a lookup, with a list of the 10,000 most commonly used passwords, to determine how many of them were being used. We also displayed the sites that allowed the use of insecure protocols.

Splunk dashboard password statistics


Splunk Wall Sheep password fun


Rare Applications

This dashboard uses TCP flow data, collected by Stream, which, amongst other things, identifies the high level application initiating the communication. Many of the most common applications we can already see in detail, so we focus on the rare applications, which tend to be the most interesting.




Bro is providing the data for the panel displayed below, and it’s a simple sorted list and bubble chart, displaying the top domains queried. It might be simple, but it’s generally an interesting one to look at, especially as attendees attempted to get their more interesting domains displayed on the wall.

Splunk Wall Sheep top domains



This view has to do with http traffic on the network. For each dashboard, we used Stream to capture the data. For the first two dashboards, there’s nothing too fancy, the first is simply a list of the top destinations and the top referrers referenced in the traffic.

Splunk Wall Sheep HTTP dashboard

The second deals with user agents; simply listing the top user agents by count. The final dashboard in the Web series is my personal favorite. Because many search engines now utilize SSL/TLS, the search terms that the user submits on their main sites are encrypted. However, when a third-party site or browser add-on is used to enter the search terms, they are often passed to the search engine via the URL in the referrer field. By focusing on the referrer URL of the main search engines, we can see many user queries.

Splunk Wall Sheep Search dashboard



Because we’re dealing with encrypted data, we are not able to see the actual content with these dashboards. However, we can still glean some information from the metadata that is collected. Our first dashboard simply shows a map of ssh (secureshell) destinations, which can be useful for picking out potentially nefarious communications to parts of the world where it may not be expected. It also shows the string that the server has identified itself with, which can be an indicator of unwanted communication or legitimate communication with a server with known vulnerabilities.  This year we plan to pair the detected server types with a lookup table showing vulnerable SSH versions and color code known vulnerable server versions with red.

Splunk Wall Sheep SSH logins

SSL/TLS Certificates gives two lists, the first showing certificate status with expiration data, useful for identifying communication with illegitimate sites. This dashboard also shows servers using self-signed certificates and the clients communicating with them. Self-signed certs are fine for setting up encryption but of course there is no third-party who has verified the identity of the creator/host.

Splunk list SSL/TLS certificates



Our final category contains dashboards that are frankly, just fun to look at. They include “Top Internal Talkers”, which simply shows a list and a bubble chart of internal clients that are making the most network connections along with “Snort Categories” displayed in a word cloud of the most common Snort categories that are triggered by the incoming data. “Stream Stats”, utilizes the Stream data to display a stacked column chart of the count of protocols being utilized, over time. Last on the list is “That’s weird, bro”, which uses data from Bro (specifically the “weird.log” file) to show the various categories being triggered by the incoming wire data.

Splunk Wall Sheep Snort categories

If you find this interesting and haven’t downloaded the app I encourage you to download it and take a look. We’ll be making some improvements and putting it to use again this year at the DEF CON Wall of Sheep and a few other cons. So if you have any suggestions or comments, please do submit them. Thanks for reading.

Steve Brant

Posted by