By Splunk February 09, 2015
A customer asked me recently if Splunk can be used to detect when a new port starts listening on a host. This seems like a pretty modest request; in fact it’s one that I get a lot. Being able to identify and track open ports on a system isn’t just a good idea, it’s the law! Well, it is if your organization is affected by compliance with:
– PCI-DSS Sections 1.1 and 2.2
– CIP-007-R1
– CSC 11
And those were just the references I had within arm’s reach!
When I talk to customers about tracking listening ports, a typical conversation can go like this:
“I need to track when a host starts listening on a new port. Can Splunk do that?”
“Of course! Check out the Port & Protocol Tracker page in our SIEM, Enterprise Security.”
“Sweet! But we don’t have Enterprise Security yet.”
“You should buy it.”
“I know. How can I solve this problem in the meantime?”
“Can you track NetFlow?”
“We can and do, but not everywhere. We’ve got some old routers that can’t handle the CPU overhead of turning on NetFlow.”
“That makes me feel a little sad inside.”
“You and me both.”
[Brief moment of silence.]
“What about Nmap? Do you run Nmap on your network?”
“Pfff… Every day! ‘What’s this host doing? Scan it!’”
“Have you seen the Splunk for Asset Discovery app?
https://apps.splunk.com/app/662/“
“Nope.” [Sounds of Web browsing…] “Dang, that is sweet.”
“You wanna see an alternative cool hack?”
“Wasn’t that the name of the place you got your hair cut in high school?”
“Nice one. Have you seen the Splunk App for Stream?
https://apps.splunk.com/app/1809/“
[More sounds of Web browsing…]
“Oh. My. God. You have to show me more.”
If you’re reading this and you haven’t seen the Splunk App for Stream yet yourself, I’ll give you a minute.
https://apps.splunk.com/app/1809/ (There’s also a separate app that’s full of examples on how to use Stream:
https://apps.splunk.com/app/1840/.)
The Splunk App for Stream is how Splunk ingests wire data. What is wire data? Wire data is the information that passes over computer networks defining communications between client and server devices. Wire data is what’s happening on your network. It’s the pulses of data that power the pulses of light in your Blinkenlights 3000 router.
If you’ve been sniffing packets as long as I have, you already know the value of wire data. If you haven’t tapped that main line yet, consider this: looking at wire data can give you tremendous insight into
EVERY business unit within your organization. OK, only those business units that use computers on a network, but with Stream it doesn’t matter if that network is on premise or in the Cloud; Stream works just as well in either environment. Wire data is holistic and comprehensive, it is real-time communication between applications and macines across various protocols. It is the whole enchilada (or burrito if you prefer).
And the best part is that you can get it completely passively. You don’t have to overhaul existing applications to instrument the code or enforce semantic logging. All you need to do is find the sweet spot (or spots) in your network to put a tap or mirror a port. But, that’s not even the best part. The best part is that with the Splunk App for Stream, you can configure what data types you want to capture from which protocols, and what you want to completely ignore. It’s a data buffet, not a hot dog eating contest.
Being choosy about which protocols you capture.
I capture and index everything I can. Because I can. And because I like to catch things I didn’t think were there.
So, back to tracking port status. One of the protocols that Splunk dissects and indexes is plain old TCP. To look for a host listening on an open port, look for Stream events that have data being sent to that host:port. Easy, right? How would you do that? Take a look at this event:
This Stream event shows SSH traffic between two hosts on a test network. I know that this is an open port and that data is actually passing back and forth, because I see that the count of ack packets in and out is greater than zero.
If you know TCP networking, you know that it all starts with a Three-Way-Handshake:
Syn ->
<- Syn-Ack
Ack ->
For the non-networkers in the crowd, this is kind of like:
Knock-knock ->
<- Who’s there?
It’s me. Let’s talk. ->
So if I’m looking for data flowing TO a host on a specific port, I can run a search where that host is the dest_ip and the port in question is the dest_port and where the count of ack_packets_in is greater than 0. In fact, if I want to see a list of ports on a system that have received data in the last 24 hours, I could run this search:
Any host that’s receiving data on a port is listening on that port, but not all listening ports receive data. (Just like trains that pick up and drop off passengers at train stations, don’t necessarily stop at every station.) Also consider that to monitor every host passively, you may need to have multiple taps in your network. Although this will catch every host that’s running a network service that’s receiving data, it won’t see a listening port that has yet to receive any data.
If you want to proactively discover listening network ports, send them some data.
Enter nmap. Nmap is a fantastic tool for sweeping a network to discover open ports on individual hosts, but you probably already knew that. If you run nmap -T and run a “Connect scan” on your network, it will connect to each open port it finds, sending the host the full three-way-handshake, including ack packets. (This is opposed to a Syn scan, or “Stealth scan” that only sends the Syn packet to initiate the handshake and sends nothing in response to the syn-ack reply from the host with an open port.)
Nmap is arguably the most widely used port scanner available, and the Splunk App for Asset Discovery was purpose-built for your nmap scans. But the truth is that there are other port scanners out there; maybe you’re already using one, or maybe you’ve written your own. One of the nice things about this solution is that although it can leverage nmap as a scanner, it is not directly dependent on it. Anything that can try to open an arbitrary port can be used as a port scanner. Got NetCat? If you’re running Mac OS X you do. Here’s how to scan your local subnet for SSH servers:
bert@bochs:~$ for i in {1..255}; do nc -w 1 -v -z -n 10.100.0.$i 22; done
Now, if you run the Splunk App for Stream on the same host that’s running your scanner, you can get all of this data on demand for every single host that’s reachable on your network, all from a single place, and all without turning on NetFlow everywhere or putting network taps all over the place. You also don’t have to worry about field extractions on the output of the scanner. Simply look for anything with a src_ip of your scanner and ack_packets_in>0.
In the following screenshot, 10.100.0.11 is the system I have running a port scanner and Stream and 10.0.0.0/8 is my test network. I’m looking for all open ports that have received traffic (ack packets) in the last 60 minutes. In this case, the ack packets are coming from my Nmap scanner; scans from NetCat would look no different. (I’m converting timestamps to and fro so that humans and machines alike have an easier time reading them.)
So now I have a cool way to use Splunk to find listening ports on my network regardless of what port scanner or connection method I actually use. How do I detect when a new port is listening? Something that we’ve never (ever ever) seen before?
That, dear reader, is a topic best left for another blog post. That’s right, it’s a cliff hanger ending! Bet you didn’t see that coming…
OK, here’s a teaser: we’ll be using something new in Splunk 6.2:
The Splunk Key Value Store.
Stay up to date, follow us.
One Trackback
-
[…] Part 1 of this post we looked at using the Splunk App for Stream to look for open ports on your networked […]
----------------------------------------------------
Thanks!
Bert Hayes