Last week I was in NYC for Interop 2007. Interop in NY is a significantly smaller conference than the big brother Interop in Vegas. I’d say there were 7,500 to 8,000 people at Interop NYC this year, compared to 18,500 in Vegas back in May. Somehow though I always find the New York show more interesting. Perhaps it’s the lack of constant firefighting in the NOC that gives us all more time to have meaningful conversations about the latest networking technologies. Plus somehow New York just seems to have more substance than Vegas. Call me crazy but…
This was also the first Interop where we had a chance to apply the magic of Splunk genre 3.0. We had a record number of searches in the NOC (despite the smaller show). I’m not surprised. 3.0 is so cool the way it automatically extracts fields out of data streams from all kinds of networking gear.
Now there are lots of people who know more about networking and security than I do, but here’s a simple investigation I did with Splunk.
1. I started with a simple search for “failed password.” This picks up firewall and router hacking attempts (typically ssh) sent to Splunk using syslog forwarding.
2. I was then able to quickly see the top “source IP”. Because the source IP field automatically gets extracted with each search I’m able to quickly click and see the list of top source IPs for the time frame in question. A single click and I’ve added the top offender to my search parameters.
3. Just a click away and I can geolocate this IP. With field actions in Splunk I can now drive workflow items right from the search results. Here I just need to click on the menu next to any IP address and I can geolocate the address with any number of free web based services. It was interesting to watch the hackers and bots travel around the world and with more time would have been fun to write a little Flash application to call the Splunk API and map things in real-time.
4. Reporting on top source_IPs every hour was easy. Like any IT guy without a bunch of time, I went for the low road. I just clicked report on all source_IPs from the field action menu and I got a nice looking flash report. It was really easy to save the report and run it on a schedule every hour. Now anyone on the NOC team alert list can get it right in their email or log into Splunk and check out the dashboard with a few other useful security searches.
You can split the same report series by user and see how a lot of these hacker bots try to use common software package and open source default configuration usernames and passwords.
If you want to check it out yourself, send me mail and I’ll let you know where you can access the server. It’s kinda fun to search on your own machine name and see all the times you were on the network at the show. You can drill down into each DHCP transaction and see all the events.