This is part nine of the "Hunting with Splunk: The Basics" series.
When hunting, advanced security Splunkers use apps. Specifically, three related apps from an incredibly generous man named Cedric Le Roux! (You can guess from the name that yes, he's French.) And frankly, you probably only know one: URL Toolbox.
Cedric blew onto the Splunk apps scene with URL Toolbox more than three years ago. One of the most popular Splunk security apps of all time, it’s URL parsingcapabilities have been leveraged by thousands who want to separate subdomain, domain, and TLD from a URL. But he wasn’t satisfied with just splitting the atom URL…oh no. He also added many other exciting analytic capabilities that have been showcased in multiple .conf and SplunkLive! talks. His tools are so powerful, we must break this blog into two separate posts! Enough with the intro though—let’s talk parsing.
How to split URLs and domains with URL toolbox
To be successful with URL-based or domain-based security analytics (we will have many examples in our next hunting blog post!), you need to be able to parse URLs and domains from your data. Many of us regex fiends think “Oh, that’s just field extraction, so what do I need an app for?”
Turns out that’s virtually impossible!
Accurately parsing international domains is particularly difficult because it requires knowledge of nearly all special TLDs in the world (e.g., .com, .co.uk). That may not seem too tough on the surface but did you know that k12.al.us is a TLD according to Mozilla? How about .இலங்கை? On top of the TLD issue, additional complexity is introduced with ports, usernames and passwords found in a URL.
Fortunately for us, that’s all built into URL Toolbox! Here is a simple search to separate domains from TLDs located in PAN logs:
index=pan_logs | eval list="mozilla" | `ut_parse_extended(url,list)`
This is a good time to point out that because URL Toolbox isn’t a custom search command, you get access to all its power via macros (so remember your `ticks`)! One of the most commonly used macros in URL Toolbox is called `ut_parse_extended(2)`. It parses your URL and passes the data to multiple different fields prefaced with ut_.
How does ut_parse_extended look when you use it? Let’s take a look at some pseudo code:
| eval list="mozilla" | ut_parse_extended(url,list)` | <additional Splunk commands like stats, sort, table, etc>
You’ll notice that we're bringing two fields into the ut_parse_extended macro. The first is the URL, which is pretty straightforward, but the second is a field called “list.” That’s part of the magic of URL Toolbox—that “list” field is the catalog of different TLDs that we are looking for.
There are a couple of common lists that exist in the world (including an official one from IANA), but if we’re trying to differentiate the domain from the top level domain (TLD), the most popular source of truth is from Mozilla. Mozilla’s list of TLDs not only has “classic” TLDs like .com and .co.uk (which is bizarrely missing from IANA), but it will also include items like .edu.tj (because you never know when someone may attack you from university websites in Tajikstan). The important takeway is that you need to use eval to make a field called “list” with the value “mozilla” or “*” (which searches all of the TLD lists available) before you actually call ut_parse_extended.
Here’s another example:
index=pan_logs | head 1 | eval list="mozilla" | `ut_parse_extended(url,list)` | table url ut* | transpose
In this example, we use the head command to return a single record. We then use the `ut_parse_extended(url, list)` macro to parse the URL based on the Mozilla TLD list.
Notice how we then create a table and the flip it with the transpose command? That allows us to see all of the values URL Toolbox creates from parsing. You don’t have to do this, but it makes it easier to understand the new fields that URL Toolbox is creating for you as you begin hunting through your data.
Many people don’t realize this, but you can use URL Toolbox macros on domains that aren’t in a URL. Here is an example from the Splunk Security Essentials app, where the domain is extracted via the rex command from an email:
index=email mail from | stats count by Sender | rex field=Sender "\@(?<domain_detected>.*)" | stats sum(count) as count by domain_detected | eval list="mozilla" | `ut_parse_extended(domain_detected, list)`
The world is your oyster with URL Toolbox. If a field has a domain with a TLD in it—whether email, DNS, web, or others—you can use URL Toolbox to extract goodness from it!
But wait! This is 2017… Surely there’s something new, right? You’re absolutely right.
Earlier this year, Cedric decided that his URL Toolbox was good but it wasn’t fast enough. When he found out that the API for Splunk got supercharged, he created a new tool: URLParser. (Note: Don’t confuse with the older “URL Parser” that has a space—no spaces for us!)
URLParser isn’t a full replacement for URL Toolbox since it doesn’t have the enhanced analytics capabilities that we will talk about in the next post, but it’s much faster. In fact, parsing a sample of 222,167 URLs with URLParser was 5x faster (257 seconds vs 45 seconds) than URL Toolbox. Pretty sweet!
index=pan_logs | stats count by url | urlparser field=url listname="mozilla" mode=extended
One important difference when leveraging URLParser is that it prefaces its fields with url_ rather than ut_. So where URL Toolbox would create the field “ut_domain”, URLParser will create url_domain (and create it 5x faster!).
Analyzing Parsed URLs
Parsing URLs is important and every analyst needs to start with that technique, but it’s not going to separate out the bad URLs from the good URLs, right? Well, Cedric has a few additional tricks up his sleeve.
In URL Toolbox, he added a suite of analysis tools that will help you find bad guys with mathematical accuracy. The most used analytic functions of URL Toolbox are Shannon Entropy and Levenshtein distance. Shannon Entropy allows you to calculate the randomness of a string so that you can find algorithmically-generated domain names, and the Levenshtein distance calculation shows bad guys phishing via typo-squatting (for example, campany.com vs company.com). For more details, stay tuned for our next post where we take a deeper dive into using these functions.
And as always: Happy Hunting :-)