Getting out the office to see successful Splunk customers is always a pleasure, and the presentations and conversations at SplunkLive in London were especially a treat. One of the most striking things about all three customers (Vodafone, Telenor and Accenture) is how Splunk has transitioned from a tool used by a couple of working teams into a cross-organization IT utility. Despite being from two different industry verticals, they also all approached the problem in a similar way, and that way suggests the new dynamic lookup feature is going to be very popular.
If you’re an existing Splunk user, you might be familiar with our transaction search-time command. It’s used to identify patterns that indicate a single, unified intention – such as buying something from an online store – even across multiple data sources. That works great when there is some common piece of data to anchor on, such as an IP address or user name. In both the online retail and telecom use cases we saw in London, that was a major part of how groups at different layers of the stack exposed their data to their peers working elsewhere; e.g. the IP address was a way for the web team to track the network behavior of a host through the router logs to look for network-layer abnormalities. These kinds of searches were common to all of our London presenters’ normal use of Splunk.
But what do you do if there is no shared piece of data tying two sources together?
Enter the dynamic field lookup feature. It’s like summary indexing light – you run a search that populates a smaller, more manageable table structure with data. But here’s the difference: dynamic lookups can act as an intermediary, joining data from one sourcetype with another at search time. For example, we use this for the Windows GUID lookup feature. When Splunk indexes Active Directory, it identifies all the GUIDs and adds the GUID and its associated common name to a lookup table. Then, if you ask Splunk to translate GUIDs, it takes all the GUIDs in your search return and checks to see if it’s in that table. If it is, a new field is dynamically added to your searched events – the common name – as if it had always been there.
That’s a fairly basic use of the feature, however. Vodafone, who was a London presenter and Splunk 4.0 beta tester, had a more ingenious use case. They’re using it to create abstracted data access points for each IT service they manage. So one service – for example, the customer management system – can return via a Splunk search the last few numbers a customer called if you search on the customer number, but not return the customer’s name or other revealing information. Other groups can then consume that information, much like a feed or other web advertised service, directly in their own searches and dashboards. Not only is the data access constrained by role, but potentially also by time as well, providing secure windows into past activity that still respect the privacy of Vodafone’s customers.
The idea of joining data from one source contingent on another source in a safe and controlled fashion using Splunk seems to resonate with almost all of our beta customers. Dynamic lookup tables may end up being one of those features that has much more mileage in it than we ever anticipated. Learn how to make yours here.