Decode First, Panic Later: A Tier 3 Analyst's Case for Staying in the Loop

Like a lot of my colleagues in the .conf26 Agentic Security Operations Center (SOC), I started the morning genuinely excited to be working as a Tier 3 analyst. The thing about Tier 3 is, by the time a finding reaches us, it's usually already survived a few rounds of scrutiny. Somebody, or something, thought it was worth escalating. Part of my job is to be the last check before a call gets made. In an Agentic SOC, that means I'm not just reviewing a human analyst's work anymore, I'm reviewing an AI's as well.

That distinction matters more than I expected going into .conf26. One of the promises of agentic security operations is speed. We have come to expect an AI assistant that can run the query, pull the context, and hand you a shortlist before you've even finished your coffee. Sounds great, doesn’t it? What I didn't always fully appreciate until I worked more than one live case in this age of AI, is that speed cuts both ways. An AI assistant can lead you down many different paths, so it’s critical to be the one in the loop that stops to ask "Wait, what does this actually say? And what does it mean for the investigation?"

The Finding: A Textbook DNS Tunneling Pattern

Whenever we stand up an event SOC, our primary directive is simple: guarantee the resilience of the network and keep the attendees connected and secure. That means triage must be rapid, but containment must be surgical.

Striking that balance was especially critical during .conf26. In my day job on Splunk's Advanced Response Team (ART), I spend my time hunting threats and defending our corporate infrastructure. But an event network operates under completely different rules. Traffic that would trigger immediate Sev-1 alerts in an enterprise-like attack simulations in Splunk University, intense challenges in Boss of the SOC (BOTS), or live vendor exploit demos-is completely expected here. The real challenge wasn't just spotting anomalies; it was isolating genuine threats hiding in plain sight within a massive sea of intentional noise.

That's where our Agentic SOC architecture fundamentally changed the game. Running inside Splunk Enterprise Security, autonomous agents took on the heavy lifting of pulling context, correlating multi-source telemetry, highlighting missing evidence, and proposing triage paths. Crucially, human analysts remained the authoritative decision-makers. Once we rigorously validated agent accuracy under live traffic, we granted them calibrated autonomy to execute repetitive data-gathering tasks independently-backed by a full, tamper-evident audit trail.

During SOC booth tours, attendees constantly asked us: "Is AI coming for analyst jobs?" Our deployment at .conf26 proved the opposite. The Agentic SOC isn't about replacing humans-it's about empowering analysts with machine-speed evidence, so people make better, faster decisions.

The finding that landed in my line of sight started with another analyst’s hunt for long DNS queries. Anantha Srinivasan, one of our Endace analysts, was searching for these types of queries over 128 characters, since legitimate lookups are short and DNS tunneling protocols love stuffing data into long, ugly hostnames. When I validated his findings in Splunk, the results looked exactly like that, which raises concerns in any analyst’s mind.

index=se_network_endace sourcetype="zeek:dns" | where not like(query, "%sophos%") | where len(query) > 128 | stats count by query

Results:

What came back were dot-separated blobs of hex, chunked into DNS-label-sized pieces, aimed at a handful of destinations, with no obvious response coming back. If you've spent any time hunting for command-and-control, this is the shape you're trained to recognize. The analyst flagged it as worth investigating, which it was, but a flag isn't a verdict.

Where the AI Assistant Earned Its Keep

I'll give the tooling its credit here, it moved fast. Instead of me hand-typing five separate follow-up queries, I worked through the investigation conversationally with my AI Assistant, proposing each next step. That approval gate mattered to me. This is live conference traffic, real attendee devices, and I wanted to build and validate queries before they executed, not just because it's good hygiene, but because it kept me anchored to what we knew versus what we were assuming.

With my query results in hand, the assistant handled the mechanical lift by pulling the source hosts we were missing, checking whether the queries actually got a DNS response, characterizing the other traffic on the same destination IPs, and pulling TLS certificate metadata once we had a live lead worth chasing. That's the part agentic tooling is genuinely good at: keeping pace with an investigation that would otherwise mean a dozen manual pivots between Splunk, Wireshark, and a scratchpad.

Where the Human Call Still Had to Happen

The part that the AI didn't do for me was decode the hex and read what was inside it.

That's a deliberate choice, not a gap in capability. Before chasing infrastructure, I wanted to know what these labels contained. That involved stripping the encoding, converting the bytes, and looking at the ASCII. What came back wasn't the random noise you'd expect from an encrypted C2 channel. It was things like:

That's the moment the story changed for me. Real C2 malware doesn't label its own beacon traffic with plaintext environment names, because that kind of transparency defeats the entire point of hiding a channel. An attacker who names their own infrastructure "Production" and "Staging" in a hostname is an attacker who wants to get caught. This wasn't a hunch an AI could arrive at from the query alone. It required someone in the seat who's spent enough time looking at real malware to know what it doesn't do.

From there, the investigation became a partnership again. We tracked down the source hosts, confirmed every one of these queries got a clean response instead of vanishing into a sinkhole, and found that two of the "suspicious" destination IPs were just our own Umbrella DNS infrastructure doing exactly what it's supposed to. The remaining three AWS-hosted IPs, though, had a real relationship with a small set of hosts: hundreds of megabytes of genuine HTTPS traffic, not a beacon check-in.

The Reveal: Following the Certificate

The last piece came from a TLS certificate, not a DNS log. One of those hosts had a 628MB HTTPS session running alongside its DNS chatter, and the SNI on that connection was sdp.[REDACTED].cloud. SDP stands for “Software-Defined Perimeter”; standard Zero Trust Network Access terminology. The same branding we'd decoded out of the DNS labels was sitting right there in the certificate hostname, and Okta was named explicitly as the app being checked, in both Production and Staging.

index=se_network_endace sourcetype=zeek:ssl earliest=-7d latest=now id.orig_h="10.x.x.x" id.resp_h="52.x.x.x"
| stats count values(server_name) as sni values(subject) as cert_subject values(issuer) as cert_issuer by id.orig_h id.resp_h

Put together, this was a ZTNA client agent doing exactly what it was built to do. Lightweight DNS-shaped health checks to its own cloud control plane, naming the app it's brokering access to, running alongside a real access tunnel. Some of these agents deliberately build their own lookup protocol over port 53 specifically so it survives captive portals and DNS-only egress rules, which is a design choice that can trip up a "long DNS query" hunt.

Agents Prepared. Humans Decided. Evidence Proved.

Nothing about this case was resolved by the AI alone, and at the same time, it would not have moved fast enough without it. The assistant kept the investigation moving at the pace a live SOC demands by proposing the next query, surfacing the source hosts, checking the response codes, and pulling the certificate. I did the part that still requires a person with judgment which included deciding that plaintext, self-describing strings are evidence against malicious intent and knowing that strings within the URLs meant something specific in the context of Zero Trust architecture.

That's the shape of the Agentic SOC I experienced. Not AI replacing the Tier 3 seat, but AI clearing away everything mechanical, so the seat could do the one thing it's there for: making the call and being able to explain exactly why.

Acknowledgements

Our thanks to the engineers who built the Agentic SOC and the Humans who provided decision making expertise.

• SOC Co-Leaders: Jessica Oppenheimer & Paul Pelletier
• Agentic SOC Innovation: Ryan Maclennan & Aditya Sankar
• Splunk Integrations: Josh Wilson & Christian Cloutier
• SOC Analysts: Christopher Van Der Made, Sean Clapper, Oxana Sannikova, Daniel Christiansen, Lily Lee, Dan Burke, Kyle Vaughan & Ray Aragon
• Cisco Security Firewall / Switching: Adam Kilgore & Andrew Merica
• SOC VIP Tours Coordinator/Ops: Michelle Hermosillo
• Threat Hunter Tier 3/IR: Richard Marsh & Allison Gallo
• Detection Engineer: Rod Soto
• AI Canvas / XDR Forensics: Rob Gresham
• AI SOC Analyst Engineering: Fred Frey
• Remote support: Bhavin Patel, Shyue Hong Chuang, Nasreddine Bencherchali, Onur Erdogan, Nathan Schoen, Paul Carrillo, Jon Lane, Ryan Stillions & Raven Tait
• Endace Full Packet Capture: Michael Morris, Tom Leahy, Anantha Srinivasan, Elliott Hinson & Andreas Lof
• Jamf proof of value: Adam Derrick