The Scalpel vs. The Sledgehammer: Lessons in Precision Triage from the .conf26 SOC

Security Kyle Vaughan

Key takeaways

  1. Blocking an entire IP address can break legitimate services since malicious sites and safe apps like Samsung Health often share the same cloud infrastructure.
  2. Investigators used AI-generated analysis, firewall logs, and packet data to isolate exactly which device was connecting to malicious domains among 15 similar connections.
  3. Blocking the specific malicious domain names instead of the shared IP address stopped the threat completely while keeping legitimate traffic running smoothly for everyone else.

1. Stepping onto the SOC Floor: The Realities of Live Conference Triage

When I stepped onto the SOC floor at .conf26 in Denver, I wasn't there to look at theoretical architecture diagrams or marketing slide decks. As a Solutions Engineer, my goal was to get hands-on with live traffic, work notable security findings coming off the wire, and see firsthand how our combined Cisco and Splunk security stack performs when thousands of attendee devices hit the network at once.

scapel-1.jpg

Conference networks are a unique beast. You have thousands of transient, unmanaged BYOD laptops, smartphones, and tablets connecting across high-density wireless subnets. You don't have an enterprise EDR agent running on these endpoints; you don't have corporate MDM profiles; and you have zero visibility into what processes, browser tabs, or background scripts are executing locally on the attendee's machine. Everything you know about a security event has to be derived from the network layer—firewall telemetry, security intelligence feeds, and packet-level truth.

During my shift, I investigated an incident—Incident ES_00244—that perfectly illustrates the practical challenge every SOC analyst faces today: how do you rapidly neutralize an active exploit link when the underlying IP infrastructure is simultaneously serving legitimate, business-critical cloud traffic?

Luckily enough, the .conf SOC was powered via Cisco Splunk Agentic AI, and the initial analyses were provided as powerful tools to pivot and build out our story as the SOC Analyst humans-in-the-loop. The below analysis is included by default with each incident, empowering analysts to interact with the findings via an agentic AI Assistant to brainstorm approaches for further investigation and containment.

scapel-2.png

scapel-3.png

Figure 1 & 2: Automated Agentic AI initial investigation brief populated in Splunk Enterprise Security, synthesizing entities, observed traffic behaviors, and preliminary risk indicators.

Via the AI Assistant, I can view the individual SPL queries, generate new queries, and gain granular understanding of the AI-assisted findings.

scapel-4.png

Figure 3: Interacting directly with the AI Assistant to inspect underlying SPL queries, generate targeted data filters, and drill into notable event telemetry.

I also could provide feedback on every interaction, to ensure we are training the agentic models as each human analyst processes an incident.

scapel-5.png

Figure 4: Built-in human-in-the-loop feedback mechanisms enabling analysts to rate, validate, and refine agentic reasoning outputs in real time. Initial investigation summary page in Splunk Enterprise Security – enabling easy investigation pivots into source/destination IP, file hashes (if applicable), DNS resolutions, and evidence findings.

scapel-6.png

Figure 5: Single-click investigation menu in Splunk ES enabling instant pivots into Cisco Talos, VirusTotal, Endace Packet Forensics, and Firewall telemetry.

The ticket landed in my queue triggered by a high-severity Cisco Secure Firewall Security Intelligence event. An internal Intel-based laptop sitting on the conference subnet was generating outbound connections toward an external destination IP: 13.226.251.40.

According to Cisco Talos intelligence, the traffic was categorized under Exploits / Malicious Sites, associated with malicious domains including hxxps://moonlighthathel[.]org and hxxps://ukankingwithea[.]com.

scalpel-graph-2-use.png

Pivoting with a single click on the Destination IP, I could investigate across multiple SOC tools without leaving the Splunk ES interface:

scapel-7.png

Figure 6: Endace persistent PCAP investigation pivot into Layer 7 HTTP request headers and SSL/TLS SNI fields for all 15 active streams

3. The Conundrum: 15 Systems Calling 1 Destination IP

As soon as I started digging into the telemetry across Splunk Enterprise Security, I ran into an immediate puzzle: 15 separate source IP addresses across multiple conference subnets were actively communicating with 13.226.251.40.

If you're an analyst looking only at a static IP reputation score, your first reflex might be to drop a blunt hammer: block 13.226.251.40 at the perimeter firewall and call the ticket resolved. But in modern cloud networking, that kind of blunt response is dangerous.

When I pulled the reverse-DNS, SSL certificate metadata, and HTTP request headers for 13.226.251.40, the reality became clear: this IP is an Amazon CloudFront Content Delivery Network (CDN) edge node. It was actively delivering legitimate, essential enterprise services, including:

14 of the 15 devices communicating with 13.226.251.40 were doing nothing more than syncing their phone apps, checking training materials on Mindtickle, or streaming video. If I had applied a broad IP-level block on 13.226.251.40, I would have broken legitimate connectivity for dozens of innocent attendees across the convention center.

Pivoting to Endace grants the ability to visualize all of the 15 streams individually, and launch a full-featured Wireshark instance directly within the Splunk ES workflow:

scapel-8.png

Figure 7: Embedded cloud Wireshark session inspecting Layer 7 HTTP request headers and SSL/TLS SNI fields for all 15 active streams.

4. Isolating the Threat: The Forensic Evidence Chain

By filtering down to the individual session level, I was able to isolate the single machine driving the malicious activity: an unmanaged Intel laptop assigned internal IP [REDACTED: 10.x.x.x].

Unlike the other 14 benign endpoints, this specific machine was actively generating web requests that redirected through the CDN edge to Cloudflare-fronted exploit domains. Examining the timeline of events revealed a clear pattern of malicious outreach over the last 24 hours:

scalpel-graph-1.png

5. The Response: Precision DNS Object Blocking in Cisco Secure Firewall

Once the evidence was gathered, the remediation path was straightforward but required precision. Because this was an unmanaged attendee endpoint, we could not isolate the machine at the endpoint operating system level. The containment had to be executed on the network perimeter.

Remediation Recommendation & Enforcement

table-use.png

6. What This Means for Security Teams: SE Key Takeaways

Investigating ES_00244 in a high-density, live-production environment reinforced three core lessons that apply directly to enterprise networks, public sector campuses, and mission-critical OT/IoT environments:

scalpel-graph-5.png

scalpel-graph-6.png

scalpel-graph-7.png

Final Thoughts

Working the floor at the .conf26 Agentic SOC was an incredible experience. Seeing how autonomous intelligence and human analyst expertise come together to solve complex investigations in real time is a glimpse into the future of cyber defense. As our networks grow more distributed and cloud-dependent, the combination of rich telemetry, automated context, and precision enforcement will be what keeps our organizations resilient.

Check out the other blogs by my human colleagues in the .conf26 Agentic SOC.

Related Articles

Hunting for Threats in VPCFlows
Security
7 Minute Read

Hunting for Threats in VPCFlows

This article will look at native AWS network telemetry — VPCFlows. We’ll explore what it is, how you can ingest it, and what value it provides from a security perspective.
The Hidden Cost Of Downtime In Manufacturing
Security
4 Minute Read

The Hidden Cost Of Downtime In Manufacturing

Downtime in manufacturing is no longer just an operations issue. It is a business issue with direct impact on revenue, compliance, productivity, innovation, and customer satisfaction.
Staff Picks for Splunk Security Reading June 2022
Security
2 Minute Read

Staff Picks for Splunk Security Reading June 2022

Hello, everyone! Welcome to the Splunk staff picks blog. Each month, Splunk security experts curate a list of presentations, whitepapers, and customer case studies that we feel are worth a read. To check out our previous staff security picks, take a peek here. We hope you enjoy.