Splunking the LLM Gateways. Search Head Meets Prompt Head

A LLM gateway is a centralized layer between applications, users, agents and AI models that routes prompts and responses to providers which may be on the cloud or local. LLM Gateways are critical for observability as they become a chokepoint for model traffic because all requests pass through it, allowing one place to enforce authentication, access controls, logging, rate limits, cost controls, logging, rate limits, content filtering, model selection and security monitoring.

For SOC and SIEM teams, prompts and responses represent a new, high-value telemetry source for threat detection, governance, and incident response. LLM Gateways provide these teams with a way to solve the challenge of visibility for LLM/Agentic operations.

The goal is to capture the core metadata surrounding every request that passes through the LLM gateway — including tenant and application identity, the user making the call, the model and provider serving it, the request route, HTTP status, latency, token usage, estimated cost, and the regions where processing occurs. Together these fields give full visibility into who is using which models, how the traffic is routed, what it costs, and where the data is handled — the foundation for monitoring, governance, and detection across the gateway. These are also enriched with LLM specific telemetry, including prompt hashes or content classifications, safety-policy decisions, guardrail outcomes and indicators showing whether retrieval augmented generation, agents, plugins or external tools were invoked.

Agentic Observability

As stated, previously, observability of Agents is challenging. LLM Gateways significantly help address those challenges by providing the following:

• - Unified multi-model visibility. Normalize traffic across different models and providers into a consistency telemetry stream, eliminating fragmented logs.
• Prompt and response tracking. Capture prompts, completions, model selections, routes, retries, and correlation IDs so analysts can reconstruct the LLM portion of an agent workflow.
• - Tool, RAG and agent invocation context. Record model-requested tool calls, function names, arguments, RAG usage, and agent delegation indicators to show how a model decision led to an action.
• - Performance, loop, and cost monitoring. Track latency, token consumption, error status, provider usage, and estimated cost per request to identify bottlenecks, runaway loops, and expressive workflow stages.
• - Centralized security policy and provenance. Apply and log authentication, guardrails, content classifications, safety decisions, and blocked requests at a common enforcement point, creating evidence of who used which model, with which prompt, under which policy and with what outcome.

In the following graph, we can see an example of the breakdown of events by Agentic workflow stages and sessions.

SOC & LLM Gateways

In short, in a properly instrumented agentic architecture, LLM Gateways can capture pipeline stages, agent handoffs, tool calls, RAG retrievals, memory access, downstream actions and resulting artifacts going beyond just prompt inputs and responses. By correlating these events under a common trace or transaction ID, the gateway gives SOC and SIEM teams the possibility of end-to-end visibility into how an agent reached a decision and what actions it performed. SOC teams need LLM gateway telemetry that goes beyond classic performance metrics and provides the context to investigate, govern, and secure model activity. Effective logs should identify the user/agent, service account, application, tenant, model, provider, and region involved in every request, while also recording the policy evaluated, its version, allow or deny decision, guardrail outcomes, and any associated approval or exception.

In the above graphic, we can see an analytic of Agentic workflows broken down by Agent Role, pipeline stage, tool names, and guardrail disposition.

Operationalizing LLM Gateway Telemetry in Splunk

The TA LLM Gateway brings LLM gateway telemetry into Splunk and turns it into a consistent, searchable data source for SOC, platform and governance teams. The technology add-on currently normalizes traffic from Bifrost and LiteLLM into their specific sourcetypes, using shared fields for model, provider, token usage, cost, latency, tool calls, retries and request failures. It also includes event types for errors, high-token activity, off-hours usage and retries, providing a foundation for usage auditing, cost monitoring, and detections aligned with popular LLM security frameworks such as OWASP and Mitre ATLAS.

In the following graph, we can see the analytics of requests by LLM provider and the number of tokens used by request operations.

We can delve further into the different LLMgateways and responses by model. As can be seen in the following screenshot, we have online models such as claude-sonet.4.6 and gpt-40-mini.

We can dig even further and see the actual input prompts and outputs by model and gateway.

Detection and Monitoring Use Cases for LLM Traffic

The TA LLM Gateway enables detection engineering for LLM traffic by normalizing Bifrost and LiteLLM telemetry into a shared Splunk field model covering prompts, models, providers, token usage, latency, cost, retries, failures, and tool calls. SOC teams can use this data to identify repeated jailbreak attempts, suspicious prompt patterns, unexpected model access, abnormal off-hours activity, excessive token consumption, and unusual sequences of failed or retried requests. These detections directly support monitoring for the OWASP Top 10 for LLM Applications, including LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency, LLM07 System Prompt Leakage, and LLM10 Unbounded Consumption.

For agentic LLM environments, gateway telemetry can also expose suspicious tool invocation, unexpected provider routing, high-risk model changes, abnormal request chains, and attempts to move retrieved information into external tools or services. Splunk correlation searches can connect the requesting user or application, model interaction, tool selected, token volume, response status, and downstream action into a single investigation timeline. This visibility helps SOC teams operationalize MITRE ATLAS techniques such as LLM Prompt Injection, AI Agent Tool Invocation, context or tool-data poisoning, collection, and exfiltration, while also providing continuous monitoring for policy violations, compromised agents, and unauthorized use of AI capabilities.

Here is another example of using LLMGateway data leveraged queries using CSA MAESTRO . MAESTRO is a layered threat-modeling framework for identifying and organizing security risks across AI models, data, agents, infrastructure, and multi-agent ecosystems. The graph below shows which layers of the AI stack were targeted by observed attack activity.

The following example is a search detecting prompt injection against online models such as gpt-40-mini and claude-sonnet-4.6 via litellm gateway. This search displays input prompt and output response.

The Splunk Threat Research Team also has several Analytic Stories covering diverse vendors and AI tools such as MCP analytics, Ollama, M365 Copilot, AWS Bedrock Claude and Shadow AI. With the advent of LLM Gateways, it is now possible to develop detections against most models either online or local.

AI Observability Meets LLM Gateways

AI observability requires visibility into how models, agents, tools, and data interact across an AI workflow, not just whether the application is available. LLM gateways provide a strategic observation point where organizations can capture user and application identity, selected models and providers, prompts and responses, latency, token consumption, cost, retries, failures, routing decisions, retrieved data, and tool calls. By normalizing this telemetry in Splunk, the new TA LLM Gateway helps teams reconstruct the complete path of an AI transaction, identify performance bottlenecks, abnormal token growth, unexpected provider changes, unauthorized tool usage, and agent behavior that deviates from policy.

It is now possible to gain a broader, more consistent perspective on AI operations and agentic workflows across supported LLM gateways and model providers, provided that the necessary telemetry is available. Find the TA LLMGateway at splunkbase and related AI Detection Content at research.splunk.com