What Is Agent Monitoring? Watch Traces, Evals, and Token Cost
Learn Shanika WickramasingheKey Takeaways
- Agent monitoring is how you watch an AI agent while it runs a task: the trajectory of spans, the tools it called, and what the run cost.
- Agent monitoring is different from APM. Dashboards can look healthy while a support agent refunds the wrong order, invents a tool, or loops.
- Start with one live agent. Put spans, sessions, evals, and token counts on the same run so a finished refund is still inspectable.
Agent monitoring is how you watch an AI agent in production: the trajectory it took, the tools it called, whether the output held up, and what the run cost in tokens. You capture that on every task so a finished request is still inspectable.
Why production agents need agent monitoring
Agent monitoring belongs in production because an agent can return a reply and still miss the job. In this article, we'll take a practical look at what to capture, how that differs from APM, and how sessions and spans make a refund path inspectable.
For example, you've shipped a customer-support agent that looks up orders and issues refunds. One run calls the order API and writes a clean reply. The next run invents a lookup tool that isn't in the allowlist, or loops on the same search until tokens spike. Application performance monitoring (APM) still looks green. The customer can still get a refund on the wrong order.
Agent monitoring vs. APM
Agent monitoring differs from APM because APM reports service health, while agent monitoring reports whether the agent used an allowed tool, stayed on intent, and completed the job. Service health can look fine on a refund ticket while the agent still refunds the wrong order.
Leave APM on the backends the agent calls, even when P50 looks fine.
What agent monitoring should tell you
Agent monitoring should tell you whether the run was reliable, which tools fired, what it cost, and where it drifted from policy. A fluent refund reply can still be wrong, unauthorized, or stuck in a loop.
If any row is blank today, that refund agent is running unwatched.
Layers of agent monitoring
Agent monitoring works as four layers: the model, the workflow path, output quality, and the infrastructure underneath. A refund that looks slow can be a stalled order API, a five-call loop, or a bad plan that still reads fluent.
LLM observability records prompts and completions. Distributed tracing is the usual shape: nested spans for retrieval, tool calls, and the reply. Model monitoring scores the live output. Evals belong on the same run as the spans, not in a quarterly notebook.
Telemetry agent monitoring captures
Agent monitoring captures four kinds of telemetry from each run: metrics, logs, events, and traces. Each type answers a different question on the same refund: what it cost, what was said, which tool fired, and which span went sideways.
Events catch a tool the allowlist never had. Traces put token use on the run, not only the invoice.
How an agent monitoring loop runs
An agent monitoring loop instruments the agent, records the trajectory, scores the output, and feeds what you learned into the next run. Let's walk through that loop.
- Add instrumentation. Emit prompts, tool choice, auth, latency, and tokens from each span. If the agent is a graph, instrument the nodes and edges, not only the final reply. OpenTelemetry is a common way to export those spans without a vendor lock.
- Capture the trajectory. Store spans in order and group turns into a session so the refund chat is one story. That's how you reconstruct a run that "succeeded" and still refunded the wrong order.
- Watch the run metrics. Track P50 and P95 latency, error rate, tokens per task, and tool-call rate. Watch for spend that doesn't match ticket volume.
- Score the output. Use LLM-as-a-judge on the misses that would cost a customer, and keep HITL (human-in-the-loop) on high-stakes actions such as a refund.
- Detect, investigate, and improve. Watch reasoning loops, hallucinated tools, and failed tool auth. Read the spans with the logs, then change the prompt, the allowlist, or the eval.
Sampled traffic will miss the cheap loop that burns tokens overnight.
Agent monitoring across sessions and multiple agents
Agent monitoring across sessions and multiple agents has to follow intent and handoffs, not only one reasoning loop.
A support ticket is often multi-turn. The customer starts with a shipping question, then asks for a refund. Group those turns in a session and watch whether intent held. If the agent keeps searching tracking data after the goal moved, the last sentence can look fine and the job still failed.
The same ticket can fan out: a retriever pulls the order, a classifier names the miss, a closer posts the refund. Map the edges, roll token cost up across the graph, and watch two agents taking opposite actions on the same ticket.
Agent monitoring vs. AI agent observability
Agent monitoring is the operational watch on each run: trajectory, cost, and quality while the agent works. AI agent observability puts evals on those same records so you can grade the decision, not only the completion.
A single completion is an LLM-observability job. A planner that selects tools or continues a session needs agent monitoring on the whole path.
Next step: Read AI agent observability when you're ready to grade those refund traces, not only store them.
FAQs about Agent Monitoring
Related Articles

The Chief Product Officer Role: CPO Responsibilities, Salary & Skills

What Is Digital Forensics? The Weapon Against Cybercrime
