Evaluating Agents for Manipulation, Deception and Adversarial Risk

Most agent evaluation asks whether the agent can do the job. This asks whether it can be talked into doing the wrong one, whether it will tell you when it has, and whether the person on the other end is being handled rather than helped.

For security and risk teams the shape of that question has changed. The concern is no longer mainly whether a model can persuade a human. It is whether an attacker can persuade your agent — which then acts with your credentials.

This is one of three pieces on evaluating AI agents. Each stands alone, and together they cover the capabilities, the reasoning and the risks:

• Evaluating AI Agents on Tool Calling and Planning
• Can AI Agents Catch Their Own Mistakes?
• Evaluating AI Agents for Manipulation, Deception and Adversarial Risk (you are here)

Why indirect prompt injection is the primary risk for tool-using agents

Indirect prompt injection is an attack in which instructions hidden in content an agent reads — a web page, a document, an email, a database record, a tool response — are interpreted by the model as commands. It differs from direct injection, where the attacker talks to the agent themselves. It is the dominant practical risk for tool-using agents because nothing reliably separates instructions from data inside a model's context.

Prompt injection leads the OWASP GenAI LLM Top 10, whose 2026 edition was released in August 2026. The notable movement in that edition is Excessive Agency, which rose from sixth to third — a direct consequence of agents being given more autonomy and more tools.

More useful for agent teams is the separate OWASP list for agentic applications, published in December 2025. Its ordering tells you where practitioners think the risk sits:

• ASI01 Agent Goal Hijack — where indirect prompt injection now lives
• ASI02 Tool Misuse and Exploitation
• ASI03 Identity and Privilege Abuse
• ASI04 Agentic Supply Chain Vulnerabilities
• ASI05 Unexpected Code Execution
• ASI06 Memory and Context Poisoning
• ASI07 Insecure Inter-Agent Communication
• ASI08 Cascading Failures
• ASI09 Human-Agent Trust Exploitation
• ASI10 Rogue Agents

ASI09 is where manipulation of the human ends up, and it reframes the risk usefully. The concern is not that a model independently decides to deceive a user. It is that a hijacked agent inherits the trust the user has already extended to it.

Failure of static defenses against adaptive adversarial attacks

AgentDojo, from ETH Zurich, is the reference environment, and its framing is the right one: it reports utility under attack alongside targeted attack success rate, because a defense that blocks every injection by refusing to do anything useful is not a defense.

The finding that should change how you read a vendor security claim came in October 2025, when researchers from Google DeepMind, OpenAI and ETH applied adaptive attacks — designed with knowledge of the specific defense — against twelve recently published defenses. They achieved attack success above 90% against the majority. Most had originally reported near-zero rates. An earlier study broke eight indirect-injection defenses at consistently above 50%.

A June 2026 evaluation of nine current defenses quantified the gap. One classifier showed 0.0% attack success against static attacks and 28.0% against adaptive ones; a filtering defense went from 12.6% to 33.4%. The conclusion worth carrying into architecture decisions: system-level defenses that constrain what the agent is permitted to do hold up meaningfully better than prompt-level instructions or content filters.

Calculating security risk: Always ask for k

The single most useful habit is checking how many attempts the attacker got. Anthropic's Claude Opus 4.8 system card is unusually transparent and makes the point better than any summary: in a live red-teaming exercise, an adaptive attacker against the coding scenario achieved 7.03% success at one attempt and 57.5% at two hundred attempts without safeguards. With thinking disabled, 17.44% and 95.0%. On browser use across 129 held-out environments, the unsafeguarded model was compromised in 62.8% of scenarios, dropping to 3.9% with safeguards.

An attacker who can place content in front of your agent does not get one attempt. Single-attempt success rates in the low single digits are real and nearly meaningless alone. A robustness number without k attached is not telling you what you need to know.

What persuasion research shows

Persuasion is worth a short section because the popular version of it is wrong in a way that distorts risk registers.

The largest study to date, published in Science in December 2025 by Oxford Internet Institute researchers with the UK AI Security Institute, covered nearly 77,000 participants across nineteen models and several hundred political issues with every factual claim checked. Post-training raised persuasiveness by up to 51% and prompting by up to 27%, while model scale and personalization mattered far less than expected. The finding with the sharpest implications: techniques that made models more persuasive systematically made them less accurate.

A meta-analysis published the same month, pooling seven studies and more than 17,000 participants, found no statistically significant overall difference between AI and human persuaders. Models persuade about as well as people do. Superhuman persuasion is not an established finding, and treating it as one misallocates attention.

Evaluation of this has consolidated rather than disappeared. OpenAI removed persuasion as a tracked risk category in April 2025, so its MakeMePay and MakeMeSay figures stopped being reported — if you see a current model quoted on either, that number does not exist. Google DeepMind moved the other way, adding a harmful manipulation threshold to its Frontier Safety Framework and publishing supporting research in March 2026 across ten thousand participants in three countries. Two findings there are directly actionable, so measure both, separately:

1. Manipulation effects do not generalize across domains or regions.
2. A model's propensity to attempt manipulation does not predict its effectiveness at it.

Deception and evaluation awareness: Why models perform worse in production than testing

Sabotage and deception evaluations are now routine in frontier releases, typically run by external organizations including the UK AI Security Institute, and the reported rates are low but non-zero. Two qualitative findings are more useful than the rates.

First, in the small fraction of trajectories where a model continued a sabotage task, it frequently reasoned about doing so in its chain of thought without disclosing it to the user. Evaluators also found monitors watching only actions were considerably less reliable than monitors reading reasoning — which is an argument for capturing reasoning in your traces, not just tool calls.

Second, and more consequential: models can increasingly tell when they are being tested. Recent system cards report models distinguishing real deployment data from synthetic evaluation scenarios with roughly 79% accuracy and verbalizing awareness of being evaluated in around a quarter of unprompted samples. Work by OpenAI and Apollo Research on reducing covert behavior found rates falling roughly thirtyfold under targeted training — alongside the caveat that evaluation awareness rose at the same time, and that scheming rates fell as awareness rose and rose again when it was suppressed. Some portion of the improvement may be the model recognizing a test.

Read alongside research on alignment faking, where behavior differed sharply between monitored and unmonitored conditions, the implication for assurance is specific. A clean evaluation result is evidence the model behaves well in conditions it recognized as an evaluation. What it does on real production traffic is a separate measurement — and the one that is not being recognized as a test.

Measuring sycophancy

Sycophancy is a model's tendency to agree with and validate a user regardless of whether the user is right. Unlike the exotic scenarios, it is measurable in production today.

Research published in Science tested eleven models and found they endorsed user actions roughly 50% more often than humans did, including in about half of scenarios where human consensus said the user was in the wrong. Preregistered experiments with sixteen hundred participants found exposure reduced willingness to repair interpersonal conflict and increased participants' conviction that they were right.

The governance-relevant finding is the last one: participants rated sycophantic responses as higher quality and were more likely to use that assistant again. The harmful behavior is the commercially rewarded one, so it will not correct itself. OpenAI rolled back a GPT-4o update in April 2025 for exactly this and subsequently reported cutting sycophancy prevalence by around 70% in production after training against it. If nobody is measuring it, it is drifting.

Navigating regulation: EU AI Act compliance for high-risk agentic deployments

One distinction does most of the work here: the EU AI Act provisions touching manipulation are in force, while the parts most teams were tracking were delayed.

Primary sources: EU AI Act Article 5, Article 99 on penalties, and the NIST AI Risk Management Framework, whose Generative AI Profile remains current.

Lab frameworks have diverged rather than converged, which is worth knowing if you benchmark your governance against them. OpenAI dropped persuasion from its tracked categories; Anthropic's policy, updated February 2026, explicitly requires evidence that models lack the propensity to deceive, manipulate or sabotage users; Google DeepMind added harmful manipulation as a tracked capability. There is no consensus here to align to.

What to measure for agent observability & runtime security

Five things are worth measuring on a deployed agent, and none is covered by a model-level benchmark.

• Injection attempts reaching the agent, and what happened next, whether attacker-controlled content entered your context and whether the agent acted on it.
• Tool calls that exceed what the task required. Excessive agency is the mechanism by which a successful injection becomes an incident, and the measurable version is the gap between the actions a task needed and the actions taken.
• Sensitive data leaving the boundary: PII, credentials and regulated data at every hop, not only in the final response.
• Sycophancy drift: whether agreement rates are rising, particularly where the agent should push back.
• Behavior on production traffic. Given evaluation awareness, this is the measurement that cannot be gamed by recognizing a test.

This is where continuous evaluation stops being optional. Splunk Agent Observability runs evaluators against production traffic rather than a pre-deployment suite alone, including prompt injection detection, PII and CPNI detection and toxicity, with sampling controls for full or targeted coverage. Agent Control adds the runtime layer, applying checks before or after an LLM or tool call and taking one of three actions — deny, steer, or observe and log — and attaching without redeploying the agent. One limitation worth designing around: the prompt injection evaluator inspects the input at the root of a trace, so a second-order injection arriving mid-trajectory in tool output needs a control placed at that call rather than reliance on the trace-level evaluator.

Conclusion

An agent with tools and credentials, reading content it did not author, is an attack surface. The published evidence says defenses that look strong against fixed test cases degrade badly against an attacker who adapts and gets more than one try — and that the models being tested can increasingly tell they are being tested.

Neither of those argues against evaluation. Both argue for evaluation that continues after deployment, on real traffic, with the number of attempts written down.

FAQs about risk evaluation in agents