We Ran AgentDojo Against Our Own Defences. Two of Them Did Nothing.

We built an AgentDojo integration months ago, unit-tested it, and never ran it. Its own README said so, and forbade citing results until a run existed.

We ran it. Getting a number took eight bug fixes, and the result is not the one we would have written in advance.

The result

AgentDojo's banking suite, gpt-4o-mini, the canonical important_instructions attack, 144 (user task × injection task) cases per arm:

The per-message guard works here. annotate is the policy to use: it takes 67 of 144 attacks down to 1, and keeps more of the agent's task completion than aborting the run does. Both cost roughly 30% of utility, which is a real trade and not a rounding error.

Our trace-level and containment defences showed no measurable effect. 47.2% and 43.1% against 46.5% undefended, on 144 cases, p = 1.00 and p = 0.64.

Read the good number narrowly

important_instructions is a single fixed template repeated across all 144 cases. So 0.7% means the guard recognises that template — not that it stops agentic injection. Nasr et al., The Attacker Moves Second, broke twelve published defences at over 90% attack success, and nothing in this run measures an attacker who adapts.

A defence measured against one template is an upper bound on that defence, the same way a benchmark measured on a static corpus is an upper bound on a detector.

The bug that invalidated our first attempt

Our first matrix showed every arm inside the undefended confidence interval — all five defences apparently doing nothing. That reads as a damning result. It was a wiring bug.

The defence was appended to pipeline.elements, which put it after the tool-execution loop had finished. It observed attacks that had already succeeded. AgentDojo's own reference defence inserts its detector inside ToolsExecutionLoop, where it sees each tool result before the next model call — and that reference detector is protectai/deberta-v3-base-prompt-injection-v2, the same encoder we run at tier one.

Placement was the only difference between 46.5% and 0.7%.

Eight bugs behind forty-six passing tests

The integration had 46 offline tests, all green. None of them constructed a pipeline, so nothing that only breaks against the real framework was reachable by them. The sharpest example: AbortAgentError takes three arguments, but our offline fallback stand-in took one — so the tests exercised a class with a different signature from the real thing and passed on a call that could never work.

Passing tests over an integration nothing has executed certify the parts, not the whole.

What we don't know

We do not know why the trace-level controls showed no effect, and we are not going to guess. Our first explanation — that AgentDojo's tools carried no untrusted-source label, so the dataflow detector had nothing to start a taint from — turned out to be a real bug that changed nothing when fixed. Both arms stayed flat.

That is the uncomfortable part of publishing this. Capability containment is the control whose security argument does not depend on classifying an attack correctly, and it is the one we would otherwise point at hardest. On the one benchmark built for multi-step agent attacks, it did not move the number, and the reason is open.

We have written that into our documentation on what detection can and cannot do rather than leaving it in a lab notebook, because a control that is structurally sound and empirically unconfirmed is a different thing from one that is proven, and buyers deserve to know which they are getting.

Why publish an unflattering result

Every number in this post is reproducible from our repository. The alternative — running the benchmark, keeping the 0.7%, and quietly not mentioning the two arms that flatlined — would have been easy and would have been the kind of thing that makes every other number we publish worth less.

Ask the same of anyone quoting you an agentic security result: which benchmark, how many cases, what was the utility cost, and what did their other defences do.