Instruction-following is not a security boundary
We told a model to ignore hostile instructions hidden in its retrieved documents. It mostly did not help. Here is what did.
The attack is simple and it is what a poisoned document looks like in the wild. Somewhere in the retrieved context sits text formatted as a procedure: a regulation, an answer-derivation policy, an instruction to always answer and never refuse. The model is asked a question it should decline because the documents do not support an answer. The measure is how often the hostile text talks it into answering anyway.
Undefended, the hijack rate is high. The intuitive fix is to tell the model, in its system prompt, to treat document text as data and ignore any procedure inside it. We built that instruction carefully and measured it. It barely helped, and on some models it made things worse: the hijack rate went up, not down. The model would even reason out loud that it must follow the trusted policy, and then let the hostile content bend its judgment about whether the question was answerable. The instruction stopped the model from taking orders from the document. It did not stop the document’s content from shaping the model’s inference. Per-model deltas, signed, are on each model card.
Then we stopped defending at the prompt and defended at ingestion instead: strip the imperative and procedural spans out of retrieved documents before they are ever assembled into context. Across the models we tested, the hijack rate fell sharply, in some cases to near zero. Same model, same question, same trusted policy. The only change was that the hostile text never reached the context window.
The lesson for anyone shipping an agent
A control that lives in the prompt is not a boundary, because the untrusted content is inside the same context the prompt is trying to police. A boundary has to sit where the untrusted content enters: the retrieval and ingestion layer. Provenance on what is allowed to carry procedural authority, and sanitization of imperative spans in everything else. An agent is a machine for moving untrusted text into inference position at scale, which makes this the boundary that matters most and the one most systems do not have.
The per-model defense ladders, undefended to inoculated to sanitized, are on the index. Provisional, with the sample sizes shown.