Relevant context made a 31B model 13 points dumber
Retrieval is supposed to help. On some models it does the opposite, and the loss is large enough to change how you budget a RAG system.
Here is the experiment. Take a set of multiple-choice questions the model can answer from memory. Ask them closed-book and record the score. Now retrieve the documents that are genuinely relevant to each question, put them in the context, and ask again. Retrieval is meant to raise the score. On one widely deployed 31B open-weight model, it dropped the score by 13 points.
The obvious objection is that the extra text is just noise, and long prompts distract any model. So we ran the control: equal-length documents pulled for a different question, irrelevant to the one being asked. If distraction were the cause, irrelevant context would hurt as much as relevant context. It did not. Irrelevant context barely moved the score. Only relevant, plausible, nearly-right documents caused the collapse.
That points at the mechanism. The harm scales with how semantically close the retrieved text is to what the model already knows. Plausible related text gets absorbed into the model’s answer and overrides its own correct recall. Unrelated text is easy to dismiss. When we counted the flips, a correct closed-book answer was overturned by the context far more often than a wrong answer was repaired by it.
Why this is a budget problem, not a curiosity
If you are paying for a retrieval pipeline on the assumption that more context is more accuracy, this inverts the assumption for an entire class of model. The fix is not more retrieval. It is choosing a model whose contamination profile is flat, or adding an instruction that lets the model discard unhelpful context, and then measuring whether that instruction actually worked on your model, because on some models it helps and on others it does nothing.
The full per-model numbers, with confidence intervals and the random-context control, are on the index, each marked provisional or final by its sample size. On the model in question the 95% interval for the effect excludes zero, so the loss is not a sampling artifact.