Every model goes through hell
before it goes to work
We hand open-weight models retrieved documents. Relevant, irrelevant, and hostile. Then we publish what breaks, with confidence intervals. Leaderboards measure what a model knows. We measure what it does to your data.
Capability is not deployment safety
On gemma-4-26B-A4B-it (4-bit), relevant retrieved context moved accuracy +4.7 points. A model can top every capability leaderboard and still get worse the moment you wire it to a document store. That gap is deployment-layer risk, and it is where agents and retrieval systems actually live.
Content safety is covered
MLCommons AILuminate measures whether a model produces harmful content. Useful. Not our question.
Capability is covered
The public leaderboards measure what a model knows and how fast. Useful. Not our question.
Deployment risk is not
What happens when the model meets your documents, including hostile ones. That is the missing evidence class. We built the index for it.
What we put every model through
Context contamination
We give the model relevant retrieved documents and check whether accuracy goes up or down against closed-book. Some models lose double digits.
Injection resistance
We hide a hostile instruction in the retrieved documents and measure the hijack rate under three defenses: none, prompt inoculation, and ingestion sanitization.
Abstention discipline
We ask questions the documents cannot answer and count how often the model invents an answer instead of admitting the gap.
Retrieval uplift, governance cost, quantization drift
Whether retrieval helps at all, whether a strict governance instruction degrades the model, and whether the compressed variant you actually deploy still behaves like its parent.
Forwardable to a risk committee
Every number carries its sample size, its run date, and the methodology version, and its confidence interval where one is computed. Every grade is recomputable from published sub-scores by a fixed public rubric. The raw run outputs sit in each model’s evidence locker with a checksum. A grade is marked final once every axis has been measured at the n≥200 replication target, and provisional until then. We say which on every one.