Part 9 of 12 · 2 min read
Mistakes that make a red-team exercise hollow
Common mistakes, and what the example does not do.
Mistakes that make a red-team exercise hollow
- A handful of famous attacks instead of a grid.
- One run per attack against a stochastic system.
- Fixing and testing on the same attacks.
- Reporting a point estimate without an interval or attempt count.
- Treating 0 percent as proof of safety.
- Text-only fixes that attackers rephrase around.
- No regression suite, so fixed bugs return.
- Ignoring severity, celebrating a drop in harmless attacks.
- Testing production with real customer data or without permission.
What the example does not do
About this example What the example does not do
- No real model: attack outcomes come from assumed susceptibilities, so the text-defence numbers do not describe real agents.
- The gate is deterministic code in the harness, so its 0 percent is a property of that code on the catalog, not of any deployed system.
- A small, single-author attack grid; no multi-turn conversation attacks, no tool-chaining, no memory poisoning.
- Attack runs are independent; real attackers adapt after each failure.
- No human red team and no scoring of the model's text output for harmful content.
- Optional real experiment: run the same grid against a real agent with mocked tools, record ASR with intervals, apply one structural and one text fix, and compare on held-out families.
Check yourself
An attack set scores 0 percent success. Is the system safe?
Show answer
No. It only means no known attack worked. At a true 1 percent success rate you need about 299 attempts to be 95 percent sure of seeing one.
For the curious Fixes that learn the attacks you saw overfit
A word-3-gram blocklist learned from 3 phrasing families cut success on those to 5.2 percent but only to 9.3 percent on 3 held-out families (22.3 percent undefended). A gate on the action does not care about phrasing: 9.0 and 9.3 percent. Adding per-customer scoping and no automatic refunds brought it to zero on this grid. The target is an imitation, so only the gate result is a property of the code.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Generate attacks from a grid | Goals, carriers, phrasing and obfuscation. |
| Report success rate with an interval | And weight it by severity. |
| Hold out phrasing families | Test fixes on attacks they were not built from. |
| Keep every success as a regression test | Fail CI only when the lower interval bound exceeds your limit. |