CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make a red-team exercise hollow

Common mistakes, and what the example does not do.

Mistakes that make a red-team exercise hollow

  • A handful of famous attacks instead of a grid.
  • One run per attack against a stochastic system.
  • Fixing and testing on the same attacks.
  • Reporting a point estimate without an interval or attempt count.
  • Treating 0 percent as proof of safety.
  • Text-only fixes that attackers rephrase around.
  • No regression suite, so fixed bugs return.
  • Ignoring severity, celebrating a drop in harmless attacks.
  • Testing production with real customer data or without permission.

What the example does not do

About this example What the example does not do
  • No real model: attack outcomes come from assumed susceptibilities, so the text-defence numbers do not describe real agents.
  • The gate is deterministic code in the harness, so its 0 percent is a property of that code on the catalog, not of any deployed system.
  • A small, single-author attack grid; no multi-turn conversation attacks, no tool-chaining, no memory poisoning.
  • Attack runs are independent; real attackers adapt after each failure.
  • No human red team and no scoring of the model's text output for harmful content.
  • Optional real experiment: run the same grid against a real agent with mocked tools, record ASR with intervals, apply one structural and one text fix, and compare on held-out families.

Check yourself

An attack set scores 0 percent success. Is the system safe?

Show answer

No. It only means no known attack worked. At a true 1 percent success rate you need about 299 attempts to be 95 percent sure of seeing one.

For the curious Fixes that learn the attacks you saw overfit

A word-3-gram blocklist learned from 3 phrasing families cut success on those to 5.2 percent but only to 9.3 percent on 3 held-out families (22.3 percent undefended). A gate on the action does not care about phrasing: 9.0 and 9.3 percent. Adding per-customer scoping and no automatic refunds brought it to zero on this grid. The target is an imitation, so only the gate result is a property of the code.

What you will use daily at work

Practice What it looks like
Generate attacks from a grid Goals, carriers, phrasing and obfuscation.
Report success rate with an interval And weight it by severity.
Hold out phrasing families Test fixes on attacks they were not built from.
Keep every success as a regression test Fail CI only when the lower interval bound exceeds your limit.