Part 10 of 12 · 1 min read
Quick notes
- Red-teaming is structured, repeatable attack practice on your own system, run as a lifecycle: implement, attack, evaluate, mitigate, repeat.
- Build a generator, not a list. Crossing goals, carriers, phrasing families and obfuscations shows which category fails, not just that something did.
- Report an attack success rate with a confidence interval, and weight it by severity: a high rate on low-harm goals is a different risk from a low rate on data exfiltration.
- Fixes that learn the attacks you saw overfit them. A blocklist did well on training attacks and worse on held-out phrasing families; a structural gate on the action did not care about phrasing.
- Zero is not safe. A zero rate says no known attack worked. Keep every successful attack as a regression test and fail the build only when the interval's lower bound exceeds the allowed rate.