Covers: Definitions; The loop; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Red team, blue team
Red team: people (or programs) that attack the system to find weaknesses.
Blue team: those who defend, monitor and fix.
Attack success rate, severity weighting
Attack success rate (ASR): the share of attack attempts that achieve the attacker's goal.
Severity-weighted ASR: successes weighted by the harm of each goal.
Held-out attacks, regression suite
Held-out attacks: attacks not used to design the fix, used to test it.
Regression suite: every attack that ever worked, kept and re-run so a fix cannot silently disappear.
Wilson interval, attempts needed
Wilson interval: a confidence interval for a proportion that behaves well near 0 and 1.
Attempts needed: how many independent tries are required to see at least one success at a given true rate.
The loop
Generate, attack, measure, fix, re-test with held-out attacks and a regression suite.
Results
Target
ASR train families
ASR held-out families
Severity-weighted (all)
No defence
20.7%
22.3%
21.9%
Blocklist learned from train attacks
5.2%
9.3%
7.5%
Gate v1 (block email, delete, large refund)
9.0%
9.3%
4.0%
Gate v2 (+ per-customer scope, refund cap 50)
0.0%
0.0%
0.0%
True success rate
Attempts for 95% chance of one success
10%
29
5%
59
1%
299
0.1%
2,995
CI rule: fail if the interval's lower bound exceeds 2% (or severity-weighted exceeds 2%)
Exit code
No defence
1
Blocklist
1
Gate v1
1
Gate v2
0
Attack success rates by defence on train and held-out attack families, with intervals and regression suite sizes.
A text fix generalises less than a structural fix. The blocklist's held-out ASR was higher than its train ASR; the gate's was the same on both.
Residual risk sits in what the gate still allows, so the next fix targets the allowed-but-abusable actions.
Interval-based gates avoid flaky builds and still catch confident regressions.
Attempts, not percentages, decide how much you can claim: 300 clean attempts do not show a rate below 1 percent with certainty.
Design rules
Generate attacks from a grid of goals, carriers, phrasings and obfuscations, and add new cells when threat modelling (module 7) finds new goals.
Run each attack many times (the model is stochastic) and report ASR with an interval.
Split attacks into design and held-out sets before fixing, and report both.
Prefer structural fixes (module 2 least privilege, module 3 sandbox, module 5 approvals) and measure them like any other.
Keep a regression suite of every successful attack and gate CI on it.
Weight by severity and report which goals remain.
State what zero means (attempts and coverage), not "secure".
Use human red-teamers and tools for creativity (automated generators cover known patterns); rotate them and share findings.
Re-run on every change to model, prompt, tool or data, and on a schedule.
Disclose and track findings like bugs, with owners, severities and fix verification.
Test responsibly: only systems you own or have written permission to test; no real personal data in attack sets.
Interview questions, with the follow-ups
How do you red-team an agent?
"I define the attacker's goals from the threat model, generate attacks as a grid of goals, carriers, phrasings and obfuscations, run each many times against the real agent with tools mocked, and report attack success rate with intervals, per category and weighted by severity. I fix, re-run on held-out attacks, and keep every successful attack in a regression suite that runs in CI." Follow-up: "Which tools?" Open-source frameworks the book names, such as DeepTeam, Garak and PyRIT, plus manual creative attacks; I would evaluate them on my own agent rather than trust their scores.
Your fix drove ASR to 0. Are you done?
"No. Zero means no known attack worked in the attempts made. At a true rate of 1 percent I need about 300 attempts to see one success. I check coverage of goals and carriers, evaluate on held-out attacks, and ask whether the fix is structural or just matches the attacks I saw: my learned blocklist went from 5 percent on training attacks to 9 percent on held-out ones." Follow-up: "How do you avoid overfitting?" Split attacks before designing fixes, rotate attackers, and prefer controls that do not depend on wording.
How do you put red-teaming in CI?
"A fixed regression suite of past successful attacks plus a sample of generated ones, run several times each, with a gate on the interval: fail only if the lower bound of the success rate exceeds the allowed rate, and track severity-weighted rate. That avoids failing on noise, and catches confident regressions." Follow-up: "How often do you run the full grid?" Nightly or on changes to model, prompt or tools; the quick suite on every pull request.
Automated or human red-teaming?
"Both. Automation covers known patterns at scale and gives measurable rates; humans find new goals and chains of steps, such as using one tool's output to steer another. I feed human findings back into the generator and the regression suite." Follow-up: "Who should do it?" People who did not build the agent, with permission and a scope.
Common mistakes
A handful of famous attacks instead of a grid.
One run per attack against a stochastic system.
Fixing and testing on the same attacks.
Reporting a point estimate without an interval or attempt count.
Treating 0 percent as proof of safety.
Text-only fixes that attackers rephrase around.
No regression suite, so fixed bugs return.
Ignoring severity, celebrating a drop in harmless attacks.
Testing production with real customer data or without permission.
What the example does not do
No real model: attack outcomes come from assumed susceptibilities, so the text-defence numbers do not describe real agents.
The gate is deterministic code in the harness, so its 0 percent is a property of that code on my catalog, not of any deployed system.
A small, single-author attack grid; no multi-turn conversation attacks, no tool-chaining, no memory poisoning.
Attack runs are independent; real attackers adapt after each failure.
No human red team and no scoring of the model's text output for harmful content.
Optional real experiment (I did not run this): run the same grid against a real agent with mocked tools, record ASR with intervals, apply one structural and one text fix, and compare on held-out families.
Check yourself
An attack set scores 0 percent success. Is the system safe? (No. It only means no known attack worked. At a true 1 percent success rate you need about 299 attempts to be 95 percent sure of seeing one.)
For the curious Fixes that learn the attacks you saw overfit
A word-3-gram blocklist learned from 3 phrasing families cut success on those to 5.2 percent but only to 9.3 percent on 3 held-out families (22.3 percent undefended). A gate on the action does not care about phrasing: 9.0 and 9.3 percent. Adding per-customer scoping and no automatic refunds brought it to zero on this grid. The target is an imitation, so only the gate result is a property of the code.
What you will use daily at work
Practice
What it looks like
Generate attacks from a grid
Goals, carriers, phrasing and obfuscation.
Report success rate with an interval
And weight it by severity.
Hold out phrasing families
Test fixes on attacks they were not built from.
Keep every success as a regression test
Fail CI only when the lower interval bound exceeds your limit.
Draft: confirm legal approval for any real red-team exercise and which tools your team uses.
What you must be able to say (one page)
Red-teaming is structured, repeatable attack practice on your own system. Design adversarial scenarios that mimic real attacks, such as prompt injection and jailbreaking, plus stress conditions (ambiguous or contradictory instructions, high-stakes contexts, data leakage), and run them as an iterative lifecycle: implement, attack, evaluate, mitigate, repeat. It must be ongoing because fine-tuning, updates and new deployments change the security profile. Open-source tools exist, such as DeepTeam, NVIDIA's Garak and Microsoft's PyRIT; I did not run any of them.
Build a generator, not a list. My harness crosses 5 attacker goals (exfiltrate data, delete an order, large refund, small refund, read another customer's record) with 3 carriers (direct message, web page, tool result), 6 phrasing families and 3 obfuscations: 270 attacks. Systematic coverage shows which category fails, not just that something did.
Report an attack success rate (ASR) with an interval. Each attack was run 20 times against an imitation agent (assumed susceptibilities per phrasing family). Undefended: 22.1 percent overall (95 percent Wilson interval 21.1 to 23.3), severity-weighted 21.9 percent.
Fixes that learn the attacks you saw overfit them. I learned a blocklist of word 3-grams from the 135 training attacks (3 phrasing families). On the training attacks ASR fell to 5.2 percent; on the held-out families (roleplay, multi-step, translated) it was 9.3 percent, against 22.3 percent undefended. The gap is modest here only because my families share goal wording ("customer list"); a real attacker changes the wording more. Always evaluate on attacks the fix has not seen.
Structural fixes do not care about phrasing. A gate on the action (block external email, deletion and large refunds) gave 9.0 and 9.3 percent on train and held-out, severity-weighted 4.0 percent: what remained were the actions it still allowed (a small refund and reading another customer's record). Adding per-customer scoping and a refund cap (gate v2) gave 0.0 percent, interval 0.0 to 0.1.
Zero is not safe. ASR measures the attack set, not the system. If the true success rate is 1 percent you need about 299 attempts to be 95 percent sure of seeing one success (2,995 for 0.1 percent; 59 for 5 percent). Gate v2's zero says "no known attack worked", and my goals map to actions the gate knows about; a new goal outside the catalog would not be tested.
Keep every successful attack as a regression test. Undefended, 260 of 270 attacks succeeded at least once in 20 runs; after gate v1, 104; after gate v2, 0. Run the suite in CI (Level 9 module 7) and fail on a confident regression: my gate returns exit code 1 only if the lower bound of the interval exceeds the allowed rate, so noise alone does not fail the build.
Weight by severity. A 22 percent ASR dominated by low-harm goals is a different risk from a 2 percent ASR on data exfiltration. Report both.
Honest limits: the target is an imitation with assumed behaviour; the gate's effect is a property of my code, while the text-defence numbers say nothing about real models; the attack grid is small and mine.