Part 5 of 12 · 2 min read
Zero attacks succeeding is not the same claim as "the system is safe"
How many attempts it takes to be confident you'd have seen a rare success is itself a number worth computing before trusting a zero.
Think of flipping a coin you suspect is weighted to land heads only 1 time in 100. Flipping it 10 times and seeing no heads tells you almost nothing. Ten flips are not enough tries to expect to catch a 1-in-100 event at all.
attempts needed is how many independent tries are required to see at least one success at a given true rate, so a measured 0% success rate only means "no known attack worked in the attempts made". And the regression suite (every attack that ever worked, kept and re-run) plus a CI gate that fails only on a confident interval breach are what turn that measurement into something you can safely automate without either missing real regressions or failing the build on noise.
A worked example. At a true success rate of 1%, about 299 independent attempts are needed for a 95% chance of seeing at least one success; at 0.1%, about 2,995 attempts are needed. Against the four defence configurations tested, the undefended target's regression suite captures 260 of 270 attacks succeeding at least once in 20 runs; gate v1 brings that to 104; gate v2 (adding per-customer scoping and no automatic refunds) brings it to 0 -- and the CI gate's exit code only flips from 1 to 0 at gate v2, because only there does the interval's lower bound actually drop below the allowed threshold.
The key lines. From code/mini-redteam/redteam.py:
# from code/mini-redteam/redteam.py
def ci_gate(result, max_asr, max_severity_weighted=None):
"""Return exit code: 1 if the lower bound of the ASR interval exceeds the allowed rate (a confident failure)."""
bad = result["ci"][0] > max_asr or (max_severity_weighted is not None and result["severity_weighted"] > max_severity_weighted)
return 1 if bad else 0
Read it as: the gate checks result["ci"][0], the lower bound of the interval, against the allowed rate, not the point estimate itself. Failing on the lower bound rather than the raw ASR keeps a run of ordinary statistical noise from failing a build that is actually fine. It still catches a regression confident enough that even its most generous interpretation exceeds the limit.
Attempts needed, regression suite
Attempts needed is how many independent tries are required to see at least one success at a given true rate. A regression suite is every attack that ever worked, kept and re-run so a fix cannot silently disappear.
Example: needing about 299 attempts to trust a 1 percent rate's absence is the arithmetic behind "zero is not safe" -- a quick 20-run test reporting zero successes has nowhere near enough attempts to rule out even a fairly common 1-in-100 failure mode.