Part 3 of 12 · 2 min read
A rate with an interval tells you what you actually know; a bare percentage does not
A percentage from 20 trials and a percentage from 20,000 trials look identical on the page but carry very different amounts of certainty.
Think of a poll of 20 people versus a poll of 20,000. Both can report "55 percent said yes." But the smaller poll's number could easily have landed anywhere from 35 to 75 percent by chance alone. The bare percentage never tells you that on its own.
The attack success rate (ASR) is the share of attack attempts that achieve the attacker's goal, severity-weighted ASR weights those successes by the harm of each goal rather than counting every success equally. A Wilson interval gives a confidence range around a measured proportion that stays well-behaved even very close to 0% or 100%, where a naive interval can misbehave.
A worked example. Running the full 270-attack grid 20 times each against the undefended target gives an overall ASR of 20.7% on the three training phrasing families, 22.3% on the three held-out ones, and a severity-weighted rate of 21.9% -- a slightly lower number than the plain ASR, because the overall mix includes some lower-severity goals succeeding at a higher rate than the one high-severity goal (data exfiltration) does.
The key lines. From code/mini-redteam/redteam.py:
# from code/mini-redteam/redteam.py
def wilson(k, n, z=1.96):
if n == 0: return (0.0, 1.0)
p = k / n; d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return (max(0.0, c - h), min(1.0, c + h))
Read it as: the interval is centred on c, a value pulled slightly toward 0.5 from the raw rate p, not on p itself. That correction keeps the interval from ever claiming impossible bounds. Examples are a lower bound below 0% or an upper bound above 100%, when k is 0 or equal to n. A naive symmetric interval around p can claim exactly that.
Attack success rate, severity weighting, Wilson interval
Attack success rate (ASR) is the share of attack attempts that achieve the attacker's goal. Severity-weighted ASR weights successes by the harm of each goal. A Wilson interval is a confidence interval for a proportion that behaves well near 0 and 1.
Example: reporting "21.9 percent severity-weighted, 95 percent interval 21.1 to 23.3" instead of just "22 percent" is the difference between a number you can act on and a number that might just be noise from this particular run.