Part 4 of 12 · 2 min read
A measured score is an estimate; the Wilson interval says how rough
A single percentage with no range attached hides exactly how much of it could be luck.
Think of a political poll that reports "52% support" without ever mentioning the margin of error. The headline number means something very different if the true range is 50-54% than if it is 35-69%.
In plain words: a confidence interval is a range that would contain the true rate in about 95% of repeated experiments, a Wilson interval is a specific formula for that range built for proportions that behaves well even with small samples or extreme rates, and sampling error is the gap between a measured value and the true one caused by only having a finite number of trials -- so the size of the interval tells you exactly how much to trust the single number in the headline.
A worked example. A measured 80% success rate on just 10 cases has a 95% Wilson interval of 0.49 to 0.94 -- the true rate could plausibly be anywhere from under half to nearly all of the time. The same measured 80% on 300 cases narrows to 0.75 to 0.84. Getting the interval down to plus-or-minus 5 points around 80% requires about 246 independent trials -- a specific, computable number, not a vague "enough."
The key lines. From code/mini-flaky/flaky.py:
# from code/mini-flaky/flaky.py
def wilson(k, n, z=1.96):
if n == 0:
return 0.0, 1.0
p = k / n
d = 1 + z * z / n
c = p + z * z / (2 * n)
m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return (c - m) / d, (c + m) / d
Read it as: the formula adjusts the raw proportion p = k/n by terms that all shrink as n grows. Every correction term has n in its denominator. That is the mathematical reason the interval narrows as you collect more trials. It is not a separate rule bolted on. It is built into the shape of the equation.
Confidence interval, Wilson interval, sampling error
A confidence interval is a range that would contain the true rate in about 95 percent of repeated experiments. A Wilson interval is an interval for a proportion that behaves well for small samples and extreme rates. Sampling error is the difference between a measured and a true value caused by finite samples.
Example: going from a 10-case interval of 0.49-0.94 to a 1000-case interval of 0.77-0.82 is sampling error shrinking in front of you -- the true rate never moved, only how precisely you could pin it down with more trials.