CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

A measured score is an estimate; the Wilson interval says how rough

A single percentage with no range attached hides exactly how much of it could be luck.

Think of a political poll that reports "52% support" without ever mentioning the margin of error. The headline number means something very different if the true range is 50-54% than if it is 35-69%.

In plain words: a confidence interval is a range that would contain the true rate in about 95% of repeated experiments, a Wilson interval is a specific formula for that range built for proportions that behaves well even with small samples or extreme rates, and sampling error is the gap between a measured value and the true one caused by only having a finite number of trials -- so the size of the interval tells you exactly how much to trust the single number in the headline.

A worked example. A measured 80% success rate on just 10 cases has a 95% Wilson interval of 0.49 to 0.94 -- the true rate could plausibly be anywhere from under half to nearly all of the time. The same measured 80% on 300 cases narrows to 0.75 to 0.84. Getting the interval down to plus-or-minus 5 points around 80% requires about 246 independent trials -- a specific, computable number, not a vague "enough."

The key lines. From code/mini-flaky/flaky.py:

# from code/mini-flaky/flaky.py
def wilson(k, n, z=1.96):
    if n == 0:
        return 0.0, 1.0
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (c - m) / d, (c + m) / d

Read it as: the formula adjusts the raw proportion p = k/n by terms that all shrink as n grows. Every correction term has n in its denominator. That is the mathematical reason the interval narrows as you collect more trials. It is not a separate rule bolted on. It is built into the shape of the equation.

Confidence interval, Wilson interval, sampling error

A confidence interval is a range that would contain the true rate in about 95 percent of repeated experiments. A Wilson interval is an interval for a proportion that behaves well for small samples and extreme rates. Sampling error is the difference between a measured and a true value caused by finite samples.

Example: going from a 10-case interval of 0.49-0.94 to a 1000-case interval of 0.77-0.82 is sampling error shrinking in front of you -- the true rate never moved, only how precisely you could pin it down with more trials.