CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 5 of 12 · 3 min read

Confidence intervals and slicing turn a bare number into actual evidence

A single accuracy number from a limited test set is itself just one noisy sample, and averaging across subgroups can hide which model actually wins where.

Think of a political poll reporting "52 percent support" with no margin of error attached. The number alone tells you much less than the number plus "give or take 3 points." That second form is what actually lets you judge whether a result is meaningful.

In plain words: an accuracy score measured on a limited test set is itself a noisy estimate — reporting a confidence interval (via bootstrap resampling of the test items) shows how much that number could move, and reporting results by subgroup rather than only in aggregate can reveal that a model losing overall actually wins on every individual slice.

A worked example. A model with a true accuracy of 0.80, measured on just 100 test items, might read 0.73 with a 95 percent bootstrap interval of 0.64 to 0.82 — an 18-point-wide range that includes the true value but also a lot of territory that isn't it. At 1,000 items, that interval tightens to 0.78-0.83, just 4.8 points wide. Separately, in an invented example, model A scores higher than model B on both the easy slice (94.0 vs. 90.0 percent) and the hard slice (32.0 vs. 28.0 percent) — yet loses overall (40.9 vs. 81.1 percent), because model A was tested on mostly hard items and model B on mostly easy ones. Averaging across slices with different weightings hid a result that was actually consistent within each slice.

The key lines. From code/mini-papers/stats.py:

# from code/mini-papers/stats.py
def bootstrap_ci(correct, reps=2000, seed=0, level=0.95):
    """Percentile bootstrap interval for accuracy (resample items with replacement)."""
    rng = np.random.default_rng(seed)
    correct = np.asarray(correct, float)
    n = len(correct)
    means = correct[rng.integers(0, n, (reps, n))].mean(axis=1)
    lo, hi = np.percentile(means, [(1 - level) / 2 * 100, (1 + level) / 2 * 100])
    return float(lo), float(hi)

Read it as: the function resamples the actual test items with replacement thousands of times and looks at how much the resulting mean score varies. The width of that spread is a direct, empirical measure of how much the reported number could have come out differently. The only change would be which items happened to be in the test set.

Confidence interval and bootstrap, Simpson's paradox

A confidence interval is a range of values consistent with the data at a stated confidence level. A bootstrap builds that interval by resampling the evaluation items with replacement to see how much a score moves. Simpson's paradox is when a model that wins in every subgroup can lose overall, because the subgroups are weighted differently between the two models being compared.

Example: the 18-point-wide interval at 100 items above is the honest answer to "how much should I trust this one number," and the 40.9-vs-81.1 reversal in the invented example is Simpson's paradox showing exactly why "report by slice" has to come before "report overall."