CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 5 of 12 · 2 min read

Detecting a small accuracy gap needs far more items than people expect

A 2-point accuracy difference between two models can easily hide inside the noise of a 300-item test set, even if it's completely real.

Think of trying to tell whether a coin is biased 52-48 instead of a fair 50-50 by flipping it only 20 times. The difference is real. But 20 flips cannot reliably distinguish it from ordinary chance variation.

In plain words: a small, genuine difference in accuracy between two models needs a correspondingly large number of test items to reliably detect, and a paired test (comparing both models on the identical items) needs far fewer items than an independent test, because it only has to resolve the items where the two models actually disagree.

A worked example. To reliably detect a real 2-point accuracy difference with 80 percent power, a paired test needs about 3,651 items. At just 300 items, the same paired test only has 13 percent power — it would fail to detect that real difference the vast majority of the time. An independent-samples test (not exploiting the fact that both models saw the same items) needs about 6,187 items for the same power — almost twice as many, because it can't isolate just the disagreements.

The key lines. From code/mini-small/swap.py:

# from code/mini-small/swap.py
def n_paired(p_discordant, diff, z_a=Z[0.05], z_b=Z[0.80]):
    """Items for a paired (McNemar-style) test.

    p_discordant: share of items where exactly one model is right. diff: accuracy difference.
    Normal approximation for the difference in discordant counts.
    """

Read it as: the formula's only inputs are the discordant-item share and the accuracy difference itself. The required sample size is a direct, calculable consequence of how small the real gap is and how often the two models actually disagree. It is not a rule of thumb you can skip.

Statistical power

Power is the chance a test detects a real difference of a given size; a paired test compares two models on the same items and gets its power mainly from the items where they disagree, needing far fewer items than an independent test for the same power.

Example: the 3,651-vs-6,187 item counts above are the paired and independent tests reaching the same 80 percent power for the same 2-point gap — the gap itself never changes, only how efficiently each test design can detect it.