CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 2 of 12 · 2 min read

Two models with exactly equal skill can still look different on a few hundred items

A reported gain has to be measured against what pure chance alone can already produce at that test size.

Think of flipping two identical, perfectly fair coins 100 times each. One will almost certainly come up heads a few more times than the other, purely by chance. Neither coin is actually biased.

In plain words: before trusting a reported improvement, check it against what two genuinely equal models would look like by pure chance at that same test size — a paired comparison, testing only the items where the two models disagree, is the honest way to check whether a gap is really there.

A worked example. Two models with exactly equal skill, compared on 100 items, show the "new" one looking at least 1 point better 45 percent of the time, and at least 2 points better 37 percent of the time — purely from sampling noise, with zero real difference. At 3,000 items, that drops to 12 percent and 1 percent respectively. Meanwhile, a proper paired significance test at the 5 percent level flags these equal pairs as "significant" almost exactly 5 percent of the time at every test size (5.5, 5.1, 4.4, 4.4 percent) — the test itself is honest; it's the unadorned headline number that misleads.

The key lines. From code/mini-papers/stats.py:

# from code/mini-papers/stats.py
def null_gain_rates(n, trials=3000, seed=0, skill=0.7):
    """Two models with the SAME skill: how often does the new one look >= 1 point / 2 points better,
    and how often is a paired test (two-sided 5%) 'significant'?"""
    rng = np.random.default_rng(seed)
    g1 = g2 = sig = 0
    for _ in range(trials):
        a, b = paired_items(skill, skill, n, rng)
        diff = b.mean() - a.mean()
        g1 += diff >= 0.01; g2 += diff >= 0.02
        sig += abs(paired_z(a, b)) > Z_A

Read it as: both models in this simulation are given the exact same skill parameter. There is no real difference to detect by construction. So every "gain" the simulation observes is pure sampling noise. That is exactly what makes the observed rates a baseline for judging a real paper's claimed gap.

Claim, setup, evidence, catch; paired comparison

Reading a paper means extracting its claim (the headline result), setup (how it was obtained), evidence (the data supporting it), and catch (what would undermine it). A paired comparison scores two systems on the same items and tests only the items where they disagree, which needs far fewer items than comparing two independent samples.

Example: the 45-percent "looks better" rate at 100 items above is exactly the gap between a paper's claim and its evidence — a claim of "1 point better" is unsupported evidence unless the test size and the significance test are both reported alongside it.