Part 4 of 12 · 2 min read
Picking the best variant or benchmark after seeing the results inflates what gets reported
The best of many equally-good options will look better than its true quality on the exact data used to pick it.
Think of a lottery where everyone who plays is equally unlikely to win. Whoever happens to win that day gets written up in the news as if they had some special skill at picking numbers.
In plain words: if you try many equally-good variants and report whichever one happened to score highest on your test set, that best score overstates the variant's true quality — and the same thing happens if you test on many benchmarks and only report the ones where you happened to win.
A worked example. Among 100 variants that all truly score 0.700, the best one measured on a single 500-item test set looks like 0.751 — more than 5 points higher than its actual quality — but scores exactly 0.700 again on a fresh test set it wasn't selected on. Separately, with 10 independent benchmarks and two genuinely equal models, the "new" model wins a statistically significant result on at least one of those benchmarks 24.4 percent of the time — nearly 1 in 4 — purely because testing on more benchmarks gives chance more opportunities to produce a win somewhere.
The key lines. From code/mini-papers/stats.py:
# from code/mini-papers/stats.py
def winners_curse(k, n, true_acc=0.7, trials=2000, seed=0):
"""k variants, all with accuracy true_acc, scored on the same test set of n items (independent per variant).
Returns (mean best observed score, mean score of that variant on a fresh test set of n items)."""
rng = np.random.default_rng(seed)
best_obs, fresh = [], []
for _ in range(trials):
obs = rng.binomial(n, true_acc, k) / n
j = int(np.argmax(obs))
best_obs.append(obs[j])
fresh.append(rng.binomial(n, true_acc) / n)
return float(np.mean(best_obs)), float(np.mean(fresh))
Read it as: every one of the k variants shares the identical true_acc, so best_obs is purely the effect of argmax over noisy measurements. fresh then resamples the same true accuracy independently. This shows exactly how much of the "best" score was really just noise that does not reappear on new data.
Winner's curse and test-set selection, multiple comparisons
Winner's curse means the best of many noisy measurements overstates the truth. Selection on the test set means choosing among variants using the exact data you then report results on. Multiple comparisons means testing many hypotheses — benchmarks, metrics, variants — so that some of them win by chance alone.
Example: the 0.751-vs-0.700 gap above is winner's curse made numeric — and the "at least one benchmark wins" rate climbing from 3.1 to 24.4 percent as benchmark count rises from 1 to 10 is multiple comparisons doing the identical thing to benchmark selection instead of variant selection.