CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Picking the best of several noisy options overstates how good the winner really is

The highest score among several candidates is partly real skill and partly luck, and the more candidates you compare, the more luck tends to be mixed in.

Think of ranking ten students by a single noisy quiz score. The student who happened to have the best day, not necessarily the best overall ability, usually comes out on top. That top score flatters them more than a longer, fairer test would.

In plain words: winner's curse is that the best of many noisy estimates is biased upward, because picking the maximum doesn't just select for real quality, it also selects for whichever candidate happened to get lucky noise on the specific sample used to judge it -- so the observed score of whatever wins a noisy comparison overstates its true underlying quality, and that gap shrinks only by collecting more votes per candidate.

A worked example. Choosing the best of 10 prompt variants by thumbs-up rate, using 100 votes per variant, the winning variant's observed rate averages 78.1% -- but its actual, true rate (measured against its own underlying probability) averages only 72.6%, a 5.5-point inflation purely from the selection process. Increasing to 1,000 votes per variant before selecting shrinks that gap dramatically: observed 75.0% versus true 74.2%, under a point of inflation -- more votes per candidate means less of the "winner" is explained by luck.

The key lines. From code/mini-feedback/feedback.py:

# from code/mini-feedback/feedback.py
def winner_curse(k=10, n_fb=100, true_mean=0.7, spread=0.03, trials=2000, seed=2):
    """Pick the variant with the best observed thumbs-up rate out of k.
    Returns (mean observed rate of the winner, mean true rate of the winner, mean true rate of a random variant)."""
    ob = [sum(rng.random() < p for _ in range(n_fb)) / n_fb for p in ps]
    i = max(range(k), key=lambda j: ob[j])
    obs.append(ob[i]); tru.append(ps[i]); rnd.append(sum(ps) / k)

Read it as: the function tracks three separate numbers for the same winning variant. They are its observed rate, its own true rate, and the true rate of a randomly chosen variant. Comparing all three is what exposes the curse. The winner's observed score is inflated relative to its own true score. This is because max was applied across noisy estimates. It is not because the winner is a weak variant in disguise.

Winner's curse

Winner's curse is that the best of many noisy estimates is biased upward.

Example: a 5.5-point gap between observed (78.1%) and true (72.6%) performance for a winner chosen from 100 votes each is winner's curse quantified -- which is exactly why the module's design rules call for a held-out confirmation sample rather than trusting the selection round's own numbers.