CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 6 of 12 · 4 min read

What the example found

Equal-model noise, seed and tuning-budget effects, test-set selection, multi-benchmark wins, sample-size requirements, interval widths, and the Simpson's-paradox example.

How to read the equal-models table: all rows face two genuinely equal models; only the test size differs; the last column is the honest paired significance test's own false-positive rate.

Test items Equal models: new looks 1+ point better 2+ points better Paired 5% test flags
100 45% 37% 5.5%
300 39% 25% 5.1%
1,000 26% 9% 4.4%
3,000 12% 1% 4.4%

What to notice: the last column stays near 5 percent at every test size (as a correctly-calibrated 5-percent test should). The raw "looks better" columns swing from 45 to 12 percent. The gap between "looks better by eye" and "flagged as significant" is the whole reason to run the actual test instead of eyeballing a headline number.

How to read the seed-spread table: each row is a different number of runs averaged per method; each column is a different run-to-run score spread.

Runs averaged Spread 0.5 point Spread 1 point Spread 2 points
1 7.5% 24% 36%
3 0.7% 11% 27%
5 0.1% 6% 21%
10 0% 1% 13%

What to notice: more averaged runs helps enormously at a small spread (7.5 to 0 percent by 10 runs) but much less at a large spread (36 to 13 percent). A noisier method needs far more repeated runs before averaging actually suppresses the false-gain rate to something small.

How to read the tuning-budget table: each row is how many tuning tries the "new" method gets against a fixed 1-try baseline, except the last row, which gives both sides equal tries.

Tries for the new method (baseline 1) Average gap from tuning alone
2 +0.85 points
5 +1.74
10 +2.31
20 +2.81
50 +3.38
20 against 20 0.00

What to notice: the gap keeps growing all the way to 50 tries. But it collapses to exactly 0.00 the moment both sides get the same number of tries. This single row is the cleanest demonstration in the whole module that an unequal tuning budget, not the method itself, produced every number above it.

How to read the test-set-selection table: each row is how many equally-good variants were tried; the two columns compare the best one's score on the test set it was picked on versus a fresh one it wasn't.

Variants compared on one 500-item test set Best reported Same variant, fresh test set
1 0.700 0.700
5 0.724 0.700
20 0.738 0.700
100 0.751 0.700

What to notice: the fresh-test-set column stays flat at 0.700 no matter how many variants were tried, while the "best reported" column keeps climbing. The inflation is entirely a property of which test set the number came from, not of the variant itself.

How to read the multi-benchmark table: each row is a different number of benchmarks reported; both models are genuinely equal on every one.

Benchmarks reported (equal models) "Significant win" on at least one
1 3.1%
3 8.5%
6 15.9%
10 24.4%

What to notice: this rate climbs toward 1-in-4 by 10 benchmarks, despite the models being genuinely tied on every single one. Reporting "we won on benchmark X" without saying how many benchmarks were tried in total hides most of the relevant context.

How to read the sample-size table: each row is a different gap size to detect; the three columns compare a rough rule of thumb against two calculated methods (independent samples vs. paired with 20 percent disagreement).

Gap to detect Rough rule of thumb (95% confidence) Independent samples, per model (80% power) Paired, 20% disagreement
30 points ~10 40 not possible (gap exceeds disagreement)
10 points ~100 389 155
3 points ~1,000 3,555 1,742
1 point ~10,000 32,648 15,696

What to notice: every column scales the same way (roughly ten times more items for each threefold-smaller gap, a 1/gap-squared law). But the paired column needs consistently fewer items than the independent column. Exploiting the fact that both models see the same items is a real, calculable advantage, not just a convenience.

How to read the interval-width table: all three rows face a model with the identical true accuracy of 0.80; only the test set size differs.

Evaluation set (true accuracy 0.80) Measured accuracy 95% bootstrap interval Width
100 items 0.73 0.64 to 0.82 18 points
300 items 0.76 0.72 to 0.81 9 points
1,000 items 0.81 0.78 to 0.83 4.8 points

What to notice: the measured accuracy at 100 items (0.73) undersells the true 0.80 by 7 points. The interval is wide enough that this is not even surprising. The interval width, not just the point estimate, is what tells you whether 0.73 and 0.80 are actually in tension or well within the same plausible range.

How to read the Simpson's-paradox table: each row is a model; the two middle columns are per-slice scores; the overall column is the weighted total, where the weights differ between the two models' actual test compositions.

Invented example Easy items Hard items Overall
Model A 94.0% (47/50) 32.0% (96/300) 40.9% (143/350)
Model B 90.0% (270/300) 28.0% (14/50) 81.1% (284/350)

What to notice: model A's 350 items are mostly hard (300 of 350), while model B's identical-sized set is mostly easy (300 of 350). The "overall" column is comparing two numbers that were computed from very different item mixes. That is exactly what makes the reversal possible.

{{svg:reading|The reading template, a checklist on a fictional paper, Simpson's paradox and sample sizes.}} {{svg:noise|How easily noise looks like progress (equal methods in every panel).}}