Definitions

Definitions

The pipeline

Results

1. Gaps of 1 to 3 points on a few hundred items are within chance.
2. Seeds, tuning budget and test-set selection each add apparent gains without any real improvement.
3. Reporting the winners among many benchmarks inflates the win rate.
4. Intervals and slices turn a number into evidence.
5. The right response is a small reproduction on your data, not trust or dismissal.

Design rules

• Write the claim in one sentence before reading the method.
• Find the test size, the seeds and the interval; if absent, treat the gain as unproven.
• Check the baseline's tuning budget and prompts against the proposed method's.
• Ask what was selected on the test set and who chose the benchmarks and when.
• Look for ablations, slices and a contamination check.
• Compare effect size with what chance produces at that test size (this module's tables).
• Reproduce the headline on your own data with a fixed budget before adopting a method.
• Keep a one-page note per paper (the template) and record what you tested.
• Report your own results with an interval, seeds and the tuning budget you gave each side.

Interview questions, with the follow-ups

Common mistakes

• Trusting a single number with no test size or interval.
• Comparing against an untuned baseline.
• Reporting the best of many runs or variants without saying so.
• Choosing benchmarks after seeing results.
• Ignoring seeds.
• Averaging across slices that differ between systems.
• Skipping contamination on public benchmarks.
• Adopting a method without a reproduction on your own data.
• Dismissing a paper because the checklist has flags; flags are questions, not verdicts.

What the example does not do

• No real papers: I read none for this module, so the fictional note shows the checklist, not how any real paper fares.
• Simulated noise: item difficulty, skill and spreads are assumed; real evaluation noise has more structure (correlated items, judge errors, prompt sensitivity).
• Assumed spreads: the 0.5, 1 and 2 point seed spreads and the 1.5 point configuration spread are my choices.
• Independent benchmarks: the many-benchmark simulation assumes independent items; real benchmarks are correlated.
• Normal-approximation sample sizes and one family of tests; no multiple-testing corrections or Bayesian alternatives.
• No reading of method sections: the template covers what to ask, not how to understand an algorithm.
• Optional real experiment (I did not run this): take one recent paper whose method you could reproduce on a small scale, fill the template, run the baseline and the method with equal budgets over five seeds, and report an interval for the gap.

For the curious Four ways a gap can appear with no real difference

Seeds: one run each makes equal methods look 1 point apart 24 percent of the time at an assumed 1-point spread. Tuning budget: a method that reports the best of 20 configurations beats an untuned equal by 2.81 points. Test-set selection: the best of 100 equal variants looks like 0.751 against a true 0.70. Many benchmarks: with 10 independent ones, an equal model wins a significant result on at least one 24 percent of the time. All are simulations with assumed spreads; I read no real papers for this module.

What you must be able to say (one page)

• Read for four things. (1) the claim: what is better, by how much, on which metric, against what; (2) the setup: data, baselines, tuning budget, compute, who chose the benchmarks; (3) the evidence: test size, seeds, intervals, ablations, slices, contamination checks; (4) the catch: what would make it false, what it does not show, whether it would hold on your data. This template is my own; the usual multi-pass approach (skim, then read, then check) is general knowledge.
• A gain smaller than the noise is not a result. Two models with exactly equal skill, compared on the same items (my simulation, 3,000 trials per size): the new one looked at least 1 point better 45 percent of the time on 100 items, 39 percent on 300, 26 percent on 1,000 and 12 percent on 3,000; at least 2 points better 37, 25, 9 and 1 percent. A paired significance test at 5 percent flagged about 5 percent of these equal pairs at every size (5.5, 5.1, 4.4, 4.4 percent), so the test is honest; the unadorned headline number is not.
• Seeds hide inside the headline. If a method's runs vary by 1 point (an assumed spread) and two equal methods each report one run, the "new" one looks at least 1 point better 24 percent of the time; averaging 3, 5 and 10 runs cuts that to 11, 6 and 1 percent. With a spread of 2 points it stays at 36, 27, 21 and 13 percent for 1, 3, 5 and 10 runs.
• Tuning budget can create the gap by itself. If a configuration's score varies by 1.5 points (assumed) and each method reports its best configuration, a method with 2, 5, 10, 20 or 50 tries beats an untuned single-try baseline of the same quality by 0.85, 1.74, 2.31, 2.81 and 3.38 points on average; with equal budgets the gap is 0.0.
• Picking the best on the test set inflates it (winner's curse). Among k variants that all truly score 0.70, measured on one 500-item test set, the best looks like 0.724 (k=5), 0.738 (k=20) and 0.751 (k=100); on a fresh test set it is 0.700 again. A leaderboard or paper that selected on the test set reports the first number.
• Many benchmarks, one chance each. Equal models on m independent 500-item benchmarks: the new one is "significantly better" on at least one 3 percent of the time for m=1, 8.5 for m=3, 16 for m=6 and 24 for m=10 (a one-sided 5 percent test would give 2.5, 7, 14 and 22 percent). Reporting the benchmarks that happened to win is not evidence. Leaderboards also differ in which benchmarks they include, so rankings differ.
• How many items? A rough rule of thumb (I recall it from OpenAI's evaluation guidance; not re-checked) says about 10, 100, 1,000 and 10,000 items to be 95 percent confident of a 30, 10, 3 and 1 point difference, with 10 times more items for each 3 times smaller gap. My calculation (80 percent power, two-sided 5 percent): for 3 points, 3,555 per model with independent samples or 1,742 items with a paired comparison where 20 percent of items disagree; for 1 point, 32,648 or 15,696. The scaling matches (it is a 1/gap^2 law) but the rule's levels are lower than mine; read it as a rough guide. For the bootstrap, resample the evaluation set with replacement and see how much the score moves.
• The test set itself is a sample. A model with true accuracy 0.80 scored 0.73 on one 100-item set (95 percent interval 0.64 to 0.82, 18 points wide), 0.76 on 300 (0.72 to 0.81, 9 wide) and 0.81 on 1,000 (0.78 to 0.83, 4.8 wide). A single number from 100 items says little.
• Slice before you average. This is Simpson's paradox. In my invented example, model A scores 94 percent on easy items and 32 on hard items, model B 90 and 28, so A wins both slices, yet B wins overall (81.1 against 40.9 percent) because B was tested on mostly easy items and A on mostly hard ones. Ask for results by slice and say which mix you care about.
• Contamination and leaderboards: benchmark data published before a model's training is likely to be in it; GPT-3's own analysis reported that some benchmarks overlapped heavily with its training data (I recall a share above 40 percent for a number of them; not re-checked); keeping part of a benchmark private helps. Public benchmarks help filter out bad models but will not find the best one for your application; running a full benchmark suite across many models can itself cost a lot.
• Apply the checklist. My red_flags function lists what a structured note leaves unsupported. On a fictional paper ("Method X lifts success by 3 points", 400 items, one seed, no interval, unclear baseline tuning, benchmarks chosen afterwards) it raises 9 flags.
• What to say in an interview: "I extract the claim, setup, evidence and catch. I check the test size against the gain, seeds and intervals, whether the baseline got the same tuning budget, whether variants were selected on the test set, whether benchmarks were chosen after the fact, contamination, and results by slice. Then I reproduce the headline on my own data at small scale before adopting it."
• Honest limits: all numbers come from simulations with assumed spreads and a simple model of item difficulty; the checklist is a tool for questions, not a verdict; I read no research papers for this module, so nothing here describes a real paper.