CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that mistake noise for progress

Common mistakes, and what the example does not do.

Mistakes that mistake noise for progress

  • Trusting a single number with no test size or interval.
  • Comparing against an untuned baseline.
  • Reporting the best of many runs or variants without saying so.
  • Choosing benchmarks after seeing results.
  • Ignoring seeds.
  • Averaging across slices that differ between systems.
  • Skipping contamination on public benchmarks.
  • Adopting a method without a reproduction on your own data.
  • Dismissing a paper because the checklist has flags; flags are questions, not verdicts.

What the example does not do

About this example What the example does not do
  • No real papers: none was read for this module, so the fictional note shows the checklist, not how any real paper fares.
  • Simulated noise: item difficulty, skill and spreads are assumed; real evaluation noise has more structure (correlated items, judge errors, prompt sensitivity).
  • Assumed spreads: the 0.5, 1 and 2 point seed spreads and the 1.5 point configuration spread are the choices.
  • Independent benchmarks: the many-benchmark simulation assumes independent items; real benchmarks are correlated.
  • Normal-approximation sample sizes and one family of tests; no multiple-testing corrections or Bayesian alternatives.
  • No reading of method sections: the template covers what to ask, not how to understand an algorithm.
  • Optional real experiment: take one recent paper whose method you could reproduce on a small scale, fill the template, run the baseline and the method with equal budgets over five seeds, and report an interval for the gap.

Check yourself

A paper reports a 1-point gain on a 500-item benchmark. What do you ask?

Show answer

How many seeds and how many tuning runs each method got, whether the test set was used to pick variants, and whether there is an interval. In the simulation two equal models looked at least 1 point apart 26 percent of the time on 1,000 items.

For the curious Four ways a gap can appear with no real difference

Seeds: one run each makes equal methods look 1 point apart 24 percent of the time at an assumed 1-point spread. Tuning budget: a method that reports the best of 20 configurations beats an untuned equal by 2.81 points. Test-set selection: the best of 100 equal variants looks like 0.751 against a true 0.70. Many benchmarks: with 10 independent ones, an equal model wins a significant result on at least one 24 percent of the time. All are simulations with assumed spreads; no real papers were read for this module.

What you will use daily at work

Practice What it looks like
Extract claim, setup, evidence and catch Four lines per paper, before you read the details.
Check test size against the gain Detecting 3 points needs about 1,742 paired items.
Ask for intervals, seeds and slices A single number from 100 items says little.
Reproduce the headline on your data Small scale first, before adopting a method.