CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • Read for four things: the claim, the setup, the evidence and the catch (what would make it false and whether it would hold on your data).
  • A gain smaller than the noise is not a result. Two equal models often look a point or two apart on a few hundred items; a paired significance test is honest, the headline number is not.
  • Seeds, tuning budget and test-set selection can each create the gap by themselves, and reporting only the benchmarks that happened to win is not evidence.
  • How many items? The need grows with one over the gap squared, so a few hundred items cannot show a small difference, and the test set itself is a sample with a wide interval.
  • Slice before you average (Simpson's paradox), check for contamination, and reproduce the headline on your own data at small scale before adopting it.