Part 10 of 12 · 1 min read
Quick notes
- Read for four things: the claim, the setup, the evidence and the catch (what would make it false and whether it would hold on your data).
- A gain smaller than the noise is not a result. Two equal models often look a point or two apart on a few hundred items; a paired significance test is honest, the headline number is not.
- Seeds, tuning budget and test-set selection can each create the gap by themselves, and reporting only the benchmarks that happened to win is not evidence.
- How many items? The need grows with one over the gap squared, so a few hundred items cannot show a small difference, and the test set itself is a sample with a wide interval.
- Slice before you average (Simpson's paradox), check for contamination, and reproduce the headline on your own data at small scale before adopting it.