Part 7 of 12 · 1 min read
How to build it so it holds up
- Use public benchmarks to shortlist, then evaluate finalists on your own eval set with intervals.
- Check the date and the version: benchmarks are retired or superseded (the SWE-bench Verified case).
- Read the harness, the budget and the number of attempts before comparing two rows.
- Look for a held-out or private split, and prefer benchmarks with fresh tasks.
- Report cost and latency with the score.
- Do not tune on the benchmark and then quote it.
- Re-run what you can, from the benchmark's own harness, with repeated trials (module 1).