CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Use public benchmarks to shortlist, then evaluate finalists on your own eval set with intervals.
  • Check the date and the version: benchmarks are retired or superseded (the SWE-bench Verified case).
  • Read the harness, the budget and the number of attempts before comparing two rows.
  • Look for a held-out or private split, and prefer benchmarks with fresh tasks.
  • Report cost and latency with the score.
  • Do not tune on the benchmark and then quote it.
  • Re-run what you can, from the benchmark's own harness, with repeated trials (module 1).