CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • A public benchmark is a hint about general ability, not a measurement of your product. Use it to shortlist; decide on your own eval set.
  • Benchmarks saturate and decay, through contamination and flawed tasks; OpenAI's 2026 decision to stop reporting SWE-bench Verified is a current example.
  • Small benchmarks rank unreliably and have little power to see a real gap; flawed tasks compress differences, and contamination can invert a ranking.
  • Rows are not comparable unless the harness is: scaffolding, tools, retries and budgets differ between leaderboard rows, so compare like with like and report cost next to score.
  • GAIA tests reasoning, browsing and tool use; tau-bench popularised repeating each task to measure consistency (the idea behind pass^k).