CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Run each case several times (3 to 5 at the start) and report the mean and an interval.
  • Fix what you can: seeds where the API allows, recorded tool responses, a frozen environment; accept what you cannot fix and measure it.
  • Report two numbers when it matters: pass@k (capability) and pass^k (reliability).
  • State the suite size next to every score.
  • Compare paired: run both versions on the same cases (module 7 uses this).
  • Keep a noise baseline: run the same version twice and record the difference; changes below it are noise.