Part 7 of 12 · 1 min read
How to build it so it holds up
- Run each case several times (3 to 5 at the start) and report the mean and an interval.
- Fix what you can: seeds where the API allows, recorded tool responses, a frozen environment; accept what you cannot fix and measure it.
- Report two numbers when it matters: pass@k (capability) and pass^k (reliability).
- State the suite size next to every score.
- Compare paired: run both versions on the same cases (module 7 uses this).
- Keep a noise baseline: run the same version twice and record the difference; changes below it are noise.