Part 7 of 12 · 1 min read
How to build it so it holds up
- Version and store results: case id, pass rate, trace link, model and prompt version, configuration and date.
- Gate in layers: critical cases (no tolerance), a paired statistical test for the average and each important slice, and cost and latency limits.
- Pin what you can: model version, temperature, tool versions, recorded tool outputs.
- Use several trials and compare per-case pass rates.
- Quarantine flaky cases, track them, and fix them within a deadline.
- Make failures actionable: list the cases that got worse, with links to old and new traces.
- Promote the baseline on merge, not on every commit, and keep old baselines for trend charts.
- Keep a fast smoke set for every commit and the full suite nightly or on release branches.
- Add every production incident as a new critical case (module 2).
- Rerun everything when a tool changes, not only the cases that mention it.