CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Version and store results: case id, pass rate, trace link, model and prompt version, configuration and date.
  • Gate in layers: critical cases (no tolerance), a paired statistical test for the average and each important slice, and cost and latency limits.
  • Pin what you can: model version, temperature, tool versions, recorded tool outputs.
  • Use several trials and compare per-case pass rates.
  • Quarantine flaky cases, track them, and fix them within a deadline.
  • Make failures actionable: list the cases that got worse, with links to old and new traces.
  • Promote the baseline on merge, not on every commit, and keep old baselines for trend charts.
  • Keep a fast smoke set for every commit and the full suite nightly or on release branches.
  • Add every production incident as a new critical case (module 2).
  • Rerun everything when a tool changes, not only the cases that mention it.