CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • A regression suite tracks quality over time and stops a change that makes the agent worse. Store results with model version and configuration, and rerun the full suite whenever a tool changes.
  • A fixed threshold on the score is a bad gate: it fails an identical agent a large share of the time on a small suite.
  • A paired gate compares candidate and baseline per case and fails only on a real drop with statistical evidence, which keeps false failures to a few percent.
  • The price is power: a gate cannot create evidence the suite does not have, so more cases and repeated trials buy detection.
  • Gate on more than the average: critical cases that must always pass, a cost or latency limit and per-slice checks. Quarantine flaky cases so they inform but do not break the build.