Part 10 of 12 · 1 min read
Quick notes
- A regression suite tracks quality over time and stops a change that makes the agent worse. Store results with model version and configuration, and rerun the full suite whenever a tool changes.
- A fixed threshold on the score is a bad gate: it fails an identical agent a large share of the time on a small suite.
- A paired gate compares candidate and baseline per case and fails only on a real drop with statistical evidence, which keeps false failures to a few percent.
- The price is power: a gate cannot create evidence the suite does not have, so more cases and repeated trials buy detection.
- Gate on more than the average: critical cases that must always pass, a cost or latency limit and per-slice checks. Quarantine flaky cases so they inform but do not break the build.