Part 9 of 12 · 2 min read
Mistakes that defeat a quality gate
Common mistakes, and what the example does not do.
Mistakes that defeat a quality gate
- A fixed score threshold on a noisy suite.
- Rerunning until green.
- Comparing against a baseline from a different model or date without recording it.
- One trial per case for a stochastic agent.
- No critical cases, so an improved average hides a new catastrophic failure.
- A giant slow suite on every commit that engineers skip.
- Updating the baseline automatically, so slow decay is never caught.
- Letting flaky cases accumulate.
- Tuning prompts against the gate's cases until they no longer represent production.
What the example does not do
About this example What the example does not do
- No real agent or harness; results are simulated pass rates.
- A uniform drop in every case; real regressions are concentrated.
- Independent cases; shared causes (one broken tool) make wins and losses correlated.
- One test: no multiple-comparison correction across slices, no sequential testing.
- The CI file is a sketch to adapt to your own CI system.
- Optional real experiment: run the same agent twice through your suite with three trials each, and use
gate.pyto confirm that the gate passes; then degrade the prompt on purpose and confirm that it fails.
Check yourself
A CI gate fails the build when the score drops more than 2 points. The agent did not change. How often does it still fail?
Show answer
45 percent of the time with 30 cases and one run each; 9 percent with 300 cases and three runs. A paired comparison with a sign test cut false failures to 2 to 5 percent.
For the curious The price of a safer gate is power
The paired gate caught a true 5-point drop only 7 percent of the time with 30 cases and one run, 45 percent with 100 cases and three runs, and 88 percent with 300 cases and three runs. Fewer false alarms means more missed regressions unless you add cases and repeats. The simulation assumes independent cases; real regressions often hide in a few slices.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Critical cases that must always pass | One safety or money failure fails the build. |
| A paired test on the average | Compare candidate and baseline case by case. |
| Cost and latency limits | A better score that doubles cost can still fail. |
| Quarantine flaky cases | They inform but do not break the build; then fix or remove them. |
Before using it, confirm how your CI system stores artifacts and which secrets the eval job may read.