CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 8 of 12 · 2 min read

Answering CI-gate questions in an interview

How do you stop a prompt change from degrading your agent?

"A gate in CI that runs the eval suite on every change and compares per case against the stored baseline. It fails on critical cases, on a statistically supported drop, or on a cost limit. In the example's simulation a naive threshold gate failed 45 percent of identical builds with 30 cases, whereas a paired sign test failed 2 percent; the cost was power, 7 percent detection of a 5-point drop with 30 cases, 88 percent with 300 cases and three runs each." Follow-up: "What do you do when the build is red?" The report lists the cases that got worse with traces, and I either fix the change, or accept it with a recorded justification and update the baseline.

Why not simply require the score not to fall?

"Because the score is a noisy measurement. Run-to-run variation alone makes a bare threshold fail identical builds, and engineers learn to rerun until green, which defeats the gate. The paired test uses per-case wins and losses, which removes the shared case difficulty and tests whether more cases got worse than better." Follow-up: "Isn't that too lenient?" Yes for small drops on small suites; that is why critical cases have zero tolerance and why I increase cases and trials where the cost of a miss is high.

What do you do with flaky eval cases?

"I find them by repeated runs (cases that pass 20 to 80 percent of the time), quarantine them so they do not break the build, and fix the cause: an ambiguous rubric, an unfrozen tool, or a genuinely unstable behaviour that deserves its own investigation. A permanently quarantined case is a hole." Follow-up: "How many trials do you need to call a case flaky?" Enough that the interval for its pass rate (module 1) excludes both 0 and 1; five is a start.

How do you gate on cost and latency?

"As separate rules with their own limits, compared with the baseline on the same cases: a cost-ratio threshold fails the build if the candidate is, say, more than 1.5 times as expensive, regardless of quality. A cheaper agent that is slightly worse can be a trade-off a human approves." Follow-up: "Who can override the gate?" A named owner, with a logged reason, never an automatic retry.