CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 5 of 12 · 2 min read

A flaky case is neither a pass nor a fail; quarantine it so it can't break the build

A case that passes sometimes and fails sometimes isn't measuring the candidate, it's measuring its own instability, and letting it vote in the gate just adds noise.

Think of a wobbly thermometer that reads anywhere from 97 to 103 degrees for the same steady body temperature. Using one reading from it to decide whether someone has a fever would be worse than not measuring at all.

In plain words: a flaky case is a case with an unstable result, found by running it several times and watching its pass rate land somewhere unreliable (neither consistently passing nor consistently failing), and quarantining it means keeping it in the suite for information while stopping it from being able to fail the build, which buys time to fix the actual cause -- an ambiguous rubric, an unfrozen tool version, or a genuinely unstable behavior -- without that instability contaminating every other build's verdict.

A worked example. Running each case multiple times and keeping cases whose pass rate across trials falls between 20% and 80% identifies exactly the cases too unstable to trust on a single run. Separately, running three trials per case instead of one, at a fixed 100-case suite size, raised the paired gate's detection of a true 5-point drop from 20% to 45% -- repeated trials buy real statistical power (the chance of spotting a real difference) by reducing the noise in each case's own pass-rate estimate, which is the same underlying idea as flagging instability by looking at results across several runs rather than one.

The key lines. From code/mini-gate/gate.py:

# from code/mini-gate/gate.py
def flaky_cases(per_trial_results, low=0.2, high=0.8):
    """per_trial_results: {case: [bool,...]}. Cases that pass sometimes but not reliably."""
    return sorted(k for k, v in per_trial_results.items() if low <= sum(v) / len(v) <= high)

Read it as: the function does not ask whether a case passed or failed. It asks whether the case's pass rate across trials landed in the ambiguous middle. A case stuck near 50% is not borderline-bad. It is unmeasured. The fix is not to count it as a failure. The fix is to run it more, or fix what makes it unstable. That is why quarantine exists as a separate state from pass or fail.

Flaky case, quarantine

A flaky case is a case with an unstable result. Quarantine means keeping a case in the suite but not letting it fail the build.

Example: a case that passes 20 to 80 percent of the time across repeated trials is, by the module's own rule, exactly the band flaky_cases flags for quarantine -- not because it's bad, but because a single run of it tells you almost nothing about whether the candidate actually regressed.