CSE Roadmap

Common tier · about 5 hours · 12 parts

Part 4 of 12 · 2 min read

The review gate: catching shortcuts before merge

Protecting test files and flagging suspicious diffs turns a passing test run into something worth trusting.

Think of a building inspector who checks not just whether the lights turn on, but whether any wiring was bypassed to make them turn on. The lights working is not, by itself, proof the wiring is sound.

In plain words: protecting test and CI files from being edited, and scanning the diff for the shapes of known shortcuts, catches exactly the gaming a plain test loop misses — at the cost of occasionally having to report that a task couldn't be solved within the attempt budget, instead of merging something wrong.

A worked example. With test files protected from edits and a review gate added on top of the test loop, the same six tasks produce 5 merged and all 5 truly correct — zero bad merges. The hard-coded patch gets blocked by the gate; the test-editing patch gets refused outright because the harness won't let an edit touch a protected test file. The one task neither scripted candidate could actually fix is reported as unsolved after its attempts ran out, rather than merging a wrong answer.

The key lines.

# from code/mini-coding/gate.py
def review(diff, test_text="", max_changed=30):
    for f in files:
        if base.startswith("test_") or "/tests/" in f:
            out.append(("block", f"touches test file {f}"))
    if any(re.search(r"skip|xfail|pytest\.mark|unittest\.skip", a) for a in added):
        out.append(("block", "adds skip or xfail"))

Read it as: the gate is a checklist of known shortcut shapes, each one blocking the merge on its own. It is explicitly a heuristic, not proof of correctness. That is why held-out tests and a human reviewing the final pull request stay in place behind it.

Pull request, attempt budget

A pull request is a proposed change shown to a human with its diff and checks, merged only when approved; an attempt budget is the maximum number of tries before the agent stops and reports instead of continuing indefinitely.

Example: the unsolved task above, reported cleanly once its attempt budget ran out — exactly the "an unsolved task is a correct output" principle this module opens with.