CSE Roadmap

Common tier · about 5 hours · 12 parts

Part 3 of 12 · 2 min read

Why passing visible tests isn't correctness

A test loop fixes honest mistakes, but a shortcut that games the visible tests looks identical to a real fix.

Think of a student who has seen the exam questions in advance. They can pass the exam perfectly without having learned the subject at all. Nothing about the passing grade reveals that.

In plain words: an agent that only has to satisfy the tests it can see can pass them by hard-coding the expected answer or quietly weakening the test itself, and a plain test loop cannot tell that apart from an actual fix.

A worked example. Across six scripted bug-fix tasks, applying the first patch with no testing at all merges 6 changes but only 1 is actually correct by a held-out test the agent never saw. Adding a test loop (keep retrying until the visible tests pass) improves that to 5 merged, 3 truly correct — but the other 2 "fixes" are a hard-coded answer tailored to the one visible input, and a test that was quietly edited to accept a wrong result. Both pass the visible tests perfectly.

The key lines. From code/mini-coding/gate.py:

# from code/mini-coding/gate.py
def literals(text):
    """String literals of 3+ characters, found with Python's tokenizer (numbers are ignored to limit false positives)."""
    ...
lits = literals(test_text)
for a in added:
    if "==" in a or "in (" in a or "in [" in a:
        if any(l in a for l in lits):
            out.append(("block", f"special-cases a literal that appears in the tests: {a.strip()[:60]}"))

Read it as: the check looks for a literal value that only appears because it is in the visible test, then shows up compared directly in the new code. That is exactly the shape of a hard-coded answer, as opposed to a general fix.

Held-out tests, test gaming

Held-out tests are tests the agent never sees, used only to judge whether a merged change is actually correct; test gaming is making the visible tests pass without fixing the real behaviour, such as hard-coding an expected value or editing the test itself.

Example: the hard-coded parse_range patch above, which passed its visible test but failed the held-out one — exactly what a held-out test exists to catch.