Part 3 of 13 · 2 min read
A test that fails on the old code and passes on the new one is proof, not a claim
Saying "I fixed it" and showing a test that only ever passes are the same amount of evidence: none.
Think of a padlock a locksmith claims to have repaired. The only real proof is watching it fail to open with the old key and then open with the new one. Watching it open once, with no "before," proves nothing about whether it was ever actually broken.
In plain words: a regression test is a test that fails on the buggy code and passes on the fixed code, so the bug cannot return unnoticed -- and the only way to know a test actually does this is to run it against both versions, because a test that merely passes on the fixed branch could be passing for reasons that have nothing to do with the bug.
A worked example. Run against the unmodified main branch, the "good" branch's two new tests both fail (2 failed, 3 passed -- the 3 are the pre-existing tests, unaffected). Run against the fixed branch, all 5 pass. The "no-test" branch's fix passes the good branch's tests too (5 passed) -- so the fix itself is correct, but without its own test, there is nothing stopping a future change from silently breaking it again, because nothing would fail.
The key lines. From code/mini-pr/repo_lab.py:
# from code/mini-pr/repo_lab.py
git(repo, "checkout", "-q", "-b", "good")
write(repo, "tinyagent/parse.py", PARSE_FIXED)
write(repo, "tests/test_parse.py", TESTS_BASE + TESTS_ADDED)
git(repo, "commit", "-qam", "Fix tool-call parsing when prose surrounds a fenced block\n\nExtract the first fenced block with a regex instead of stripping backticks.\nAdds two regression tests.")
Read it as: the fix and its tests are committed together, on a branch built from real git. The module runs actual pytest against this actual repository. It does not describe what a test result would look like. That is the only way the "2 failed on main, 5 passed on the fix" claim means anything. It is a real, reproducible run, not an assertion.
Regression test
A regression test is a test that fails on the buggy code and passes on the fixed code, so the bug cannot return unnoticed.
Example: "the new tests failed on main (2 failed) and passed on the fixed branch (5 passed)" is the regression test doing its one job -- proving the fix fixes something, rather than just existing alongside a change and passing by coincidence.