CSE Roadmap

Common tier · about 5 hours · 12 parts

Part 5 of 12 · 1 min read

What the experiment measured

Merged versus truly correct across three modes, and which shortcut each task's scripted patches tried.

How to read the first table: all three rows face the identical six tasks; "Truly correct" (the held-out check) is the number that matters, not "Merged."

Mode (six tasks, scripted patches) Merged Truly correct (held-out) Bad merges Attempts
Apply the first patch, no tests 6 1 5 6
Test loop only 5 3 2 9
Test loop, protected tests, review gate 5 5 0 11

What to notice: going from a test loop to a gated test loop costs 2 extra attempts overall but eliminates every bad merge. The attempts spent arguing with the gate are cheaper than the bad merges they prevent.

How to read the second table: each row is one task's scripted candidates, read left to right as the attempts that were actually tried.

Task Pattern in the scripted candidates Gated outcome
slugify wrong, then right right on attempt 2
median wrong, then right right on attempt 2
parse_range hard-coded for the visible input, then right gate blocked the first, right on attempt 2
dedupe edits the test, then right guard refused the test edit, right on attempt 2
chunk two wrong candidates unsolved after 2 candidates, reported
moving_avg right first right on attempt 1

What to notice: the gate and the protected-path guard each catch a different kind of shortcut. The gate catches the hard-coded answer after it is proposed. The protected-path guard refuses the test edit before it can even be written to disk.

{{svg:loop|The harness around the model, from task to human review.}} {{svg:results|Merged versus actually correct in each mode.}}