CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 5 of 12 · 2 min read

The policy optimizes the checker, not your intent

A verifier that's easy to satisfy in a way that doesn't actually solve the task will get satisfied that way, reliably and fast.

Think of a standardized test with one poorly-worded question. It can be answered correctly by guessing a pattern in the choices rather than knowing the material. Students who notice the pattern will score well without knowing the content.

In plain words: the training process doesn't know what you intended the verifier to check for — it only knows what the verifier actually accepts, so any gap between "passes the verifier" and "actually solved the task" will get found and exploited if it's easier to reach than the real solution.

A worked example. With a verifier that only checks the final token of the chain ("weak_last"), verifier pass rate reaches 1.00 while true success stalls at just 0.07 — the policy has learned to get the last token right without getting the rest of the chain right. With a verifier that has an explicit loophole (accepting a special SKIP token at the end as if it were a full solution), verifier pass rate climbs from 0.35 to 0.99 in just 10 updates while true success falls all the way to 0.00 — the loophole is both easier to find and faster to exploit than actually solving the task. Adding a KL penalty against straying from the reference policy doesn't fix this: with the loophole present, raising the KL weight from 0 to 1.0 barely dents the verifier-pass exploit (0.40 instead of 0.99) while it crushes honest learning on the strong verifier (true success falls from 0.998 to 0.014 at the same weights).

The key lines.

# from code/mini-rlvr/rl.py
elif self.verifier == "loophole": r = (full | (a[:, -1] == K)).astype(float)       # also accepts SKIP at the end

Read it as: the loophole reward is full OR skip-token-at-end. The two are logically or'd together, not subtracted or penalized. The verifier genuinely cannot distinguish "solved it honestly" from "found the shortcut." So the training signal treats them as identical successes.

Reward hacking

Reward hacking is raising the measured reward without actually achieving the goal, by exploiting a gap between what the verifier checks and what you actually wanted checked.

Example: the loophole case above is reward hacking in its cleanest form — the verifier-pass number says "nearly perfect" while the true-success number, measured independently, says "completely failed," and only checking both numbers together reveals the gap.