Part 2 of 12 · 2 min read
RLVR grades attempts with a program, not a learned preference
A unit test doesn't care about style or tone, only whether the code actually works, which is exactly the kind of reward RLVR trains on.
Think of a spelling bee judge versus a taste-test judge. The spelling bee judge checks the word against a fixed list, with no interpretation involved. The taste-test judge is making a subjective call every time.
In plain words: reinforcement learning with verifiable rewards (RLVR) scores each attempt with an automated checker — a test, an answer match, a schema validator — rather than a learned reward model, which works well exactly where there's a clear right answer and poorly everywhere else.
A worked example. In the toy, each prompt needs an exact 3-step chain of token choices, and only one chain out of 64 possible combinations is correct per prompt — so the starting policy, sampling close to uniformly, passes about 1.3 percent of attempts. The loop is: sample a batch of attempts, run each through the verifier to get a reward of 0 or 1, and nudge the policy toward whichever attempts the verifier accepted.
The key lines. From code/mini-rlvr/rl.py:
# from code/mini-rlvr/rl.py
def reward(self, p, a):
"""p: (n,) prompt ids, a: (n, L) tokens. Returns the verifier's reward (what training sees)."""
t = self.target[p]; match = (a == t)
full = match.all(1)
if self.verifier == "strong": r = full.astype(float)
Read it as: the strong verifier's reward is just "did every single step match the target chain," expressed as a plain boolean-to-float conversion. There is no partial grading and no interpretation. It is exactly the kind of rule a program can check. A learned reward model would have to be trained to approximate it.
Policy and rollout, verifiable reward (RLVR)
The policy is the model's distribution over outputs; a rollout is one sampled attempt (here, a chain of tokens; for an agent, a whole episode). RLVR is reinforcement learning where the reward comes from a program or checker instead of a learned preference model.
Example: the full.astype(float) line above is the entire reward function for the strong verifier — a program, not a model, and the simplest possible instance of what RLVR means.