Part 4 of 12 · 2 min read
Sparse rewards over long chains make success exponentially rare
A single pass/fail signal at the very end of a long sequence of steps gives almost nothing to learn from, because the chance of getting every step right by accident shrinks fast.
Think of a relay race scored only by whether the baton crosses the finish line. If any one of six runners drops it, there is no record of which handoff actually failed. There is just one final pass or fail for the whole chain.
In plain words: when the only reward comes at the very end of a multi-step chain, the chance of a fully correct chain shrinks multiplicatively with each added step, so sparse, outcome-only rewards become almost useless to learn from as the chain gets longer — partial credit for correct leading steps changes that.
A worked example. With outcome-only reward, a 2-step chain starts at a 4.9 percent pass rate and needs 13 updates to reach 90 percent true success. A 6-step chain starts at just 0.02 percent and needs 1,293 updates — and even then, 4 of 10 seeded runs hadn't gotten there after 1,500 updates. Giving partial credit for correct leading steps changes the picture entirely: the same 6-step chain needs only 31 updates instead of 1,293, because the policy gets useful signal on partially-correct attempts instead of waiting for a fully correct one by chance.
The key lines. From code/mini-rlvr/rl.py:
# from code/mini-rlvr/rl.py
if self.process: # partial credit: correct leading steps
lead = np.cumprod(match, axis=1).sum(1) / self.L
r = np.maximum(r, lead) if self.verifier == "strong" else r
Read it as: cumprod on the per-step matches means the credited prefix stops counting the instant one step is wrong. A chain correct for 2 of 3 steps still only gets credit for however many steps were right before the first mistake. It does not get credit for however many matched overall.
Sparse reward and credit assignment, process reward / partial credit
A sparse reward gives feedback only at the end of a sequence. Credit assignment is the problem of deciding which earlier steps deserve credit or blame for that final outcome. A process reward (partial credit) scores intermediate steps directly instead of only the final outcome.
Example: the 1,293-to-31-update improvement above is credit assignment solved by brute force — instead of making the policy infer which step mattered, partial credit tells it directly.