Part 6 of 12 · 3 min read
What the toy found
Algorithm comparison, group-size trade-offs, chain-length sparsity, and reward hacking under three verifiers.
How to read the algorithm-comparison table: each row is a different advantage method; each column is a different learning rate; compare within a row (same method, rate rising) and within a column (same rate, method changing).
| Updates to 90% true success (20 seeds) | lr 1 | lr 3 | lr 10 | lr 30 | lr 100 |
|---|---|---|---|---|---|
| REINFORCE | 457 | 161 | 54 | 25 | 17 |
| Batch-mean baseline | 469 | 166 | 133 | 227 | 572 |
| RLOO | 460 | 162 | 56 | 25 | 17 |
| GRPO | 200 | 74 | 29 | 18 | 16 |
What to notice: the batch-mean baseline is the only row that gets worse at high learning rates (572 updates at lr 100, worse than its own lr 1 result). The other three methods all improve monotonically. That is why comparing algorithms at a single shared learning rate can be misleading.
How to read the group-size tables: the left table shows the compute-versus-updates trade-off; the right table shows why group size matters at all — the share of groups that carry any learning signal.
| Group size (64 rollouts per update) | Updates | Rollouts used |
|---|---|---|
| 2 | 61 | 3,882 |
| 4 | 42 | 2,710 |
| 8 | 29 | 1,846 |
| 16 | 27 | 1,696 |
| 32 | 25 | 1,626 |
| Group size | Share of groups with a learning signal at the start |
|---|---|
| 2 | 2.5% |
| 4 | 4.9% |
| 8 | 9.6% |
| 16 | 18.1% |
| 32 | 32.5% |
| 64 | 53.2% |
What to notice: both updates and rollouts keep improving from group size 2 to 32. But the gains shrink fast past 8 or 16. More group size helps mainly by raising the odds that any given group has a mix of outcomes to learn from.
How to read the chain-length table: each row is a longer chain; outcome-only and partial-credit columns face the identical chain length and starting pass rate.
| Chain length (10 seeds) | Start pass rate | Updates, outcome only | Updates, partial credit |
|---|---|---|---|
| 2 | 4.923% | 13 | 10 |
| 3 | 1.255% | 33 | 15 |
| 4 | 0.292% | 89 | 20 |
| 5 | 0.082% | 289 | 30 |
| 6 | 0.020% | 1,293 (4 of 10 not reached) | 31 |
What to notice: outcome-only updates roughly triple with each added step, while partial-credit updates grow far more slowly. The gap between the two columns is the entire cost of sparse, end-only rewards over long horizons.
How to read the verifier/KL table: each block of rows is a different verifier; within a block, KL weight rises from 0 to 1.0, and the two right-hand numbers (verifier pass, true success) should be compared to each other, not just down the column.
| Verifier and KL weight | Verifier pass | True success |
|---|---|---|
| Strong, 0 | 1.00 | 1.00 |
| Strong, 0.05 / 0.2 / 1.0 | 0.88 / 0.55 / 0.01 | 0.88 / 0.55 / 0.01 |
| Last step only, 0 | 1.00 | 0.07 |
| Last step only, 0.05 / 0.2 / 1.0 | 1.00 / 0.97 / 0.39 | 0.06 / 0.06 / 0.03 |
| Loophole, 0 | 1.00 | 0.00 |
| Loophole, 0.05 / 0.2 / 1.0 | 1.00 / 0.97 / 0.40 | 0.00 / 0.00 / 0.01 |
What to notice: with the strong verifier, verifier pass and true success are always equal. The two numbers only diverge for the weak verifiers. That divergence (1.00 pass against 0.07 or 0.00 true success) is reward hacking showing up as a number, not an assumption.
{{svg:loop|The RLVR loop, how often a group carries signal, group size and chain length.}} {{svg:hacking|Verifier pass rate against true success under three verifiers and four KL weights.}}