Part 6 of 12 · 2 min read
What the toy found
Beta sweep, data-size sweep, label-noise sweep, reward-model regularization, and best-of-N.
How to read the beta table: all rows face the same 3,000 clean pairs and 10 random worlds. Only beta changes, and RLHF/DPO/perfect should track each other closely within a row.
| Beta (3,000 clean pairs, 10 worlds) | RLHF | DPO | Perfect KL-regularised policy | KL of DPO from reference |
|---|---|---|---|---|
| 0.05 | 2.14 | 2.13 | 2.16 | 1.46 |
| 0.1 | 2.13 | 2.13 | 2.16 | 1.41 |
| 0.3 | 2.06 | 2.07 | 2.08 | 1.08 |
| 1 | 1.65 | 1.68 | 1.69 | 0.37 |
| 3 | 1.18 | 1.21 | 1.21 | 0.07 |
What to notice: RLHF, DPO and the perfect closed-form policy stay within a few hundredths of each other at every beta, which is the toy's central finding. The three columns are different ways of reaching the same target, not three different targets.
How to read the data-size table: all three columns face the same beta (0.3). Only how many comparison pairs were available to train on changes.
| Pairs of data (beta 0.3) | RLHF | DPO | Perfect |
|---|---|---|---|
| 30 | 1.36 | 1.36 | 2.08 |
| 100 | 1.65 | 1.65 | 2.08 |
| 300 | 1.78 | 1.82 | 2.08 |
| 1,000 | 2.03 | 2.04 | 2.08 |
| 3,000 | 2.06 | 2.07 | 2.08 |
What to notice: the gap to "perfect" shrinks steadily as data grows but never fully closes by 3,000 pairs. Both methods are approximating the same ceiling from below, limited by how much comparison data they've seen, not by which method is used.
How to read the label-noise table: both columns face the same 1,000 pairs and beta 0.3. Only the share of deliberately flipped labels changes.
| Labels flipped (1,000 pairs, beta 0.3) | RLHF | DPO |
|---|---|---|
| 0% | 2.03 | 2.04 |
| 10% | 1.92 | 1.93 |
| 20% | 1.68 | 1.69 |
| 30% | 1.25 | 1.28 |
| 40% | 0.78 | 0.83 |
What to notice: going from 0 to 40 percent flipped labels cuts the achievable reward by more than half for both methods equally. Noisy preference data is a shared vulnerability, not something one method handles better than the other.
How to read the reward-model-regularization table: all rows use the same RLHF run at beta 0.3 with 1,000 pairs. Only the reward model's L2 regularization strength changes.
| Reward-model L2 (1,000 pairs, RLHF at beta 0.3) | True reward | KL | Spread of fitted reward |
|---|---|---|---|
| 0.1 | 1.01 | 0.02 | 0.05 |
| 0.01 | 1.65 | 0.41 | 0.26 |
| 0.001 | 2.00 | 0.96 | 0.75 |
| 0.0001 | 2.03 | 1.13 | 1.26 |
What to notice: heavier L2 regularization shrinks the reward model's spread, and a shrunk reward behaves exactly like a larger beta would. This is the hidden setting that made an early, uncontrolled run of this experiment look like DPO was simply better than RLHF.
How to read the best-of-N table: this is a single row. Each column is a different sample count N against the same reward model.
| Best of N (reward model on 300 pairs) | N=1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| True reward | 0.80 | 1.28 | 1.60 | 1.76 | 1.83 | 1.81 |
What to notice: the true reward rises steadily through N=16 and then dips slightly at N=32. More sampling isn't free of risk once the reward model's own errors become the thing being selected for.
{{svg:pipelines|RLHF and DPO pipelines and the optimum they share.}} {{svg:results|Beta, data, noise, best-of-N and over-optimisation in the toy.}}