CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 5 of 12 · 2 min read

The proxy can be gamed, so check something it was never trained on

A learned reward going up is not the same as the real goal going up. Goodhart's law applies whenever a proxy stands in for what you actually want.

Think of a student who discovers the grading rubric rewards citation count, not argument quality, and starts padding essays with citations. Their grade climbs while the actual essay gets worse.

In plain words: a learned reward model is a proxy for what people actually want, and optimizing hard enough against a proxy. Especially a noisy or poorly-regularized one. Can raise the proxy's score while the true underlying quality falls.

A worked example. With 1,000 pairs and labels increasingly flipped (simulating noisy human disagreement): at 0 percent flipped, DPO reaches 2.04 true reward. At 20 percent flipped, 1.69. At 40 percent flipped, only 0.83. Noise alone cuts the achievable reward by more than half. Separately, with 100 noisy pairs and a small beta, 60 percent of 20 simulated runs ended up worse than the untuned reference model on true reward. This happened even though the learned (proxy) reward rose from 0.20 to 3.24 in the worst case, while the true reward fell from 0.85 to -0.39. Best-of-N (sampling N responses and keeping the one the reward model scores highest) shows the same shape. True reward rises from 0.80 at N=1 to 1.83 at N=16, then actually dips slightly to 1.81 at N=32. Pushed far enough, the reward model's own errors start getting exploited.

The key lines.

# from code/mini-preference/pref.py
def rlhf(world, R, beta, steps=2000, lr=0.5, track=False):
    """Maximise E_pi[R] - beta KL(pi || pi_ref) by exact policy gradient on the logits."""
    ...
    if track and t % 25 == 0:
        hist.append(dict(step=t, proxy=float((p * R).sum(-1).mean()), true=true_value(world, th), kl=kl(p, softmax(ref))))

Read it as: the tracking code records proxy (the learned reward) and true (the real reward) as two separate numbers at every checkpoint. The only way to catch the proxy diverging from the real goal is to measure both side by side, which is exactly what let this example's worst run be caught at all.

Reward over-optimization (Goodhart), best-of-N

Over-optimization, or Goodhart's law here, means pushing a proxy reward until the true objective it was supposed to track actually gets worse. Best-of-N means sampling N outputs and keeping the one the reward model scores highest, as a cheap alternative to actually training a policy.

Example: best-of-N's dip from N=16 to N=32 above is over-optimization in miniature. More samples should only ever help if the reward model is a perfect proxy, and the dip is direct evidence that it isn't.