Part 3 of 12 · 2 min read
Beta is the leash between gaining reward and drifting from what you trust
Optimizing purely for a reward model with no restraint finds whatever the reward model rewards, including answers nobody would actually want.
Think of a dog on a leash chasing a thrown ball: a short leash keeps it close and safe but limits how far it can run after the ball. A long leash lets it get the ball faster but it can also run into traffic.
In plain words: the RLHF objective doesn't just maximize reward. It maximizes reward minus beta times how far the new policy has drifted (in KL divergence) from the reference model, and beta is the dial that sets how tight that leash is.
A worked example. With beta at 0.05 (a long leash), the optimal policy reaches a true reward of 2.16 but drifts a KL distance of 1.52 from the reference. With beta at 3 (a short leash), it reaches only 1.21 true reward but stays within a KL of just 0.06. Every intermediate beta value traces out this same trade-off. More reward always costs more distance from the model you started trusting.
The key lines. From code/mini-preference/pref.py:
# from code/mini-preference/pref.py
def optimal_policy(world, beta, reward=None):
"""Closed-form maximiser of E[r] - beta KL(pi||pi_ref): pi* proportional to pi_ref * exp(r / beta)."""
if beta <= 0:
raise ValueError("beta must be positive")
r = world["rstar"] if reward is None else reward
return softmax(world["ref_logits"] + r / beta)
Read it as: the reward is divided by beta before being added to the reference logits. A small beta means the reward term dominates the reference and pushes the policy far from it. A large beta shrinks the reward's influence until the policy barely moves off the reference at all.
KL divergence and the reference model
The reference model is the starting (usually SFT) policy that tuning is not supposed to drift too far from. KL divergence measures how far the new policy's distribution is from that reference. Both RLHF and DPO penalize it with weight beta.
Example: the 1.52-to-0.06 KL range across the beta sweep above is exactly what "penalize divergence with weight beta" produces in practice. Not an abstract formula, but a policy that visibly stays closer or wanders further depending on one number.