Part 4 of 12 · 2 min read
DPO skips the reward model by working the same closed form in reverse
If the optimal policy has a known closed form in terms of the reward, you can train on that relationship directly instead of fitting a reward model first and then optimizing against it.
Think of two ways to solve for a missing ingredient amount in a recipe: measure the other ingredients and compute it from the ratio, or just adjust by taste until it matches a reference dish. Both can land at the same place by a different route.
In plain words: RLHF's closed-form optimal policy implies that the reward is proportional to beta times the log-ratio of the new policy to the reference. DPO uses that relationship backward, training the policy directly on the comparison pairs without ever fitting a separate reward model.
A worked example. At beta 0.3 with 3,000 pairs, RLHF reaches a true reward of 2.06 and DPO reaches 2.07. Nearly identical, because they're optimizing the same underlying objective by two different routes. At beta 1, RLHF gets 1.65 and DPO gets 1.68. With only 100 pairs of data, both land at exactly 1.65. The two methods track each other closely across every setting in this toy, once the reward model's own calibration is accounted for.
The key lines. From code/mini-preference/pref.py:
# from code/mini-preference/pref.py
def dpo(world, data, beta, steps=2000, lr=0.5):
"""Minimise -mean log sigmoid(beta*[(log pi(w)-log ref(w)) - (log pi(l)-log ref(l))]). The softmax normalisers cancel,
so d loss / d logit(w) = -beta*s and d loss / d logit(l) = +beta*s where s = sigmoid(-z)."""
Read it as: the docstring says outright that the softmax normalizers cancel. DPO's update reduces to "raise the winner's logit, lower the loser's," which is simple enough to run without ever training or sampling from a separate reward model.
RLHF, PPO, DPO, RLAIF
RLHF is reinforcement learning from human feedback via a reward model. PPO is the reinforcement learning algorithm usually used for the optimization step. DPO is direct preference optimization, training on the comparison pairs without a reward model. RLAIF substitutes an AI judge for human preferences.
Example: the 2.06-vs-2.07 near-match above is specific to this toy's tabular setting with an exact gradient. A first, less careful run of the same experiment (noted in the module's deeper box) showed DPO well ahead of RLHF. The real cause turned out to be the reward model's own regularization setting, not a genuine advantage of the method.