Part 1 of 4 · 1 min read
Overview
Agents make many small choices (which tool, how to phrase a refusal, when to ask). If you have comparison data from users, graders or verifiers, preference tuning can shape those choices, and a reward model can also act as a judge or ranker at inference (best-of-N). The same machinery is behind the "reward hacking" failures that module 4 treats in agentic reinforcement learning.
About the example, and what is real
Real: the Bradley-Terry reward model, the RLHF objective with exact gradients, the DPO loss with its hand-derived gradient (checked against finite differences), the closed-form optimum, best-of-N, and the 17 tests in code/mini-preference. Assumed: the policy is a table of 6 prompts x 10 responses; the true reward is a random draw (standard deviation 1.5); the reference policy is partly informative about it; labelers follow Bradley-Terry exactly, optionally with flipped labels. Not real: no language model, no PPO (updates use the exact expected gradient, which hides sampling noise and instability), no human labelers. Do not read the RLHF-versus-DPO tie as a general result: it is what the algebra predicts in a tabular setting.