Part 8 of 12 · 2 min read
Answering RL-for-reasoning questions in an interview
What is RLVR and when would you use it?
"Reinforcement learning where the reward is a programmatic check: tests pass, answer matches, schema valid, tool call correct. I would use it when I have a resettable environment, a trustworthy verifier, prompts with non-trivial pass rates and the budget for many rollouts. Otherwise SFT, preference tuning or best-of-N are cheaper." Follow-up: "What goes wrong?" No signal when pass rates are near 0 or 1, reward hacking, and high compute.
Explain GRPO and how it differs from PPO.
"GRPO samples a group of attempts per prompt and uses the group's mean and standard deviation as the baseline, so it needs no learned value network; PPO normally learns a value function to compute advantages and uses clipped updates. In the example's toy GRPO was 2 to 3 times faster at the same learning rate but about tied at the best rate for each method, because the standard-deviation scaling acts like an adaptive step size." Follow-up: "What is the weakness?" Groups with identical rewards give no gradient, and scaling by the standard deviation changes how hard and easy prompts are weighted.
How do you detect and prevent reward hacking?
"Attack the verifier first, hold out an independent check, and plot the training reward against that check: in the example's toy a loophole took the verifier pass rate from 0.35 to 0.99 in ten updates while true success fell to zero. A KL penalty slowed honest learning and did not stop the hack. I would also sandbox the environment and audit samples by hand." Follow-up: "What if the hack is subtle?" Use multiple verifiers with different weaknesses, adversarial test generation and periodic human review.
Why are long agent tasks hard for RL?
"A single terminal reward gives almost no signal when the chance of success is tiny: in the example's toy the pass rate falls from 4.9 percent at two steps to 0.02 percent at six, and updates needed went from 13 to 1,293. Partial credit, easier subtasks first, or imitation from successful traces help." Follow-up: "What is the catch with partial credit?" It can be gamed: the policy may collect partial rewards without finishing.