CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 3 of 12 · 2 min read

A group with no disagreement teaches nothing

If every attempt in a batch gets the same grade, there's no "better" or "worse" to point the gradient toward.

Think of grading a class where every student failed the same exam identically. There is no way to tell the strong students from the weak ones that day. The data gives you nothing to compare.

In plain words: a group-relative method like GRPO needs its sampled group of attempts to contain both a pass and a fail to produce any gradient at all — when every attempt in the group gets the same reward, the advantage is exactly zero and nothing is learned from that batch.

A worked example. At the toy's 1.3 percent starting pass rate, a group of 8 attempts has both a success and a failure only 9.6 percent of the time — over 90 percent of groups that size carry zero learning signal at the start. Larger groups help: at group size 32, 32.5 percent of groups carry signal; at 64, 53.2 percent. This directly affects training speed: with 64 rollouts spent per update, group size 2 needs 61 updates and 3,882 total rollouts to reach 90 percent true success, while group size 32 needs only 25 updates and 1,626 rollouts — bigger groups waste fewer rollouts on all-same-outcome batches.

The key lines.

# from code/mini-rlvr/rl.py
def group_has_signal(p_success, G):
    """Probability that a group of G samples contains both successes and failures (needed for a non-zero GRPO advantage)."""
    return 1 - p_success ** G - (1 - p_success) ** G

Read it as: the formula subtracts the chance of an all-fail group and the chance of an all-success group from 1. Whatever is left over is the chance of a mixed group. That is the only kind that produces a nonzero gradient for a group-relative method.

Advantage and baseline, GRPO and RLOO

The advantage is how much better an attempt was than expected; the baseline is what gets subtracted to compute that. GRPO's advantage is (reward minus group mean) divided by group standard deviation; RLOO's baseline is the mean of the other attempts in the group, leaving the attempt itself out.

Example: the 9.6-percent-of-groups-carry-signal number above is exactly the denominator problem GRPO faces — its advantage formula divides by the group's own standard deviation, which is itself zero whenever a group has no signal to divide by.