Part 7 of 12 · 1 min read
How to build it so it holds up
- Start with a verifier you trust and attack it yourself before training: look for shortcuts.
- Keep a held-out, independent check (another verifier, a human audit) and plot it beside the training reward.
- Track pass rate per prompt; drop prompts that are always solved or never solved, and build a curriculum.
- Use groups large enough to see both outcomes, often 8 to 16, and measure the share of groups with signal.
- Tune each algorithm's learning rate separately before comparing.
- Prefer outcome rewards; add partial credit only for long horizons and audit it.
- Make the environment cheap, resettable and sandboxed; an agent that can edit its own tests will.
- Cap and monitor behaviours such as output length, tool calls and test edits.
- Keep SFT or a reference model as an anchor and watch the KL, but do not rely on it against hacking.
- Evaluate on tasks the training never saw, including ones that a gamed verifier would pass.