CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Start with a verifier you trust and attack it yourself before training: look for shortcuts.
  • Keep a held-out, independent check (another verifier, a human audit) and plot it beside the training reward.
  • Track pass rate per prompt; drop prompts that are always solved or never solved, and build a curriculum.
  • Use groups large enough to see both outcomes, often 8 to 16, and measure the share of groups with signal.
  • Tune each algorithm's learning rate separately before comparing.
  • Prefer outcome rewards; add partial credit only for long horizons and audit it.
  • Make the environment cheap, resettable and sandboxed; an agent that can edit its own tests will.
  • Cap and monitor behaviours such as output length, tool calls and test edits.
  • Keep SFT or a reference model as an anchor and watch the KL, but do not rely on it against hacking.
  • Evaluate on tasks the training never saw, including ones that a gamed verifier would pass.