Part 6 of 11 · 1 min read
Setting a goal
- Write the success criteria as checks. "Report includes three sourced findings" and "tests pass" are checkable; "a good report" is not, unless a rubric and a judge are part of the design.
- Separate the claimer from the checker. A different model, a test suite or a person; never the same prompt grading itself.
- Set limits per layer: steps per task, tokens, wall-clock time, and number of replans.
- Define the escalation path (who, with what context) before you need it, and test it with an impossible task.
- Log a progress ledger per step (subgoals done, steps idle, spend), so monitoring is also your trace (Level 9).
- Use a cheaper model for monitoring when possible, since it runs on every step; the simulation charged 0.3 model calls per step.
- Keep a budget even with a monitor, since monitors fail too.