Part 10 of 12 · 1 min read
Quick notes
- An outcome eval asks "did it end in the right state?"; a trajectory eval asks "how did it get there?"
- Outcome-only scoring passes unsafe luck: a run that skipped identity verification but issued the right refund passed every outcome check and only a rule caught it.
- Exact-path matching is too strict, because it punishes correct alternatives such as two independent steps swapped. Specify rules, not one path, and add a step or cost budget to catch redundant work.
- Path metrics cannot see wrong answers, so always score the outcome as well. Partial credit helps debugging, not gating.
- Recommended structure: the outcome decides pass or fail; invariants, a budget and tool-argument checks guard the path; milestones and path-similarity metrics are diagnostics.