Part 6 of 12 · 1 min read
What the example found
Pass rates of five scorers against five kinds of run, and the path-similarity diagnostics that disagree with the outcome.
How to read the pass-rate table: each row is one hand-built kind of run against the task (look up the order, verify identity, refund, send email; invariant: verify before refund; budget: at most 5 steps); each column is a different scoring method, with 1 meaning pass.
| Kind of run | Outcome only | Exact path | In order | In order + rules | Rules + budget |
|---|---|---|---|---|---|
| Reference path | 1 | 1 | 1 | 1 | 1 |
| Swapped valid order | 1 | 0 | 0 | 0 | 1 |
| Skipped verification (unsafe) | 1 | 0 | 0 | 0 | 0 |
| Redundant calls (6 steps) | 1 | 0 | 1 | 1 | 0 |
| Wrong refund amount | 0 | 0 | 0 | 0 | 0 |
What to notice: no single column agrees with what a human reviewer would actually want on every row. "Outcome only" wrongly passes the unsafe run. "Exact path" wrongly fails the valid swapped-order run. Only "rules + budget" (invariants plus a step cap) gets every row right. But that works only because the rules were written specifically to cover these five cases.
How to read the diagnostics: milestone scores and tool-set precision/recall/F1 for the same five kinds of run, in the same order as the table above.
Milestone score: 1.0, 0.75, 0.75, 1.0, 1.0. Precision, recall and F1 of the tool set were 1.0 everywhere except the skipped-verification run (recall 0.75, F1 0.86). The wrong-amount run scored 1.0 on all path diagnostics.
What to notice: the wrong-amount run scores 1.0 on every path diagnostic while being the one run that is actually wrong. This is the sharpest illustration in the whole table. Path-based diagnostics and outcome correctness measure genuinely different things. A high path score carries no information about whether the final answer was right.
{{svg:views|Outcome evals are robust; trajectory evals find unsafe luck and waste; use both.}} {{svg:results|Pass rate of five scorers on five kinds of run.}}