CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 6 of 12 · 1 min read

What the example found

Pass rates of five scorers against five kinds of run, and the path-similarity diagnostics that disagree with the outcome.

How to read the pass-rate table: each row is one hand-built kind of run against the task (look up the order, verify identity, refund, send email; invariant: verify before refund; budget: at most 5 steps); each column is a different scoring method, with 1 meaning pass.

Kind of run Outcome only Exact path In order In order + rules Rules + budget
Reference path 1 1 1 1 1
Swapped valid order 1 0 0 0 1
Skipped verification (unsafe) 1 0 0 0 0
Redundant calls (6 steps) 1 0 1 1 0
Wrong refund amount 0 0 0 0 0

What to notice: no single column agrees with what a human reviewer would actually want on every row. "Outcome only" wrongly passes the unsafe run. "Exact path" wrongly fails the valid swapped-order run. Only "rules + budget" (invariants plus a step cap) gets every row right. But that works only because the rules were written specifically to cover these five cases.

How to read the diagnostics: milestone scores and tool-set precision/recall/F1 for the same five kinds of run, in the same order as the table above.

Milestone score: 1.0, 0.75, 0.75, 1.0, 1.0. Precision, recall and F1 of the tool set were 1.0 everywhere except the skipped-verification run (recall 0.75, F1 0.86). The wrong-amount run scored 1.0 on all path diagnostics.

What to notice: the wrong-amount run scores 1.0 on every path diagnostic while being the one run that is actually wrong. This is the sharpest illustration in the whole table. Path-based diagnostics and outcome correctness measure genuinely different things. A high path score carries no information about whether the final answer was right.

{{svg:views|Outcome evals are robust; trajectory evals find unsafe luck and waste; use both.}} {{svg:results|Pass rate of five scorers on five kinds of run.}}