Part 9 of 12 · 2 min read
Mistakes that let an unsafe path pass
Common mistakes, and what the example does not do.
Mistakes that let an unsafe path pass
- Outcome-only scoring for agents that take actions.
- One exact reference path for tasks with valid alternatives.
- No step or cost budget, so a slow, expensive agent passes.
- Checking tool names but not arguments.
- Using path similarity as the pass criterion.
- Reading the final text instead of the state (the agent said "refund issued" but did not issue it).
- Rules written after the incident, and never added to the suite.
What the example does not do
About this example What the example does not do
- No real agent, and each kind of run is fixed, so there are no rates.
- One task and one invariant; real suites have dozens of tools and rules.
- Tool names only: arguments are not modelled.
- No LLM judge for fuzzy trajectory quality (module 4).
- No cost or latency budget, only a step count.
- Optional real experiment: log 50 real runs of an agent, score them with outcome-only and with outcome plus invariants, and count how many outcome passes violate an invariant.
Check yourself
A run skipped identity verification but still issued the right refund. Does an outcome check catch it?
Show answer
No. It passed the outcome check every time in the five-run example. A rule such as verify before refund catches it.
For the curious Path scores cannot see a wrong answer
The wrong-amount run got full milestone credit (1.0) and precision, recall and F1 of 1.0, yet the outcome was wrong. Exact-path matching also failed a valid run that swapped two independent steps. So the outcome decides pass or fail, invariants and a step budget guard the path, and milestones are only a debugging aid. The five run kinds were hand-made, so these are illustrations, not measurements.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Check the end state, not the final text | Does the refund exist with the right amount? |
| Write invariants for ordered or unsafe steps | Verify before refund; never delete. |
| Set a step or cost budget | Two redundant lookups passed every other check; only the budget flagged them. |
| Check tool arguments, not only names | The right tool with the wrong amount is still wrong. |