CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that let an unsafe path pass

Common mistakes, and what the example does not do.

Mistakes that let an unsafe path pass

  • Outcome-only scoring for agents that take actions.
  • One exact reference path for tasks with valid alternatives.
  • No step or cost budget, so a slow, expensive agent passes.
  • Checking tool names but not arguments.
  • Using path similarity as the pass criterion.
  • Reading the final text instead of the state (the agent said "refund issued" but did not issue it).
  • Rules written after the incident, and never added to the suite.

What the example does not do

About this example What the example does not do
  • No real agent, and each kind of run is fixed, so there are no rates.
  • One task and one invariant; real suites have dozens of tools and rules.
  • Tool names only: arguments are not modelled.
  • No LLM judge for fuzzy trajectory quality (module 4).
  • No cost or latency budget, only a step count.
  • Optional real experiment: log 50 real runs of an agent, score them with outcome-only and with outcome plus invariants, and count how many outcome passes violate an invariant.

Check yourself

A run skipped identity verification but still issued the right refund. Does an outcome check catch it?

Show answer

No. It passed the outcome check every time in the five-run example. A rule such as verify before refund catches it.

For the curious Path scores cannot see a wrong answer

The wrong-amount run got full milestone credit (1.0) and precision, recall and F1 of 1.0, yet the outcome was wrong. Exact-path matching also failed a valid run that swapped two independent steps. So the outcome decides pass or fail, invariants and a step budget guard the path, and milestones are only a debugging aid. The five run kinds were hand-made, so these are illustrations, not measurements.

What you will use daily at work

Practice What it looks like
Check the end state, not the final text Does the refund exist with the right amount?
Write invariants for ordered or unsafe steps Verify before refund; never delete.
Set a step or cost budget Two redundant lookups passed every other check; only the budget flagged them.
Check tool arguments, not only names The right tool with the wrong amount is still wrong.