Part 2 of 12 · 2 min read
The right final state can still hide an unsafe path to get there
An agent that reaches the correct outcome by skipping a safety step is not correct, it is lucky.
Think of a surgeon who gets the right outcome but skipped sterilizing their instruments: the patient recovering doesn't retroactively make the missing step acceptable.
In plain words: an outcome eval scores the final result or the final state of the world, while a trajectory eval scores the sequence of steps the agent actually took to get there -- and because an agent acts on real systems, a trajectory can be unsafe or wasteful even when the outcome it produces looks perfectly fine.
A worked example. A fictional support task (look up an order, verify identity, refund, send email) has one run that skips the identity-verification step entirely, going straight from lookup to refund. The final outcome -- a refund issued -- is still correct, so an outcome-only check passes it 100% of the time. But the step that exists specifically to prevent an unauthorized refund never happened; the agent got lucky that this particular case didn't need it.
The key lines. From code/mini-trajectory/trajectory.py:
# from code/mini-trajectory/trajectory.py
def make_run(kind, rng):
if kind == "reference":
return list(REF), True
...
if kind == "skipped_verify":
return ["lookup_order", "refund", "send_email"], True # outcome looks right
Read it as: the comment on the skipped_verify case says it outright: "outcome looks right." The code is deliberately constructing a case where the final state is indistinguishable from a correct run. Its purpose is to prove that scoring the outcome alone cannot tell the two apart.
Outcome eval, trajectory eval
An outcome eval scores the final result or the final state of the world. A trajectory eval scores the sequence of steps, such as tool calls and their arguments.
Example: the skipped-verification run passing every outcome check is the entire argument for trajectory evaluation stated as one data point -- an agent that reaches the right answer by an unsafe shortcut is a security incident that happens to look, from the outcome alone, like a success.