Part 8 of 12 · 1 min read
Answering outcome-versus-trajectory questions in an interview
Why isn't checking the final answer enough for an agent?
"Because the agent acts. It can reach the right end state by an unsafe or wasteful path: in the example's example a run that skipped identity verification still issued the correct refund, and outcome-only scoring passed it. I add invariants and budgets on the trajectory and keep the outcome as the gate." Follow-up: "Why not compare against a reference trajectory?" Exact matching rejects valid alternatives; my swapped-order run failed it. I specify rules instead of one path.
How would you evaluate tool use?
"Three layers: did it choose the right tool, were the arguments right, and was the sequence acceptable. I check names and arguments against expectations or schemas, enforce ordering and forbidden-call invariants, and count calls against a budget." Follow-up: "What if two tools would both work?" Any-order or multiple accepted sets, not one exact tool name.
What is partial credit for, and what is it not for?
"It helps locate where runs fail and rank near misses, for example a milestone score based on the longest common subsequence with the reference. It should not gate releases: the wrong-amount run got full milestone credit." Follow-up: "Would you ever use an LLM for this?" For fuzzy steps (was the explanation faithful?), calibrated against human labels (next module), not for checks that code can do exactly.
How do invariants relate to safety?
"They turn policies into tests: 'never call refund without verification' is a property of the path, checkable on every test trace and, with the same code, on production traces. It is detection, not prevention; prevention is a gate in the tool layer (Level 10)." Follow-up: "What about unknown unsafe paths?" Run them through monitoring and red-teaming (Level 10) and add each new violation as an invariant.