Part 8 of 12 · 2 min read
Answering testing questions in an interview
How do you test an agent whose output changes on every run?
"I treat the score as a measurement. I run each case several times, report the mean with an interval, and report both pass@k and pass^k, because capability and reliability differ: for a 0.8 task, pass@4 is 0.998 but all four passing is 0.41. I fix the environment where I can, record tool outputs, and keep the suite big enough: in the example's simulation, two identical agents on 20 tasks looked 3 points apart 84 percent of the time." Follow-up: "How many cases?" Enough for the interval to be narrower than the effect I care about; for a proportion near 0.8 that is about 246 independent trials for plus or minus 5 points, and more if errors cluster.
Why do long agent trajectories fail more?
"Reliability compounds multiplicatively: 95 percent per step is 60 percent over ten steps and 8 percent over fifty. So I shorten the path, verify and retry steps, and checkpoint. The numbers assume independent steps, which overstates the effect when steps are correlated and understates it when one early mistake poisons the rest." Follow-up: "What does a retry cost?" Extra calls and latency, and it only helps if the failure is detected.
What is the difference between pass@k and pass^k?
"pass@k is the share of tasks solved by at least one of k tries: it answers whether the agent can ever do the task. pass^k is the share solved by all k tries: it answers whether I can rely on it every time. For an 80 percent per-try task, pass@4 is 0.998 but pass^4 is only 0.41, so the two numbers can tell very different stories about the same agent." Follow-up: "Which one matters for production?" Usually pass^k, because production needs the agent to succeed on the one try a real user gets, not eventually across several.
A new prompt scores 3 points higher. Do you ship it?
"Not yet. I check the suite size, the interval, and whether the gain exceeds the run-to-run noise measured by rerunning the old prompt. With a small suite the same agent can show that gap by chance. I use paired comparison on the same cases and more trials." Follow-up: "What if you cannot afford more runs?" Prefer a rollout with monitoring (module 9) over a decision on weak evidence.