Part 2 of 12 · 2 min read
One trial tells you almost nothing about a non-deterministic agent
Asking an agent the same question twice can get two different answers, so a single run is a coin flip, not a verdict.
Think of asking a friend for directions on two different days. They might give you a slightly different route each time, even to the same destination. Small things vary, such as their mood, what they remember, or which street came to mind first.
In plain words: non-determinism means the same input can give different outputs, a trajectory is the full sequence of reasoning, tool calls and observations in one run, and a trial is one complete run of the agent on one task -- so judging an agent from a single trial is judging it from a single trajectory out of many it could have taken, which tells you almost nothing about what it will do next time.
A worked example. The simulation builds a suite of tasks whose true per-task difficulty varies (drawn from a Beta distribution with a mean around 0.7), then runs the whole suite once to get one suite score. Doing that twice for the exact same, unchanged agent still produces two different suite scores purely from which tasks happened to succeed on which run -- at 20 tasks, the standard deviation of that single-run score is 0.077, meaning a swing of several points between two runs of the identical agent is completely ordinary.
The key lines. From code/mini-flaky/flaky.py:
# from code/mini-flaky/flaky.py
def run_suite(task_ps, rng):
return sum(rng.random() < p for p in task_ps) / len(task_ps)
def make_suite(n, mean, rng, spread=2.0):
a = mean * spread
b = (1 - mean) * spread
return [rng.betavariate(a, b) for _ in range(n)]
Read it as: run_suite draws one random outcome per task from that task's own fixed success probability, not from an average. The suite score is the sum of many individual coin flips. That is why running the same suite twice gives two different totals. Nothing about the agent changed. Only which flips came up heads changed.
Non-determinism, trajectory, trial
Non-determinism is the same input giving different outputs. A trajectory is the sequence of reasoning, tool calls and observations in one run. A trial is one complete run of the agent on one task.
Example: the 0.077 standard deviation at 20 tasks is non-determinism made visible as a number -- it's the size of the swing you should expect between two trials of the same unchanged agent, before you've even changed anything about it.