Part 3 of 12 · 2 min read
pass@k asks if it can ever succeed; pass^k asks if you can rely on it
A task an agent solves on one try out of four is not the same as a task it solves reliably every time.
Think of a locksmith who can pick a specific lock eventually, given four attempts. Compare that with one who picks it correctly on the first attempt every single time. Both are "successful" by some definition. But only one is reliable enough to hire for a one-shot job.
In plain words: pass@k is the share of tasks solved by at least one of k tries, which answers "can it ever do this," while pass^k is the share solved by all k tries, which answers "can I count on it every time" -- and because pass^k is a probability raised to a power, it drops fast even when pass@k looks almost perfect.
A worked example. For a task with an 80% chance of success on any given try, pass@4 (at least one of four tries succeeds) is 0.998 -- essentially certain. But pass^4 (all four tries succeed) is only 0.41 -- worse than a coin flip. The two numbers describe the exact same underlying task and the exact same agent; they just answer two different questions, and an agent that looks nearly perfect by one measure can look unreliable by the other.
The key lines. From code/mini-flaky/flaky.py:
# from code/mini-flaky/flaky.py
def pass_at_k(p, k):
"""At least one of k independent tries succeeds."""
return 1 - (1 - p) ** k
def pass_hat_k(p, k):
"""All k independent tries succeed (consistency)."""
return p ** k
Read it as: pass_at_k computes "1 minus the chance every try fails," while pass_hat_k computes "the chance every try succeeds." The two formulas are near-opposites, built from the same p. That is why they move in opposite directions as k grows. More tries make "at least one works" easier and "every one works" harder.
pass@k, pass^k, consistency
pass@k is the share of tasks solved by at least one of k tries. pass^k (the notation used by tau-bench) is the share solved by all k tries; the probability of success per try raised to the power k when tries are independent. Consistency is giving the same correct result across repeated tries.
Example: reporting only pass@4 = 0.998 for an 80%-per-try task would hide that pass^4 = 0.41 -- a team that ships on the first number alone might be surprised when the agent fails on the very next real request, because "it worked at least once in testing" and "it works reliably in production" are different claims.