CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 2 of 12 · 2 min read

A leaderboard rank on a small benchmark is less stable than it looks

The agent ranked first on a 50-task benchmark might just be the one that got lucky on those particular 50 tasks.

Think of ranking five very similarly skilled runners by the result of one short sprint. A strong gust of wind at the wrong moment could easily flip who crosses the line first. Their real abilities barely differ.

In plain words: a benchmark is a fixed set of tasks and a scoring rule used to compare systems, a leaderboard is a ranked list of reported scores, and a harness (scaffold) is the code, prompts and tools wrapped around a model when it runs on the benchmark -- and because a benchmark is still just a finite sample of tasks, the agent that tops the leaderboard on a small one is not guaranteed to be the genuinely best agent.

A worked example. Five simulated agents with skills 0.50, 0.53, 0.56, 0.59 and 0.62 (deliberately close together, like real competing models) are ranked against each other repeatedly on benchmarks of different sizes. On a 50-task benchmark, the truly best agent (skill 0.62) actually comes out ranked first only 54% of the time -- essentially a coin flip plus a bit. On a 1,000-task benchmark, that rises to 96%. The same five agents, the same true skill gap, but the benchmark's size alone determines whether the leaderboard reliably reflects reality.

The key lines. From code/mini-bench/bench.py:

# from code/mini-bench/bench.py
def top1_correct(skills, n_tasks, trials=600, seed=1):
    """Share of benchmark runs in which the truly best agent is ranked first."""
    rng = random.Random(seed)
    hit = 0
    for _ in range(trials):
        diffs = task_difficulties(n_tasks, rng)
        scores = [sum(run_agent(s, diffs, rng)) for s in skills]
        best = max(range(len(skills)), key=lambda i: (scores[i], rng.random()))
        hit += best == skills.index(max(skills))
    return hit / trials

Read it as: the function reruns the entire ranking 600 times with different random task sets. Each time it checks only whether the agent with the highest true skills value also got the highest simulated score. Every one of those 600 trials uses the exact same five agents with the exact same true skills. So any disagreement across trials about who is "first" is pure sampling noise, not a change in ability.

Benchmark, leaderboard, harness

A benchmark is a fixed set of tasks and a scoring rule used to compare systems. A leaderboard is a ranked list of reported scores. A harness (scaffold) is the code, prompts and tools wrapped around a model when it is run on the benchmark.

Example: a leaderboard rank that's only 54% reliable at 50 tasks is a reminder that the rank itself is a measurement with error, the same way a single eval score is (module 1) -- and comparing two leaderboard rows is only fair when their harnesses, not just their scores, are the same.