The step that actually decides: a small test set, cost and speed recorded, and a simulated four-model comparison.
This step decides. Build an evaluation set from real, representative tasks (Level 9 goes deep; here is the minimum):
50 to 200 cases, taken from real data where you can. Tag each as easy or hard.
A scoring rule you can run automatically (exact match, a check function, or a rubric graded by a model whose grading you have spot-checked against human judgement).
The same cases for every model, so comparisons are paired.
Cost and latency recorded alongside quality.
The example evaluates four simulated models on 150 cases (100 easy, 50 hard). A simulated model answers each case correctly with a fixed probability that depends on the tag. Real output:
Model
Accuracy (95% interval)
Cost per 1,000 tasks
Mean latency
frontier-large
0.94 (0.90 to 0.97)
$24.00
4.2 s
mid-size
0.87 (0.82 to 0.93)
$4.80
1.6 s
small-fast
0.81 (0.75 to 0.87)
$0.40
0.5 s
open-14b
0.82 (0.75 to 0.88)
$0.20
1.0 s
Because these models are simulated, we know their true accuracy, so we can see how far a 150-case measurement can stray. The true values are 0.920, 0.873, 0.763 and 0.813. The measurements were 0.94, 0.873, 0.807 and 0.82: close, but small-fast read 4 points too high. A single test run is a noisy estimate.
Cost per 1,000 tasks comes from the prices: each task uses 800 input and 200 output tokens, so for frontier-large (15 per million input, 60 per million output) it is (800 x 15 + 200 x 60) / 1,000,000 = $0.024 per task, or $24.00 per 1,000.
Cost against accuracy. Blue models are on the Pareto front; grey is dominated.
Pareto front
The set of options where nothing else is both cheaper and at least as good. A model off the front is never the best choice on these two criteria alone.
The code reports the front as open-14b, mid-size and frontier-large. small-fast is dominated by open-14b, which costs less and scored one point higher. Be careful: that one point is within noise, and the true accuracies (0.763 against 0.813) differ more than the measured ones. The front is only as good as the measurements.
Is the gap real?
Look at the intervals. A bootstrap confidence interval shows how much an accuracy could move with a different sample: resample the cases with replacement many times and take the middle 95% of the resampled accuracies.
For two models on the same cases, compare them paired: take each case's difference (1, 0 or -1) and bootstrap those. In the example, frontier-large minus mid-size has a paired interval of +0.007 to +0.12, so the larger model is probably better, but possibly only slightly. Two tests make the lesson concrete:
With only 40 cases, two models that truly differ by two points cannot be separated: the interval spans zero.
With 400 cases and a 25-point gap, the interval is clearly above zero.
Small test sets cannot decide close calls. If two models are close, pick on cost and speed, or collect more cases.
One caution. The simulated models make independent mistakes. Real models tend to fail on the same hard cases, which makes real paired intervals narrower than these. The method is the same; the width is not.
Deciding
State the bar before you look: for example "at least 85% on our cases, and as cheap as possible". Here cheapest_that_clears(points, 0.85) returns mid-size. If your agent can use a cheap model for easy steps and a strong one for hard steps (routing, Level 3), test that combination too.
Check yourself
Why must every model see the same cases?
Why can 40 cases not separate models that differ by two points?