Part 1 of 6 · 1 min read
Overview
In this module you will
- Filter models by hard limits, shortlist them with public information, and decide with your own test set.
- Compare models on quality, cost and speed together, and find the ones worth considering (the Pareto front).
- Say whether a difference in scores is real or luck, using a confidence interval.
- Explain benchmark contamination, and why leaderboards are only a starting point.
- Decide between a hosted API and an open-weight model you run yourself.
An agent makes many model calls per task, so a model that is slightly too large multiplies cost and delay on every step, and one that is slightly too weak multiplies errors. Teams that pick by reputation, or by the top of a leaderboard, regularly overpay or ship something unreliable.
A small repeatable process beats any single recommendation, because the best model changes every few months.
Names and prices go stale
This module deliberately names no real models and no real prices. They would be out of date quickly. All the numbers here come from simulated models, so you can see the method without paying for API calls. When you do the exercises, look up current models and prices on the providers' own pages.