Part 9 of 12 · 2 min read
Mistakes that pick the wrong model or route badly
Common mistakes, and what the example does not do.
Mistakes that pick the wrong model or route badly
- Comparing price per token without utilisation and staffing.
- Assuming a model fits because the weights do, ignoring the cache.
- Routing before measuring the small model.
- A confidence signal nobody has evaluated.
- Ignoring licence terms on output use or user counts.
- Switching models on 100 test items.
- Not pinning the open-weight version or the API model version.
- Quoting vendor benchmark gaps as if they applied to your task.
What the example does not do
About this example What the example does not do
- No real model: correctness is a sigmoid of skill minus difficulty; real errors are structured and correlated with the confidence signal in messier ways.
- Assumed prices and hardware: only the 2 TB/s A100 bandwidth is a published figure; GPU rental, API price, device bandwidths and engineering cost are assumed.
- Throughput is a bytes-over-bandwidth model with a multi-head cache layout; it ignores compute, scheduling, prefill and network time.
- Router and confidence signal are simulated, not trained on text.
- The power calculation uses a normal approximation and one family of models; it does not cover multiple metrics or multiple comparisons.
- No safety, privacy-compliance or licensing analysis beyond the three licence questions in the exercises.
- Optional real experiment: take 500 real requests from your agent, run them through a small open model (for example with llama.cpp or vLLM) and a hosted model, score both with the same judge and compute the paired differences and the share of requests where a cheap confidence check agrees with the judge.
Check yourself
Why can an idle self-hosted GPU cost more than the API?
Show answer
You pay for the hour whether tokens flow or not. In the model batch 32 at full load cost $0.55 per million tokens but $5.46 at 10 percent utilisation, against an assumed $6 API price.
For the curious Router quality is everything
A cascade with an informative confidence signal came within 2 points of the large model at cost 16.0, against 20 for always using the large model. With no signal it cost 20.2, worse than always using the large model, because the small call was wasted. The saving shrinks as the price ratio shrinks (about 5 percent at 5x, 20 percent at 20x). A few hundred test items cannot show a 2-point difference: the paired test needed about 3,651. Prices, hardware and the correctness model are assumed; no real model was served.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Size memory before you choose | Weights = parameters x bytes, plus the KV cache. |
| Compute cost per token at real utilisation | Include engineering time. |
| Route only after measuring the small model | On your own traffic and your own eval. |
| Read the licence for each model | Commercial use, restrictions, output use. |