Definitions

Definitions

The pipeline

Results

1. Memory sets what runs where; quantisation moves a model down a device class.
2. Self-host cost is mostly utilisation: the same GPU cost $0.55 or $59 per million tokens depending on batching and load.
3. Break-even is a volume and staffing question, and may exceed what one GPU can serve.
4. A cascade is only as good as its confidence signal, and a useless one costs more than not routing.
5. Routing pays most when the small model is already decent and the price gap is large.
6. Small accuracy gaps need thousands of items to detect.

Design rules

• Write down the seven axes and which ones are hard constraints (privacy, region, on-device) before comparing models.
• Read each licence for commercial use, user-count limits and output-use restrictions.
• Size memory as weights plus cache, then check decode speed with a bandwidth bound before renting hardware.
• Compute cost at realistic utilisation, not full load, and include engineering and on-call time.
• Measure the small model on your own traffic before designing a router; the gain depends on it.
• Make routers and confidence checks cheap and measure their own accuracy.
• Keep an escape hatch: log escalations and sample the cheap path for review.
• Use a paired comparison on a fixed evaluation set and plan the item count for the gap you care about.
• Pin versions of open weights and record them with every evaluation.

Interview questions, with the follow-ups

Common mistakes

• Comparing price per token without utilisation and staffing.
• Assuming a model fits because the weights do, ignoring the cache.
• Routing before measuring the small model.
• A confidence signal nobody has evaluated.
• Ignoring licence terms on output use or user counts.
• Switching models on 100 test items.
• Not pinning the open-weight version or the API model version.
• Quoting vendor benchmark gaps as if they applied to your task.

What the example does not do

• No real model: correctness is a sigmoid of skill minus difficulty; real errors are structured and correlated with the confidence signal in messier ways.
• Assumed prices and hardware: only the 2 TB/s A100 bandwidth is a published figure; GPU rental, API price, device bandwidths and engineering cost are mine.
• Throughput is a bytes-over-bandwidth model with a multi-head cache layout; it ignores compute, scheduling, prefill and network time.
• Router and confidence signal are simulated, not trained on text.
• The power calculation uses a normal approximation and one family of models; it does not cover multiple metrics or multiple comparisons.
• No safety, privacy-compliance or licensing analysis beyond the three licence questions in the exercises.
• Optional real experiment (I did not run this): take 500 real requests from your agent, run them through a small open model (for example with llama.cpp or vLLM) and a hosted model, score both with the same judge and compute the paired differences and the share of requests where a cheap confidence check agrees with the judge.

For the curious Router quality is everything

A cascade with an informative confidence signal came within 2 points of the large model at cost 16.0, against 20 for always using the large model. With no signal it cost 20.2, worse than always using the large model, because the small call was wasted. The saving shrinks as the price ratio shrinks (about 5 percent at 5x, 20 percent at 20x). A few hundred test items cannot show a 2-point difference: the paired test needed about 3,651. Prices, hardware and the correctness model are assumed; no real model was served.

What you must be able to say (one page)

• "Open" is a spectrum. A common distinction separates open weights (you can download the model) from open models (the training data is public too). "Open source" is often used loosely for any public-weight model, though most are open-weight only. Licences differ model by model; ask: does it allow commercial use, are there restrictions (the Llama licences require a special licence above 700 million monthly active users), and may you use its outputs to train other models (Llama 2 forbade it; later Llama versions relaxed this with conditions, which shows why you must read the current text). I did not check any current licence text.
• Seven axes for self-host versus API: data privacy, data lineage, performance, functionality, cost, control, on-device. In short: self-hosting keeps data in-house, gives logprobs and the ability to fine-tune and freeze a version; APIs give the strongest models, scaling, function calling and structured outputs out of the box. Providers tend to keep their best models behind APIs, so open models often lag the frontier, while for many uses they suffice.
• Will it fit, and how fast? Weights = parameters x bytes (16-bit 2, 8-bit 1, 4-bit 0.5). With 20 percent headroom: a 7B model needs 14 GB in 16-bit (fits a 32 GB laptop or a 24 GB GPU, not an 8 GB phone) and 3.5 GB in 4-bit (fits the phone); 13B in 16-bit (26 GB) fits only the 80 GB GPU; 70B in 4-bit (35 GB) fits the 80 GB GPU, in 16-bit (140 GB) fits none of my four devices. Decode speed is bounded by bandwidth / weight bytes: on a 1 TB/s GPU (assumed) a 7B model gives at most 71 tokens/s in 16-bit and 286 in 4-bit at batch 1; 13B 4-bit gives 154. These are upper bounds that ignore the KV cache and overhead (module 6).
• Self-host economics are about utilisation. My model (8B 16-bit, $2.50 per GPU-hour assumed, about 2 TB/s for an A100-80GB): batch 1 gives 117 tokens/s, batch 8 651, batch 32 1,271. Cost per million tokens at batch 32: $0.55 at full load, $0.91 at 60 percent, $1.82 at 30 percent, $5.46 at 10 percent; at batch 1 it is $5.93 at full load and $59 at 10 percent. Against an assumed $6 per million API price, an idle GPU loses; a busy, batched one wins by a wide margin. API cost per token barely changes with scale, while self-hosted cost falls as you fill a fixed cluster.
• Break-even needs volume and engineering time. Monthly fixed cost = GPU hours + the people. GPU only ($1,825): break-even at 304 M tokens a month (9 percent of one GPU's 3,340 M capacity at batch 32). Add $7,500 of engineering (assumed): 1,554 M tokens, 47 percent utilisation. Add $15,000: 2,804 M, 84 percent. Engineering time can cost more than the API bill it saves.
• Routing and cascades (routers and cascades are common ways to mix models). My simulation: small model cost 1, large cost 20 (assumed), correctness = sigmoid(10 x (skill - difficulty)); small skill 0.4 gets 37.3 percent, large (0.8) gets 84.3 percent. A random mix only moves along the straight line between them (25 percent large: 48.9 percent at cost 5.74). A cascade (ask small, escalate when a confidence signal is low) with an informative signal beats the line: the cost to come within 2 points of the large model is 16.0 with a signal of strength 2 and 13.1 with strength 4, but 20.2 (worse than always using the large model) with no signal, because you paid for the small call too.
• The savings depend on how good the small model is. Cost to match the large model within 2 points with a cascade of signal strength 2: 16.0 when the small model scores 37 percent, 11.3 at 57 percent, 4.7 at 74 percent. A difficulty-predicting router with noise 0.05 got 17.0, 14.6 and 9.1; with noise 0.2 it got 18.2 (small skill 0.4); with noise 0.5, 18.9. Router quality is everything. A verifier that costs 1 or 3 per call raised the cascade cost from 16.0 to 17.0 and 19.0.
• The price ratio sets the prize. With the large model at 5x, 20x and 50x the small one, the cascade cost to match it was 4.8, 16.0 and 38.6: a saving of about 5, 20 and 23 percent. Routers should be small and cheap, for example an intent classifier built on a small language model or an even smaller classifier.
• Switching models is a statistical decision. To detect a 2-point accuracy gain between two models on the same items, my paired test needed about 3,651 items for 80 percent power; an independent-samples test would need 6,187 per model. In simulation power at 3,000 items was 72 percent (paired) and 48 percent (independent test applied to the shared items, which is conservative). A gap from 70 to 80 percent needs about 295 per model; from 70 to 73 percent 3,555. A few hundred items cannot show a small difference, either way.
• Why teams pick open models (an a16z enterprise survey, from memory): control and customisability; and you can freeze a version, whereas commercial models change, sometimes silently.
• What to say in an interview: "I decide on seven axes: privacy, lineage, quality, features, cost, control and where it runs. I size by weights = parameters x bytes plus the KV cache, estimate decode speed from bandwidth, and compute cost per token at the utilisation I will actually achieve, including engineering time. I route or cascade only after measuring the small model on my own traffic, and I decide a switch with enough items to see the gap."
• Honest limits: every price, GPU rental rate, bandwidth other than the published A100 figure, and the correctness model are assumptions; no real small model was served or evaluated; the router and confidence signal are simulated, not trained.