Part 6 of 12 · 5 min read
What the example found
Memory and speed by precision, device fit, utilisation-driven cost, break-even volume, routing strategies, cascade signal strength, and detection power.
How to read the memory-and-speed table: each row is a model size; the three precision columns show the same model's memory footprint shrinking; the last column is a decode-speed upper bound at a fixed 4-bit precision and 1 TB/s bandwidth.
| Model | 16-bit | 8-bit | 4-bit | Batch-1 decode bound at 1 TB/s, 4-bit (the 70B row does not fit that GPU) |
|---|---|---|---|---|
| 3B | 6 GB | 3 GB | 1.5 GB | 667 tokens/s |
| 7B | 14 GB | 7 GB | 3.5 GB | 286 tokens/s |
| 13B | 26 GB | 13 GB | 6.5 GB | 154 tokens/s |
| 34B | 68 GB | 34 GB | 17 GB | 59 tokens/s |
| 70B | 140 GB | 70 GB | 35 GB | 29 tokens/s |
What to notice: memory and decode-speed move in opposite directions down the table. Bigger models need more memory. By this formula they also run slower per-token, even at the same bandwidth, since there is simply more weight data to read through every step.
How to read the device-fit table: each row is a model size; each column is a precision level; the cell lists which devices (from a fixed list of four) the model actually fits on with 20 percent headroom.
| Where it fits (20% headroom) | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 7B | laptop, 24 GB GPU, 80 GB GPU | laptop, 24 GB GPU, 80 GB GPU | also the 8 GB phone |
| 13B | 80 GB GPU only | laptop, 24 GB GPU, 80 GB GPU | laptop, 24 GB GPU, 80 GB GPU |
| 70B | none | none | 80 GB GPU only |
What to notice: 13B at 16-bit fits only the largest GPU, but the identical model at 8-bit fits three device classes. A single quantisation step can be the entire difference between "needs a data-center GPU" and "runs on a laptop."
How to read the utilisation-cost table: each row is a different batch size; the four right-hand columns show cost per million tokens at different utilisation levels, all for the same model and hardware.
| 8B model, sequence 2,048 | Tokens/s | $ per M tokens at 10% | at 30% | at 60% | at 100% |
|---|---|---|---|---|---|
| Batch 1 | 117 | 59.28 | 19.76 | 9.88 | 5.93 |
| Batch 8 | 651 | 10.67 | 3.56 | 1.78 | 1.07 |
| Batch 32 | 1,271 | 5.46 | 1.82 | 0.91 | 0.55 |
What to notice: the cheapest cell (batch 32, 100 percent utilisation, $0.55) and the most expensive (batch 1, 10 percent utilisation, $59.28) describe the exact same hardware and model. That is a 100x difference, entirely from batching and load, not anything about the model's quality.
How to read the break-even table: each row adds a different amount of engineering cost to the same hardware; break-even volume and the utilisation it implies both rise together.
| Engineering cost per month | Fixed cost per month | Break-even tokens per month | Utilisation needed (batch 32) |
|---|---|---|---|
| $0 | $1,825 | 304 M | 9% |
| $7,500 | $9,325 | 1,554 M | 47% |
| $15,000 | $16,825 | 2,804 M | 84% |
What to notice: adding $15,000 of engineering cost roughly quintuples the fixed monthly cost but increases break-even volume by more than 9x. Fixed costs and the volume needed to cover them don't scale together one-to-one. So a seemingly modest staffing addition can require nearly saturating a GPU's total capacity.
How to read the routing-strategy table: each row is a different way of mixing small and large model calls; cost 1 is "always small," cost 20 is "always large," and the random-mix rows show the straight line between them.
| Strategy (small cost 1, large cost 20) | Accuracy | Cost |
|---|---|---|
| Always small | 37.3% | 1 |
| Always large | 84.3% | 20 |
| Random, 25% large | 48.9% | 5.74 |
| Random, 50% large | 60.9% | 10.5 |
| Random, 75% large | 72.4% | 15.2 |
What to notice: the random-mix rows trade accuracy for cost in direct proportion. There is no shortcut here. This is exactly the baseline that a genuinely informative cascade or router needs to beat to justify its own complexity.
How to read the cascade-signal table: each row is a different confidence-signal strength (or router noise level); "cost" is what it takes to come within 2 points of the large model's accuracy.
| Cost to come within 2 points of the large model | Cost | Saving vs always large |
|---|---|---|
| Cascade, no confidence signal | 20.2 | none (worse) |
| Cascade, signal strength 1 | 18.8 | 6% |
| Cascade, signal strength 2 | 16.0 | 20% |
| Cascade, signal strength 4 | 13.1 | 34% |
| Difficulty router, noise 0.05 / 0.2 / 0.5 | 17.0 / 18.2 / 18.9 | 15% / 9% / 6% |
What to notice: the savings scale directly with signal quality, from negative (no signal) to 34 percent (strong signal). There is no fixed "routing saves X percent" number. It is entirely a function of how good the signal actually is on your task.
How to read the small-model-skill table: each row is a different small-model accuracy; both columns show the cost to match the large model within 2 points, using either a cascade or a router.
| Small model skill (accuracy) | Cascade (signal 2) cost | Router (noise 0.05) cost |
|---|---|---|
| 0.40 (37%) | 16.0 | 17.0 |
| 0.55 (57%) | 11.3 | 14.6 |
| 0.70 (74%) | 4.7 | 9.1 |
What to notice: as the small model improves from 37 to 74 percent accuracy, the cascade's cost drops by more than threefold (16.0 to 4.7). Routing pays off far more when the small model is already decent. It pays off less when the small model is weak and the large model has to be called constantly anyway.
How to read the price-ratio table: each row scales how much more expensive the large model is relative to the small one; "always large" scales directly, "cascade" does not.
| Large model price ratio | Always large | Cascade (signal 2) |
|---|---|---|
| 5x | 5 | 4.8 |
| 20x | 20 | 16.0 |
| 50x | 50 | 38.6 |
What to notice: the cascade's relative saving grows as the price ratio widens (about 4 percent at 5x, versus about 23 percent at 50x). Routing matters more, not less, the bigger the price gap between the two models.
How to read the power table: each row is a different evaluation set size; the two columns show statistical power for a paired versus an independent test, both aimed at the same 2-point accuracy gap.
| Items | Power, paired test | Power, independent test on the same items |
|---|---|---|
| 300 | 13% | 4% |
| 1,000 | 31% | 13% |
| 3,000 | 72% | 48% |
| 3,651 | 79% | 59% |
| 6,187 | 95% | 86% |
What to notice: at every single item count, the paired test has substantially higher power than the independent test on identical data. Pairing is free statistical leverage you get just from comparing both models on the same items. Ignoring that leverage effectively throws away evidence you already collected.
{{svg:sizing|What fits, how fast, and what it costs (assumed hardware and prices).}} {{svg:routing|Routing frontier and the number of evaluation items a model switch needs.}}