CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 6 of 12 · 5 min read

What the example found

Memory and speed by precision, device fit, utilisation-driven cost, break-even volume, routing strategies, cascade signal strength, and detection power.

How to read the memory-and-speed table: each row is a model size; the three precision columns show the same model's memory footprint shrinking; the last column is a decode-speed upper bound at a fixed 4-bit precision and 1 TB/s bandwidth.

Model 16-bit 8-bit 4-bit Batch-1 decode bound at 1 TB/s, 4-bit (the 70B row does not fit that GPU)
3B 6 GB 3 GB 1.5 GB 667 tokens/s
7B 14 GB 7 GB 3.5 GB 286 tokens/s
13B 26 GB 13 GB 6.5 GB 154 tokens/s
34B 68 GB 34 GB 17 GB 59 tokens/s
70B 140 GB 70 GB 35 GB 29 tokens/s

What to notice: memory and decode-speed move in opposite directions down the table. Bigger models need more memory. By this formula they also run slower per-token, even at the same bandwidth, since there is simply more weight data to read through every step.

How to read the device-fit table: each row is a model size; each column is a precision level; the cell lists which devices (from a fixed list of four) the model actually fits on with 20 percent headroom.

Where it fits (20% headroom) 16-bit 8-bit 4-bit
7B laptop, 24 GB GPU, 80 GB GPU laptop, 24 GB GPU, 80 GB GPU also the 8 GB phone
13B 80 GB GPU only laptop, 24 GB GPU, 80 GB GPU laptop, 24 GB GPU, 80 GB GPU
70B none none 80 GB GPU only

What to notice: 13B at 16-bit fits only the largest GPU, but the identical model at 8-bit fits three device classes. A single quantisation step can be the entire difference between "needs a data-center GPU" and "runs on a laptop."

How to read the utilisation-cost table: each row is a different batch size; the four right-hand columns show cost per million tokens at different utilisation levels, all for the same model and hardware.

8B model, sequence 2,048 Tokens/s $ per M tokens at 10% at 30% at 60% at 100%
Batch 1 117 59.28 19.76 9.88 5.93
Batch 8 651 10.67 3.56 1.78 1.07
Batch 32 1,271 5.46 1.82 0.91 0.55

What to notice: the cheapest cell (batch 32, 100 percent utilisation, $0.55) and the most expensive (batch 1, 10 percent utilisation, $59.28) describe the exact same hardware and model. That is a 100x difference, entirely from batching and load, not anything about the model's quality.

How to read the break-even table: each row adds a different amount of engineering cost to the same hardware; break-even volume and the utilisation it implies both rise together.

Engineering cost per month Fixed cost per month Break-even tokens per month Utilisation needed (batch 32)
$0 $1,825 304 M 9%
$7,500 $9,325 1,554 M 47%
$15,000 $16,825 2,804 M 84%

What to notice: adding $15,000 of engineering cost roughly quintuples the fixed monthly cost but increases break-even volume by more than 9x. Fixed costs and the volume needed to cover them don't scale together one-to-one. So a seemingly modest staffing addition can require nearly saturating a GPU's total capacity.

How to read the routing-strategy table: each row is a different way of mixing small and large model calls; cost 1 is "always small," cost 20 is "always large," and the random-mix rows show the straight line between them.

Strategy (small cost 1, large cost 20) Accuracy Cost
Always small 37.3% 1
Always large 84.3% 20
Random, 25% large 48.9% 5.74
Random, 50% large 60.9% 10.5
Random, 75% large 72.4% 15.2

What to notice: the random-mix rows trade accuracy for cost in direct proportion. There is no shortcut here. This is exactly the baseline that a genuinely informative cascade or router needs to beat to justify its own complexity.

How to read the cascade-signal table: each row is a different confidence-signal strength (or router noise level); "cost" is what it takes to come within 2 points of the large model's accuracy.

Cost to come within 2 points of the large model Cost Saving vs always large
Cascade, no confidence signal 20.2 none (worse)
Cascade, signal strength 1 18.8 6%
Cascade, signal strength 2 16.0 20%
Cascade, signal strength 4 13.1 34%
Difficulty router, noise 0.05 / 0.2 / 0.5 17.0 / 18.2 / 18.9 15% / 9% / 6%

What to notice: the savings scale directly with signal quality, from negative (no signal) to 34 percent (strong signal). There is no fixed "routing saves X percent" number. It is entirely a function of how good the signal actually is on your task.

How to read the small-model-skill table: each row is a different small-model accuracy; both columns show the cost to match the large model within 2 points, using either a cascade or a router.

Small model skill (accuracy) Cascade (signal 2) cost Router (noise 0.05) cost
0.40 (37%) 16.0 17.0
0.55 (57%) 11.3 14.6
0.70 (74%) 4.7 9.1

What to notice: as the small model improves from 37 to 74 percent accuracy, the cascade's cost drops by more than threefold (16.0 to 4.7). Routing pays off far more when the small model is already decent. It pays off less when the small model is weak and the large model has to be called constantly anyway.

How to read the price-ratio table: each row scales how much more expensive the large model is relative to the small one; "always large" scales directly, "cascade" does not.

Large model price ratio Always large Cascade (signal 2)
5x 5 4.8
20x 20 16.0
50x 50 38.6

What to notice: the cascade's relative saving grows as the price ratio widens (about 4 percent at 5x, versus about 23 percent at 50x). Routing matters more, not less, the bigger the price gap between the two models.

How to read the power table: each row is a different evaluation set size; the two columns show statistical power for a paired versus an independent test, both aimed at the same 2-point accuracy gap.

Items Power, paired test Power, independent test on the same items
300 13% 4%
1,000 31% 13%
3,000 72% 48%
3,651 79% 59%
6,187 95% 86%

What to notice: at every single item count, the paired test has substantially higher power than the independent test on identical data. Pairing is free statistical leverage you get just from comparing both models on the same items. Ignoring that leverage effectively throws away evidence you already collected.

{{svg:sizing|What fits, how fast, and what it costs (assumed hardware and prices).}} {{svg:routing|Routing frontier and the number of evaluation items a model switch needs.}}