Part 10 of 12 · 1 min read
Quick notes
- "Open" is a spectrum: open weights is not open data, and licences differ on commercial use, size limits and whether outputs may train other models. Read the current text.
- Seven axes for self-host versus API: data privacy, lineage, performance, functionality, cost, control and on-device. APIs give the strongest models and features; self-hosting gives control and a frozen version.
- Will it fit, and how fast? Weights are parameters times bytes per weight, plus headroom and the KV cache; decode speed at batch 1 is bounded by bandwidth over weight bytes.
- Self-host economics are about utilisation. A busy, batched GPU beats the API price by a wide margin; an idle one loses, and engineering time can cost more than the API bill it saves.
- Cascades and routers save money only with a good signal and a capable small model, and switching models is a statistical decision that needs thousands of paired items to show a small gap.