Part 10 of 12 · 1 min read
Quick notes
- Two phases, two bottlenecks: prefill is compute-bound and sets time to first token; decode is memory-bandwidth-bound and sets time per output token.
- The KV cache turns quadratic work into linear work, and it costs memory: 2 x batch x sequence x layers x hidden x bytes, which for a 13B model at a decent batch size exceeds the weights. The cache sets the batch size.
- Decode is memory-bound, so batching is nearly free until the cache catches up, and continuous batching cuts latency far more than static batching at the same load.
- Speculative decoding keeps the target's output distribution and pays off when the draft's acceptance rate is high enough to cover its cost.
- Quantise weights or cache only after measuring quality on your model, and cache stable prompt prefixes; check the provider's current discount before planning on it.