CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • Two phases, two bottlenecks: prefill is compute-bound and sets time to first token; decode is memory-bandwidth-bound and sets time per output token.
  • The KV cache turns quadratic work into linear work, and it costs memory: 2 x batch x sequence x layers x hidden x bytes, which for a 13B model at a decent batch size exceeds the weights. The cache sets the batch size.
  • Decode is memory-bound, so batching is nearly free until the cache catches up, and continuous batching cuts latency far more than static batching at the same load.
  • Speculative decoding keeps the target's output distribution and pays off when the draft's acceptance rate is high enough to cover its cost.
  • Quantise weights or cache only after measuring quality on your model, and cache stable prompt prefixes; check the provider's current discount before planning on it.