CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Measure TTFT and TPOT separately, and name which one your users feel.
  • Size the cache first: batch x sequence x layers x hidden x 2 x bytes decides how many requests fit.
  • Use continuous batching for variable-length outputs; static batching wastes slots on the longest reply.
  • Quantise one thing at a time and compare quality on your own evaluation set, not only perplexity.
  • Check acceptance rate on your real traffic before adopting speculative decoding; pick K from it.
  • Put stable content first in prompts so prefix caching can reuse it.
  • Report hardware, batch, sequence length and precision with every latency number.
  • Treat vendor speed-ups as upper bounds until you reproduce them on your workload.