Part 7 of 12 · 1 min read
How to build it so it holds up
- Measure TTFT and TPOT separately, and name which one your users feel.
- Size the cache first: batch x sequence x layers x hidden x 2 x bytes decides how many requests fit.
- Use continuous batching for variable-length outputs; static batching wastes slots on the longest reply.
- Quantise one thing at a time and compare quality on your own evaluation set, not only perplexity.
- Check acceptance rate on your real traffic before adopting speculative decoding; pick K from it.
- Put stable content first in prompts so prefix caching can reuse it.
- Report hardware, batch, sequence length and precision with every latency number.
- Treat vendor speed-ups as upper bounds until you reproduce them on your workload.