Part 9 of 12 · 2 min read
Mistakes that misread a latency or throughput number
Common mistakes, and what the example does not do.
Mistakes that misread a latency or throughput number
- Quoting one latency number without batch, sequence length and hardware.
- Ignoring the cache when sizing a GPU.
- Using static batching for variable-length generation.
- Assuming quantisation is free or uniformly harmful.
- Choosing K without measuring acceptance rate.
- Putting changing text at the start of a prompt, defeating prefix caching.
- Optimising TPOT when TTFT is the complaint.
- Trusting a vendor's speed-up on a different workload.
What the example does not do
About this example What the example does not do
- No GPU or kernels: time comes from a two-term formula (bytes over bandwidth, FLOPs over peak); real kernels, launch overhead and communication change it.
- Assumed peak: 312 TFLOP/s is the assumption for an A100-class GPU; the 2 TB/s figure is the published A100-80GB bandwidth, rounded.
- The batching simulation uses one step as the time unit, no prefill cost and no preemption.
- The quantisation toy uses random weights, so its tolerance to bits is not evidence about trained models; the cache vs weights ordering may differ.
- No paged attention, FlashAttention, tensor parallelism, or disaggregated prefill.
- Speculative decoding is tested on a synthetic draft with controlled disagreement, not a real small model.
- Optional real experiment: serve a small open model with vLLM or llama.cpp, measure TTFT, TPOT and throughput at batch 1, 8 and 32, then repeat with a quantised build and prefix caching on.
Check yourself
Why does the KV cache, not the weights, limit how many requests one GPU can serve?
Show answer
Weights are a fixed cost, while the cache grows with batch size and sequence length. For Llama 2 13B at batch 32 and 2,048 tokens the cache is 53.69 GB, twice the 26 GB of weights.
For the curious Batching policy changes latency more than speed
Static batching holds every reply until the longest in its batch finishes; continuous batching refills a slot the moment a reply ends. At 0.04 requests per step the p95 was 1,547 steps for static and 211 for continuous, and at 0.07 static was overloaded. Speculative decoding keeps the target model's output distribution exactly, and pays off only when the draft's acceptance rate is high (about 0.8 or more for a 2x gain at c = 0.128). The hardware figures are assumed round numbers and the model is tiny.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Cache stable prompt prefixes | System prompt, tool list, long documents. |
| Size the batch from cache memory | Weights fixed, cache per sequence. |
| Quantise weights or cache only after measuring quality | Compare on your own eval. |
| Watch TTFT and TPOT separately | Prefill and decode have different fixes. |