Definitions

Definitions

The pipeline

Results

1. The cache trades memory for compute, and at scale memory is the scarcer resource.
2. Decode is memory-bound, so reducing bytes (quantisation, fewer KV heads, shorter context) helps more than adding FLOPs.
3. Batching raises throughput sublinearly because the cache grows with the batch.
4. Continuous batching mostly removes waiting, not compute: latency stayed near 60 steps median while static batching collapsed.
5. Speculative decoding pays only when acceptance is high and the draft is cheap.
6. Quantisation sensitivity must be measured; it differs between weights and cache and between models.

Design rules

• Measure TTFT and TPOT separately, and name which one your users feel.
• Size the cache first: batch x sequence x layers x hidden x 2 x bytes decides how many requests fit.
• Use continuous batching for variable-length outputs; static batching wastes slots on the longest reply.
• Quantise one thing at a time and compare quality on your own evaluation set, not only perplexity.
• Check acceptance rate on your real traffic before adopting speculative decoding; pick K from it.
• Put stable content first in prompts so prefix caching can reuse it.
• Report hardware, batch, sequence length and precision with every latency number.
• Treat vendor speed-ups as upper bounds until you reproduce them on your workload.

Interview questions, with the follow-ups

Common mistakes

• Quoting one latency number without batch, sequence length and hardware.
• Ignoring the cache when sizing a GPU.
• Using static batching for variable-length generation.
• Assuming quantisation is free or uniformly harmful.
• Choosing K without measuring acceptance rate.
• Putting changing text at the start of a prompt, defeating prefix caching.
• Optimising TPOT when TTFT is the complaint.
• Trusting a vendor's speed-up on a different workload.

What the example does not do

• No GPU or kernels: time comes from a two-term formula (bytes over bandwidth, FLOPs over peak); real kernels, launch overhead and communication change it.
• Assumed peak: 312 TFLOP/s is my assumption for an A100-class GPU; the 2 TB/s figure is the published A100-80GB bandwidth, rounded.
• The batching simulation uses one step as the time unit, no prefill cost and no preemption.
• The quantisation toy uses random weights, so its tolerance to bits is not evidence about trained models; the cache vs weights ordering may differ.
• No paged attention, FlashAttention, tensor parallelism, or disaggregated prefill.
• Speculative decoding is tested on a synthetic draft with controlled disagreement, not a real small model.
• Optional real experiment (I did not run this): serve a small open model with vLLM or llama.cpp, measure TTFT, TPOT and throughput at batch 1, 8 and 32, then repeat with a quantised build and prefix caching on.

For the curious Batching policy changes latency more than speed

Static batching holds every reply until the longest in its batch finishes; continuous batching refills a slot the moment a reply ends. At 0.04 requests per step the p95 was 1,547 steps for static and 211 for continuous, and at 0.07 static was overloaded. Speculative decoding keeps the target model's output distribution exactly, and pays off only when the draft's acceptance rate is high (about 0.8 or more for a 2x gain at c = 0.128). The hardware figures are assumed round numbers and the model is tiny.

What you must be able to say (one page)

• Two phases, two bottlenecks. Prefill (processing the input tokens) is compute-bound and decode (generating output tokens one by one) is memory-bandwidth-bound. The two latency metrics follow: TTFT (time to first token, dominated by prefill) and TPOT (time per output token, dominated by decode).
• The KV cache removes repeated work. Without it, each new token recomputes keys and values for the whole prefix. In my NumPy transformer, cached and uncached generation gave identical tokens and a maximum logit difference near 7e-15; positions processed to generate N tokens after a 5-token prompt were 25 vs 290 (N=20), 45 vs 980 (N=40), 85 vs 3,560 (N=80) and 165 vs 13,520 (N=160): linear against quadratic. Wall time at N=160 was 0.022 s vs 0.310 s on my machine (machine-specific).
• The cache costs memory. Size = 2 x batch x sequence x layers x hidden x bytes. For Llama 2 13B (batch 32, sequence 2,048, 16-bit) the cache is 53.69 GB, more than the 26 GB of weights. Per sequence: 0.42, 1.68 and 6.71 GB at 512, 2,048 and 8,192 tokens.
• So the cache sets the batch size. On an 80 GB GPU with 13B 16-bit weights, my arithmetic gives a largest batch of 128, 32 and 8 at 512, 2,048 and 8,192 tokens. An 8-bit cache doubles that (257, 64, 16); 4-bit weights help much less (175, 43, 10) because weights are a fixed cost and the cache scales with batch.
• Decode is memory-bound, so batching is nearly free until it is not. My roofline-style model (312 TFLOP/s assumed for an A100-class GPU, about 2 TB/s for an A100-80GB) at sequence 2,048: batch 1 takes 13.8 ms (72 tokens/s), batch 32 takes 39.8 ms (803 tokens/s): 32 times the work for 2.9 times the time, but only 11 times the throughput, because the growing cache adds bytes to read. Compute is under 10 percent of the step time at every feasible batch size. Ignoring the cache, the weights alone would turn compute-bound near a batch of 156.
• Batching policy changes latency more than speed. Slot limit 8, mean output 78 tokens, mean longest-of-8 203 tokens. Static batching holds every reply until the longest in its batch finishes and saturates near 0.039 requests per step; continuous batching (the idea behind Orca) refills a slot the moment a reply ends and saturates near 0.10. At 0.04 requests per step: static p50 791, p95 1,547; continuous p50 58, p95 211. At 0.07 static is overloaded (p50 7,811) while continuous holds 64.
• Speculative decoding: a cheap draft model proposes K tokens; the target model checks them in one pass and keeps the longest agreeing prefix. With the accept/reject rule the output distribution equals the target's: in my test the total-variation distance to the target was 0.006 and 0.005 while the draft itself was 0.15 and 0.47 away. Expected tokens per verification = (1 - a^(K+1)) / (1 - a) for acceptance rate a; speed-up = that divided by (K c + 1) where c is draft cost relative to a target step.
• How much speed-up? With c = 0.128 (a 70B Chinchilla target at 14.1 ms against a 4B draft at 1.8 ms, from the speculative sampling paper as I recall it): a=0.5 gives 1.39x (K=2); 0.7 gives 1.83x (K=4); 0.8 gives 2.25x (K=5); 0.9 gives 3.03x (K=9). In my model, cutting latency by more than half needs roughly an acceptance rate of at least 0.8.
• Quantisation (tiny model, KL from the 16-bit model and top-1 agreement): weights at 8, 6, 4, 3, 2 bits gave KL 0.00007, 0.0014, 0.027, 0.113, 0.847 and agreement 0.997, 0.975, 0.931, 0.875, 0.705; the KV cache at the same bit widths gave KL 0.00001, 0.0001, 0.0019, 0.011, 0.091 and agreement 0.998, 0.997, 0.982, 0.959, 0.884. In this toy the cache tolerated fewer bits than the weights; this is a toy, so measure on your model.
• Prompt caching: reusing a processed prefix. My arithmetic: a 1,000-token prompt at 1M calls a day is 1B tokens, $3,000 a day at an assumed $3 per million, or $300 with a 90 percent discount (Anthropic announced up to 90 percent lower cost for cached input when it launched prompt caching; check current terms before you plan on any discount).
• What to say in an interview: "Prefill is compute-bound, decode is memory-bound. I use a KV cache, size it to set the batch, use continuous batching and quantise weights or cache after measuring quality, add speculative decoding when the draft's acceptance rate is high enough to pay for its cost, and cache stable prompt prefixes."
• Honest limits: a tiny NumPy model, an arithmetic hardware model with assumed peak FLOPs, a toy request simulator; no GPU kernels, no paged attention, no FlashAttention, no real serving stack.