CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 8 of 12 · 2 min read

Answering inference-optimisation questions in an interview

Why is decoding memory-bound and prefill compute-bound?

"Prefill processes many tokens in one pass, so the weights read from memory are reused across all of them and arithmetic dominates. Decode produces one token per sequence per step, so every step reads all the weights and the cache for little arithmetic. In the example's model compute was under 10 percent of the step time at every feasible batch." Follow-up: "What would make decode compute-bound?" A large batch; without the cache the ridge in the example's model was near batch 156, but the cache grows with the batch and memory fills first.

How does the KV cache work and what does it cost?

"It stores keys and values of past tokens so each step computes only the new one: linear instead of quadratic work, with identical outputs (mine matched to 1e-14). The cost is 2 x batch x sequence x layers x hidden x bytes; the Llama 2 13B example is 54 GB at batch 32 and 2,048 tokens." Follow-up: "How do you reduce it?" Quantise it, use fewer key-value heads, use paged allocation, or shorten context.

Static versus continuous batching?

"Static waits for the slowest request in a batch; with variable output lengths most slots idle. Continuous batching refills a slot as soon as a request ends. In the example's simulation static saturated near 0.039 requests per step and continuous near 0.10, and at 0.07 static latency exploded while continuous stayed near 64 steps median." Follow-up: "What do you give up?" Scheduler complexity and per-step overhead, which the example's simulation ignores.

When does speculative decoding help?

"When the draft is much cheaper than the target and agrees often. Expected tokens per verification is (1 - a^(K+1)) / (1 - a) and the cost is K draft steps plus one target step. At draft cost 0.128 I needed acceptance near 0.8 for a 2x gain. The rejection rule keeps the target distribution, so quality is unchanged." Follow-up: "Does it help at large batch?" Less: a compute-bound server has no spare compute to spend on verifying extra tokens (my model does not cover this).