CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 3 of 12 · 2 min read

Decode is memory-bound, so the cache, not the weights, sets how many requests fit

Decoding spends most of its time moving bytes from memory, not doing arithmetic, so adding more compute barely helps while adding more memory traffic hurts a lot.

Think of a librarian who has to walk to the shelf and read the spine of every single book on a very long shelf just to answer one quick question. The bottleneck is not the thinking. It is the walking and reading.

In plain words: a decode step reads the entire model's weights and the entire KV cache from memory for comparatively little arithmetic, so decode time is set mostly by how many bytes have to move, not by how much computing power is available — which is why batching helps throughput without proportionally raising per-step time, up to a point.

A worked example. At a sequence length of 2,048 tokens, going from batch 1 to batch 32 — 32 times the arithmetic work — only takes 2.9 times as long (13.8 ms to 39.8 ms), because compute is under 10 percent of step time at every feasible batch size; the memory traffic from weights plus a growing cache dominates throughout. Throughput rises from 72 to 803 tokens per second across that same range — an 11x gain for 32x the batch, sublinear because the cache itself grows with batch size and adds more bytes to move each step.

The key lines. From code/mini-inference/memory.py:

# from code/mini-inference/memory.py
def decode_step_seconds(batch, params, seq, layers, d_model, bytes_per_param=2, bytes_per_value=2, kv_fraction=1.0, peak=PEAK_FLOPS, bw=BANDWIDTH):
    """One decoding step for the whole batch: the larger of compute time and memory time (weights read once, every sequence's KV cache read)."""
    compute = batch * 2 * params / peak
    memory = (weight_bytes(params, bytes_per_param) + kv_cache_bytes(batch, seq, layers, d_model, bytes_per_value, kv_fraction)) / bw
    return max(compute, memory), compute, memory

Read it as: step time is the maximum of the compute and memory estimates, not their sum. Whichever one is slower sets the pace. For ordinary serving batches, the code above shows memory winning that comparison almost every time.

TTFT and TPOT, memory-bound and compute-bound

TTFT is time to first token; TPOT is time per output token after the first. Memory-bound means step time is set by bytes moved from memory; compute-bound means it's set by arithmetic — the ridge point is where the two would take equal time.

Example: the compute-vs-memory comparison above is exactly how to tell which bottleneck you're facing on a specific workload — TPOT (decode time) tracks the memory estimate here, while TTFT (prefill time) tends to track the compute estimate instead, because prefill processes many tokens in parallel.