Part 6 of 12 · 4 min read
What the example found
Cache-vs-no-cache position counts, cache sizing by bit-width, batch throughput, batching-policy latency, speculative speed-ups, and quantisation sensitivity.
How to read the cache-vs-no-cache table: both middle columns face the same generation task (a 5-token prompt, N tokens produced); only whether the cache is used differs.
| Tokens generated (5-token prompt) | Positions, no cache | Positions, cache | Max logit difference |
|---|---|---|---|
| 20 | 290 | 25 | 4e-15 |
| 40 | 980 | 45 | 5e-15 |
| 80 | 3,560 | 85 | 7e-15 |
| 160 | 13,520 | 165 | 7e-15 |
What to notice: the gap between the two position-count columns grows faster than the token count itself. Doubling tokens generated roughly quadruples the no-cache cost but only doubles the cached cost. This is the quadratic-vs-linear difference made concrete.
How to read the cache-sizing table: each row is a sequence length; each column is a different way of shrinking memory use (lower bit-width cache, lower bit-width weights, or structurally fewer KV heads); the number shown is the maximum batch size that fits.
| Sequence length | 16-bit cache | 8-bit cache | 4-bit weights | 1/8 as many KV heads (hypothetical) |
|---|---|---|---|---|
| 512 | 128 | 257 | 175 | 1,029 |
| 2,048 | 32 | 64 | 43 | 257 |
| 8,192 | 8 | 16 | 10 | 64 |
What to notice: halving the cache's bit-width roughly doubles the max batch at every sequence length. Shrinking the weights helps less. Weights are a fixed cost paid once regardless of batch, while the cache scales directly with batch. So shrinking the cache attacks the part of memory that actually limits how many requests fit.
How to read the batch-throughput table: each row is a larger batch at the same 2,048-token sequence length; compute and memory time are shown separately so you can see which one dominates.
| Batch (sequence 2,048) | Compute ms | Memory ms | Tokens per second |
|---|---|---|---|
| 1 | 0.08 | 13.8 | 72 |
| 4 | 0.33 | 16.4 | 245 |
| 8 | 0.67 | 19.7 | 406 |
| 16 | 1.33 | 26.4 | 606 |
| 32 | 2.67 | 39.8 | 803 |
What to notice: at every single batch size, compute time stays well under 10 percent of memory time. This workload never gets close to being compute-bound within any realistic batch size. That is exactly why reducing memory traffic (quantisation, fewer KV heads, shorter context) is the lever that actually moves decode speed.
How to read the batching-policy table: each row is a different request arrival rate; all three policies face the identical arrival stream, and utilisation shows how much of the available slot-time was actually doing work.
| Requests per step | Static p50 / p95 | Dynamic p50 / p95 | Continuous p50 / p95 | Utilisation (s / d / c) |
|---|---|---|---|---|
| 0.01 | 483 / 1,134 | 139 / 467 | 58 / 204 | 0.38 / 0.16 / 0.18 |
| 0.02 | 370 / 735 | 199 / 560 | 58 / 202 | 0.39 / 0.22 / 0.24 |
| 0.03 | 369 / 715 | 299 / 726 | 59 / 206 | 0.39 / 0.29 / 0.32 |
| 0.04 | 791 / 1,547 | 805 / 1,567 | 58 / 211 | 0.39 / 0.37 / 0.41 |
| 0.07 | 7,811 / 14,559 | 8,022 / 14,685 | 64 / 216 | 0.39 / 0.39 / 0.69 |
What to notice: continuous batching's p50/p95 barely move across the entire table (58-64 / 202-216), while static and dynamic both degrade sharply past 0.04. Continuous batching's advantage isn't modest. It is the difference between a system that scales and one with a hard wall.
How to read the speculative-decoding table: the top two rows show the best K (number of drafted tokens) and resulting speedup at two draft costs; the bottom row shows raw tokens produced per verification pass at a fixed K.
| Draft cost c | a = 0.5 | a = 0.7 | a = 0.8 | a = 0.9 |
|---|---|---|---|---|
| 0.05 (best K, speed-up) | 3, 1.63x | 6, 2.35x | 8, 3.09x | 13, 4.67x |
| 0.128 (best K, speed-up) | 2, 1.39x | 4, 1.83x | 5, 2.25x | 9, 3.03x |
| Tokens per verification at K=4 | 1.94 | 2.77 | 3.36 | 4.10 |
What to notice: a cheaper draft (c=0.05) doesn't just give a bigger speedup at the same acceptance rate. It also makes a larger K worth using (3 or more vs. 2). A cheap draft that gets rejected costs very little. So it is worth risking more speculative tokens per verification pass.
How to read the quantisation table: both columns face the same bit-widths; only which component (weights or cache) is quantised differs.
| Bits | Weights: KL, top-1 agreement | KV cache: KL, top-1 agreement |
|---|---|---|
| 8 | 0.00007, 0.997 | 0.00001, 0.998 |
| 6 | 0.0014, 0.975 | 0.0001, 0.997 |
| 4 | 0.027, 0.931 | 0.0019, 0.982 |
| 3 | 0.113, 0.875 | 0.011, 0.959 |
| 2 | 0.847, 0.705 | 0.091, 0.884 |
What to notice: at every single bit-width, the cache's KL divergence is roughly an order of magnitude smaller than the weights'. In this toy, the cache tolerates aggressive compression far better than the weights do. That is exactly the kind of asymmetry that has to be measured per model rather than assumed.
{{svg:memory|KV cache work, batch limits and decode-step time (assumed hardware).}} {{svg:serving|Batching policies, speculative decoding speed-up and quantisation (toy simulations).}}