CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 5 of 12 · 2 min read

Speculative decoding and quantisation both trade a measured risk for speed

Both techniques only pay off when you actually measure the thing they risk — acceptance rate for one, output quality for the other — on your own workload.

Think of a junior assistant drafting several paragraphs quickly for a senior editor to approve or correct in one pass. Compare that with the editor writing every word themselves. It is faster overall, but only if the junior's drafts are usually right.

In plain words: speculative decoding has a cheap draft model guess several tokens ahead and a target model verify them all in one pass, keeping the target's exact output distribution — and quantisation stores numbers in fewer bits to save memory and bandwidth — but both only help once you've actually measured the thing each one risks: draft acceptance rate, and output quality respectively.

A worked example. With a draft model that costs 12.8 percent of a target step (c=0.128), reaching an 80 percent acceptance rate gives a 2.25x speedup; at 90 percent acceptance, 3.03x. Below roughly 70-80 percent acceptance, the gains get much smaller, because the draft's cost has to be paid even on the tokens it gets wrong. Separately, quantising the KV cache tolerates far more compression than quantising the weights in this toy: at 4 bits, the weights' output distribution diverges from the 16-bit baseline by a KL of 0.027 (93.1 percent top-1 agreement), while the cache at the same 4 bits only diverges by 0.0019 (98.2 percent agreement) — which component tolerates compression better is not something you can assume, it has to be measured per model.

The key lines. From code/mini-inference/spec.py:

# from code/mini-inference/spec.py
def speculative_sample(p, q, rng):
    """Draw x ~ q; accept with probability min(1, p/q); otherwise resample from the residual max(0, p - q), normalised. The result is distributed as p."""
    x = int(rng.choice(len(q), p=q))
    if rng.random() < min(1.0, p[x] / q[x]): return x, True
    r = np.maximum(p - q, 0); r /= r.sum(); return int(rng.choice(len(p), p=r)), False

Read it as: the docstring states outright that the result is distributed as p (the target), not q (the draft). The accept/reject math is specifically constructed so that speculative decoding never changes what the model would have said. It only changes how fast it says it. A rejected draft token is actively corrected, not just thrown away.

Speculative decoding, quantisation

Speculative decoding has a draft model propose K tokens, which the target model verifies in one pass, with a rejection rule that keeps the target's exact distribution. Quantisation stores numbers (weights, cache, or activations) in fewer bits to cut memory and bandwidth, at some cost in accuracy.

Example: the cache-vs-weights quantisation gap above (0.0019 vs. 0.027 KL at 4 bits) is the entire reason this module's design rules say to quantise one component at a time and measure each separately, rather than assuming a single bit-width choice applies uniformly.