Part 1 of 4 · 1 min read
Overview
Agents make many sequential model calls with long, repeated prefixes (system prompt, tools, history), and users feel TTFT and TPOT on every step. The cache, batching policy and prompt caching decide cost and latency per step; stacked optimisations compound, but each one can change quality, so re-run your evaluation after each.
About the example, and what is real
Real: a NumPy transformer with a working KV cache (equal outputs to the uncached version), the memory and decode-time formulas, a request-level batching simulator, the speculative accept/reject algorithm with distribution checks, a quantisation experiment, and 23 tests in code/mini-inference. Assumed: the A100-like peak of 312 TFLOP/s, the 2 TB/s bandwidth, request lengths (lognormal, median 60), one decoding step as the time unit, a $3 per million token price. Not real: no GPU, no kernel effects, no network or scheduling overhead, no prefill modelling in the batching simulation; the quantisation toy is a small random-weight model, not a trained one.