Part 4 of 12 · 2 min read
Continuous batching removes waiting, not compute
A batch that waits for its slowest member wastes every other slot's time; refilling slots as they free up removes that waste without changing how fast any single token is produced.
Think of a shuttle bus that waits for every passenger to finish their errand before leaving. Compare it with one that picks up a new passenger the instant a seat opens. The second gets far more people where they are going per hour, without driving any faster.
In plain words: static batching holds every request in a batch until the longest one finishes, which wastes slots on requests that finished early — continuous batching swaps a finished request out and a new one in on every step, so slots are almost never idle, even though each individual decoding step takes exactly the same amount of time either way.
A worked example. At a load of 0.04 requests per step, static batching has a p95 latency of 1,547 steps while continuous batching holds at 211 steps — roughly 7 times better, from scheduling alone. Push the load to 0.07 requests per step and static batching collapses entirely (p50 of 7,811 steps), while continuous batching barely moves (p50 of 64 steps) — static batching has a hard saturation point around 0.039 requests per step, while continuous batching keeps working smoothly well past it.
The key lines. From code/mini-inference/batching.py:
# from code/mini-inference/batching.py
if policy == "continuous":
active = [] # [request, remaining]
while nxt < n or queue or active:
while nxt < n and arr[nxt] <= step: queue.append(nxt); nxt += 1
while queue and len(active) < slots: i = queue.popleft(); active.append([i, ln[i]])
Read it as: the second inner loop refills open slots from the queue on every single step, not just when the whole batch finishes. This is the entire mechanical difference from static batching. It is what keeps slots from sitting idle while one long request finishes alone.
Static, dynamic and continuous batching
Static batching waits for a full batch and finishes everything together; dynamic batching also starts after a time window even if the batch isn't full; continuous batching swaps finished requests out and new ones in on every single step.
Example: the p95-at-0.04 comparison (1,547 vs. 211 steps) above is the direct cost of the "wait for the slowest" rule that static batching follows and continuous batching doesn't.