CSE Roadmap

Core tier · about 4 hours · 10 parts

Part 2 of 10 · 2 min read

Why servers are rarely the bottleneck for an agent

Little's law turns a traffic estimate into a worker count, and the real limit usually shows up in tokens, not in servers.

Think of a queue at a coffee shop. How many people are in line depends on how fast they arrive and how long each order takes. It does not depend on how many baristas you imagine having. Count arrivals and service time first, and the right number of baristas falls out of that.

In plain words: the number of concurrent runs a system needs to support follows directly from how often requests arrive and how long each one takes, and for agents the number that actually binds is usually the tokens flowing through the model provider, not the worker count.

A worked example. A support platform estimated at 200 tenants, 500 tickets per tenant per day, with traffic peaking at 5 times the average: that works out to about 100,000 runs per day, 5.8 runs per second at peak, and — by Little's law — roughly 231 concurrent runs at peak, needing 12 async workers holding 20 runs apiece. None of that is the real constraint: the same platform needs about 10.4 million tokens per minute at peak against an assumed 2-million-token-per-minute provider limit, a headroom of just 0.19 — the tokens run out long before the workers do.

The key lines. From code/mini-design/design.py:

# from code/mini-design/design.py
def concurrent_runs(w): return peak_rps(w) * w.run_seconds           # Little's law: L = lambda * W
def workers_needed(w): return math.ceil(concurrent_runs(w) / w.runs_per_worker)

def tpm_needed(w, caching=False):
    i, c, o = tokens_per_run(w, caching)
    return peak_rps(w) * 60 * (i + o)

Read it as: concurrency is just arrival rate times how long each run takes. The token demand is that same peak rate multiplied by what each run actually costs in tokens. That is why the capacity question for an agent almost always has to be answered in tokens per minute, not instances or CPU.

Little's law, peak factor, headroom

Little's law says the average number of runs in a system equals the arrival rate times the average time each spends in it; a peak factor is peak load divided by average load; headroom is capacity divided by demand, with anything below 1 meaning the limit is being hit.

Example: 0.19x headroom on tokens per minute above — the limit that actually binds, found well before any worker count matters.