Part 3 of 12 · 2 min read
Why an agent's cost grows faster than its step count
Each step resends the whole history, so cost grows roughly with the square of the steps.
Think of someone retelling an entire story from the beginning every time they add one new sentence to it. By sentence forty, most of what they say is just repeating what was already said.
In plain words: an agent resends its whole history with every step, so a run's cost grows much faster than the number of steps. Caching the unchanged part of that history is the single biggest lever.
A worked example. With a 2,000-token system prompt and 850 tokens added per step, a 5-step run costs $0.083. Doubling to 10 steps more than doubles the cost to $0.230; 20 steps costs $0.716; 40 steps costs $2.451. The cost roughly quadruples each time the step count doubles. Caching the repeated prefix, by billing the unchanged part of the history at a tenth of its normal price, cuts those same four runs to $0.039, $0.078, $0.177 and $0.450. The saving grows with run length, because there is more repeated history to cache.
The key lines.
# from code/mini-cost/costlab.py
def agent_run_cost(model, steps, system=2000, new_per_step=600, out_per_step=250, caching=False):
total, hist = 0.0, system
for k in range(steps):
cached = hist if (caching and k > 0) else 0 # previous prefix is a cache hit
total += call_cost(model, hist + new_per_step, out_per_step, cached_in=cached)
hist += new_per_step + out_per_step
return total
Read it as: the history grows by one step's worth each time. With caching on, everything except the newest addition is billed at the cheap cached rate. The cache only helps if the prefix is byte-identical. So changing content has to go at the end, never in the system prompt or tool list.
Prompt cache, prefix cache
Reuse of the already-processed form of an unchanged prompt prefix, billed at a fraction of the normal input price.
Example: cached_in=hist on every step after the first, once the system prompt and tool definitions stop changing.