Bending the cost curve

1. Prompt caching. If the start of the prompt is identical to the previous call, the provider can reuse its work and bill that prefix at a lower price. In the demo this takes a 40-step run from $2.59 to $0.45. It works only when the prefix stays exactly the same: put stable content (system prompt, tool definitions, earlier turns) first and changing content last. Never edit earlier messages.
2. Trim the history. Keep only the last few rounds, or replace old rounds with a summary. Keeping the last three rounds makes growth linear: $0.69 at 40 steps. The cost is that the model forgets what you removed.
3. Shrink tool results. An 800-token tool result is repeated at every later step. Return only the fields the model needs, not a whole web page or a set of database rows. With 200-token results instead of 800, the 40-step run costs $1.19 instead of $2.59. This is often the largest win, and it costs nothing.

Other levers: use a smaller model for easy steps (module 5), cap the number of steps, and cap output length.

Strategies interact

Caching and trimming do not simply add up. Trimming changes the start of the prompt, which breaks the cached prefix. In the example's model, I get these costs:

For runs up to 20 steps, adding trimming on top of caching makes things worse, because it throws away cached input. Only at long runs does it break even. Before you stack tricks, measure the combination.

Break-even for trimming alone (no caching): keeping only the last 1 round becomes cheaper from step 3, keeping 3 from step 5, and keeping 6 from step 8. Before those steps there is nothing to trim.