CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 4 of 12 · 2 min read

History is the hidden cost of letting an agent see

Keeping every earlier screenshot in context doesn't just add a fixed cost per step — it makes total token use grow quadratically with the number of steps taken.

Think of taking a photo at every single step of a hike and carrying every single one of them with you for the rest of the hike. Compare that with just keeping the most recent photo and a short note about what you have seen so far.

In plain words: a screenshot costs far more tokens than a filtered text representation of the same screen, and if every earlier observation is kept in the model's context rather than discarded, the total tokens used across a multi-step task grow quadratically with the number of steps, not linearly.

A worked example. A single screenshot costs 1,334 tokens versus 848 tokens for a filtered DOM (interactive elements only) of the same page — already noticeably more expensive. Across a 10-step task, keeping every observation costs 73,370 tokens (about $0.22); keeping only the latest costs 13,340 tokens (about $0.04) — more than 5x less, purely from dropping stale screenshots. Shrinking the screenshot to half size and keeping only the latest brings it down further still, to 3,450 tokens (about $0.01).

The key lines. From code/mini-multimodal/costs.py:

# from code/mini-multimodal/costs.py
def keep_all_tokens(steps, per_step):
    """Input tokens summed over all steps when every earlier observation stays in the context."""
    return per_step * steps * (steps + 1) // 2

def keep_latest_tokens(steps, per_step):
    return per_step * steps

Read it as: keep_all_tokens multiplies by steps * (steps + 1) / 2, the classic sum-of-integers formula. keep_latest_tokens scales only by steps. This is the literal difference between quadratic and linear growth, written as two one-line functions.

Multimodal model (LMM), DOM and accessibility tree

A multimodal model takes more than one kind of input — text, images, audio. The DOM is the page's element tree; the accessibility tree is a platform-provided list of interface elements, roles and names, often cheaper and more precise than a screenshot when it's available.

Example: the 848-vs-1,334-token comparison above is exactly the DOM-vs-screenshot trade-off these two representations create — text costs less to transmit per step, which is why it's the default whenever it exposes the targets an agent needs.