Part 9 of 12 · 2 min read
Mistakes that waste tokens or miss the target
Common mistakes, and what the example does not do.
Mistakes that waste tokens or miss the target
- Sending the whole DOM instead of the interactive elements.
- Asking a model for raw coordinates on small targets.
- Shrinking screenshots without testing the smallest target.
- Keeping every screenshot in the context.
- No verification after an action.
- Assuming the DOM covers every widget.
- Letting page text act as instructions.
- Reporting step accuracy as task accuracy.
What the example does not do
About this example What the example does not do
- No language model: the grounder is a template matcher with the exact template; real models read text, handle layout and also err in ways unlike pixel noise.
- Glyphs, not words: random binary patterns stand in for labels; real labels carry meaning, fonts and scale changes.
- One screen at a time: no scrolling, pop-ups, dynamic content, drag, or multi-step state changes.
- Independent steps in the compounding arithmetic; real failures correlate.
- Assumed constants: tile size, characters per token, price, DOM success and failure-detection rates.
- No security test, no real browser, no Playwright.
- No audio or video; the module covers screens and images only.
- Optional real experiment: pick a form-heavy site, run a Playwright agent three ways (DOM only, screenshot only, hybrid) on 30 tasks, and record success, steps, tokens and dollars.
Check yourself
Why does verifying every action matter more than a slightly better model?
Show answer
Errors compound: at 0.95 per step, 20 steps succeed only 36 percent of the time. A check that notices 80 percent of failures and retries once lifts that to 79 percent.
For the curious History is the hidden cost
Keeping every earlier screenshot in the context makes input tokens grow quadratically: a ten-step task used 73,370 tokens against 13,340 when only the latest was kept (about $0.22 against $0.04 at an assumed $3 per million). Small targets break first: with a pixel matcher, 11 px buttons failed once the screenshot was shrunk 4x. The matcher, the token constants and the screens are synthetic, so none of this measures a real multimodal model.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Prefer the DOM or accessibility tree when it exists | Add a screenshot when it is incomplete or visual. |
| Keep only the latest observation | Summarise the rest. |
| Verify each action's effect | Then retry once. |
| Sandbox the agent and confirm sensitive actions | Treat page content as untrusted input. |