Part 7 of 12 · 1 min read
How to build it so it holds up
- Prefer text (DOM or accessibility tree) when it exposes the targets; add screenshots for visual or missing content.
- Filter the DOM to interactive elements before sending it.
- Annotate screenshots with marks and ask for a mark, not coordinates.
- Size the screenshot to the smallest target you must hit; test at your real resolution.
- Keep only the latest observation unless history is needed; summarise the rest.
- Verify each action (expected change appeared) and retry once before escalating.
- Set a token and step budget per task and log cost per task.
- Sandbox, filter and confirm: isolated browser, redacted screenshots, least-privilege credentials, human approval for sensitive actions.
- Treat page content as data, not instructions.