Part 1 of 9 · 3 min read
Overview
In this module you will
- Define a browser agent and a computer-use agent, and describe their loop: observe, decide, act, observe again.
- Compare three ways of looking at a page (the page source, the accessibility tree, a screenshot): what each can read, what each can click, what each costs, and how each fails.
- Explain, with a table, which hiding place for text reaches which kind of agent, and why a page can say one thing to a person and another to an agent.
- Judge an action by what it would do, not by what the agent says it is doing, and build the harness that does this.
- Use six layers (hygiene, a pattern detector, a host list, a task contract, confirmation by the person, a password vault), say what each stops and what it does not, and show with an ablation which layers overlap.
- Handle screenshot coordinates correctly, including the bug that makes every click miss.
- Test a harness: 30 checks, 31 planted defects, and a comparison of my page model with a real Chromium.
- Answer "how would you secure a browser agent?" in a way that says what the model cannot be asked to do, and what remains unsolved.
Everything earlier in this level gave an agent interfaces made for programs: a function, an MCP server, another agent. A browser agent is different. It uses the interface that was made for people, because for much of the work people want done no program interface exists.
The price of using the human interface is that the agent now reads what any web page author wrote. Module 8 covered an agent that runs code from a model; this one covers an agent that reads text from strangers and holds the keys to your accounts. The page can contain a sentence addressed to the agent, and the agent has a mouse.
About the examples, and what is real
Real: the page-visibility rules (which text is in the source, in the accessibility tree, in the pixels) were measured in headless Chromium and my page model agrees with it on all 12 cases (section 2). The coordinate arithmetic, the layer logic, the contracts and the mutation testing are ordinary code, run here.
Imitated: the "web" is a small offline world: a shop, a partner shop, a mail site, a bank, a look-alike bank and an attacker's collector, all on reserved names (.example, .test) that cannot resolve on the real internet. All people, addresses and numbers are invented; the card number is the well-known test number. The pages are tiny, with a handful of elements, so token counts show proportions, not what real pages cost. The agent is a script, not a model. Its "reading" of a screenshot is perfect text recognition, which flatters it. The attacker's stand-in obeys every instruction it can read. It is the worst case: what the harness stops here it would stop for any model, but how often a real model would obey is a different number and I have not measured it (section 11 quotes Anthropic's figures, which are theirs, not mine).
Missing: real HTML and CSS layout, frames, shadow DOM, pop-up windows, file dialogs, downloads, CAPTCHAs (a real agent should stop and hand those to the person), two-factor sign-in, native desktop applications, time and latency. Numbers in the notes are exact for this code and direction only for the real world. Hard exercises say "I did not run this, so your numbers are the ground truth".