Experiment 1: what each view can and cannot do

Experiment 1: what each view can and cannot do

The task: buy the cheapest USB-C cable on the shop (five products; one is a Lightning cable that is cheaper, to catch a script that takes the lowest price blindly; one is an adapter). The plan is the same for all views, written against what they can see: read the prices, open the cheapest cable, check the price on the product page, add it, check out. Each view uses only what it has: the source view can use a test identifier, the tree cannot, the screenshot can recognise an icon's look. On the plain page all five tasks succeed in all three views. Then I add one real-world difficulty at a time to the shop.

What the table shows, view by view:

• Page source fails on a hidden look-alike button. A second "Add to cart" link, positioned off screen and adding a different product, comes first in the source. The script clicks the first match and buys the Lightning cable (red). It also fails when the price is painted as pixels, because the source holds no text for it, and it gives up rather than guess.
• The accessibility tree fails in four places: the look-alike button (the tree lists off-screen elements), painted prices, an icon button with no name, and a price marked aria-hidden. For the unnamed icon the plan's rule is "never click what you cannot name", which is my choice. When I let it guess (guess_unnamed), it picked the single unnamed button for a different query (closing the cookie banner) and added the item twice, ordering two cables. A guess made for one purpose is applied to another.
• The screenshot succeeds on every one of these, and fails where the others do not: a cookie banner that the agent ignores covers the page, the prices cannot be read (they are under it), and it gives up. A far-away button costs two scrolls (8 steps against 6). It is the most expensive: about 1,049 tokens per look (my estimate: width times height over 750), 7 looks, 7,343 tokens for the purchase, against 346 for the source and 364 for the tree.
• Waiting matters for all: when the list loads a moment after the page, each view needs one extra wait (7 steps against 6).

Each view fails somewhere, and the failures are different, which is why systems that need reliability mix them (a screenshot to see, a tree to name). Mixing also means their hiding places add up (section 6).

The cost of reading, as pages grow

The example's pages are small. To see the shape, I built result pages of 5, 50, 200 and 1,000 items and counted what each view returns (text at about four characters a token; a screenshot at the formula above).

A screenshot costs the same however long the page is, so the lines cross between 50 and 200 items on these very plain pages. On real pages every visible item carries wrappers, classes and scripts, so text views grow faster. In Chromium, a 200-row page of mine (laid out by my own generator, so its source is inflated by inline styles) was 55,101 characters of source, 3,482 of text and 8,794 of accessibility snapshot, and a 16,954-byte screenshot. Those ratios, not my token counts, are the real information: the cheapest view depends on the page, and a screenshot also needs the model to read it correctly. The documentation I read adds a fixed cost: declaring the computer-use tool set adds about 4,500 input tokens to every request.

Experiment 2: pointing at things

A screenshot agent says "click at (412, 307)". Two things can go wrong.

The picture was shrunk and the coordinates were not scaled back. The documentation I read gives size limits (about 1,568 pixels on the long edge and 1.15 megapixels for earlier models, 2,576 pixels and 3.75 megapixels for newer ones; the API does not shrink images for you) and says to shrink the screenshot yourself, then divide the model's coordinates by the same factor before clicking. Its recommended screen sizes are 1024 by 768 or 1280 by 720 for general use and nothing above 1920 by 1080. To test the rule I clicked the middle of a 32-pixel-high button after taking screenshots at four scales:

The bug is silent at full size and catastrophic below it, and at 0.75 it clicks the wrong button more often than the right one. The planted defect no_rescale is caught by check C4.

The model's aim is off. Real models can miss or invent coordinates, and the documentation says so, suggesting zoom and smaller steps. I modelled it as an error of a fixed number of image pixels in a random direction and clicked the middle of buttons of different heights with small gaps (6 pixels) between them:

Small, close controls fail first, and when they fail the click often lands on a neighbour, which is worse than a miss because it does something. On the whole purchase task (20 seeds per error size), success was 100% up to 16 pixels of error, 15% at 32, 0% at 64 (the rows are 32 pixels high, so the cliff is where the error passes the half-height). With a small icon button the cliff is at the same place. Numbers depend on my layout, which is the point: target size and spacing decide how much aim you need, and a page designed for fingers is easier for an agent.