Part 6 of 12 · 3 min read
What the example found
Grounding accuracy by button size and shrink, sub-pixel phase search, click precision by error, token cost, step-compounding, and DOM coverage.
How to read the button-size table: each row is a button size; each column is a different screenshot shrink factor; all cells face the same noise level.
| Button size | Full size | Shrink 2x | Shrink 4x | Shrink 8x |
|---|---|---|---|---|
| 11 px | 1.00 | 0.99 | 0.23 | 0.01 |
| 18 px | 1.00 | 1.00 | 0.88 | 0.26 |
| 25 px | 1.00 | 1.00 | 1.00 | 0.81 |
| 32 px | 1.00 | 1.00 | 1.00 | 0.99 |
What to notice: the smallest button (11px) is the first to break, and it breaks hardest. It collapses from 0.99 to 0.23 between 2x and 4x shrink. None of the larger buttons show a cliff like that at the same step.
How to read the phase-search table: the top four rows compare noise levels with sub-pixel phase search enabled; the bottom row shows the same button size and noise without it.
| 18 px buttons | Full size | Shrink 2x | Shrink 4x |
|---|---|---|---|
| Phase search, noise 0.05 | 1.00 | 1.00 | 0.97 |
| Phase search, noise 0.1 | 1.00 | 1.00 | 0.88 |
| Phase search, noise 0.2 | 1.00 | 0.99 | 0.60 |
| Phase search, noise 0.4 | 0.99 | 0.81 | 0.47 |
| Aligned-pixel template only, noise 0.1 | 1.00 | 0.78 | 0.33 |
What to notice: at the same noise level (0.1), dropping sub-pixel phase search alone drops 4x-shrink accuracy from 0.88 to 0.33. The phase search is a larger lever on accuracy here than cutting the noise in half would be.
How to read the click-error table: each row is a button size; each column is a different amount of coordinate error in pixels.
| Click error (pixels) | 2 | 4 | 8 | 16 |
|---|---|---|---|---|
| 11 px button | 0.99 | 0.69 | 0.26 | 0.07 |
| 18 px button | 1.00 | 0.95 | 0.55 | 0.18 |
| 25 px button | 1.00 | 1.00 | 0.78 | 0.32 |
| 32 px button | 1.00 | 1.00 | 0.91 | 0.47 |
What to notice: 4 pixels of coordinate error is already a serious problem for the smallest button (0.69). It barely registers for the largest (1.00). The same absolute error has a wildly different effect depending purely on the target's own size.
How to read the token-cost table: each row is a different interface representation; the two right-hand columns show the compounding effect of keeping full history versus only the latest observation over ten steps.
| Observation (assumed constants) | Tokens per step | Ten steps, keep all | Ten steps, keep latest |
|---|---|---|---|
| DOM, interactive only | 848 | 46,640 ($0.14) | 8,480 ($0.025) |
| Screenshot 1280 x 800 | 1,334 | 73,370 ($0.22) | 13,340 ($0.04) |
| Screenshot at half size | 345 | 18,975 ($0.057) | 3,450 ($0.01) |
| Hybrid (filtered DOM + half screenshot) | 1,193 | 65,615 ($0.20) | 11,930 ($0.036) |
What to notice: for every single representation, "keep all" costs roughly 5.5x more than "keep latest." That is the exact ratio of the sum-of-integers formula at 10 steps. This ratio is structural. It is not a property of which representation you chose.
How to read the per-step-success table: each row is a per-step success rate; each column pair shows success without and with retry, at three different task lengths.
| Per-step success | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 0.99 (no retry / retry) | 0.95 / 0.99 | 0.90 / 0.98 | 0.82 / 0.96 |
| 0.95 | 0.77 / 0.94 | 0.60 / 0.89 | 0.36 / 0.79 |
| 0.90 | 0.59 / 0.87 | 0.35 / 0.75 | 0.12 / 0.57 |
| 0.80 | 0.33 / 0.69 | 0.11 / 0.47 | 0.01 / 0.22 |
What to notice: at 0.80 per-step success and 20 steps, raw task success is just 1 percent, which is essentially always failing. Retry brings it to 22 percent. Retry's absolute benefit grows larger exactly where raw compounding has done the most damage.
How to read the DOM-coverage table: each row is a different share of targets actually exposed by the DOM; the two columns compare DOM-only success against a hybrid that falls back to vision for the rest.
| Share of targets in the DOM | DOM only | Hybrid (screenshot fallback 0.875) |
|---|---|---|
| 50% | 0.49 | 0.93 |
| 80% | 0.78 | 0.96 |
| 95% | 0.93 | 0.98 |
| 100% | 0.98 | 0.98 |
What to notice: the hybrid row and the DOM-only row converge exactly at 100 percent coverage, where there is nothing left to fall back for. The entire value of the hybrid approach is concentrated in whatever share of targets the DOM is missing.
{{svg:vision|How resolution, button size and noise limit a screenshot grounder.}} {{svg:cost|Tokens, compounding errors and DOM coverage.}}