CSE Roadmap

Specialist tier · about 8 hours · 12 parts

Part 5 of 12 · 3 min read

Pointing at things

Reading a map at one scale and then walking by the measurements of a different-scale map puts you somewhere else entirely — sometimes close, sometimes nowhere near. Screenshot coordinates have exactly this failure mode.

In plain words: a coordinate the model gives you is only correct for the screenshot it was given; sending it straight to a screen of a different size silently moves every click.

A screenshot agent says "click at (412, 307)." Two things can go wrong. The picture was shrunk and the coordinates were not scaled back. The documentation checked gives size limits. For earlier models they are about 1,568 pixels on the long edge and 1.15 megapixels. For newer ones they are 2,576 pixels and 3.75 megapixels. The API does not shrink images for you. The page says to shrink the screenshot yourself, then divide the model's coordinates by the same factor before clicking. The browser's own rescale step is one line:

# from code/mini-browser/browser.py
def to_screen(self, x: float, y: float) -> tuple[float, float]:
    if bugs.on("no_rescale"):
        return x, y
    return x / self.shot_scale, y / self.shot_scale

Read it as: the correct path divides by the scale the screenshot was shrunk by. The planted defect no_rescale simply returns the model's coordinates unchanged, as if the screenshot were still full size.

Screenshot scale, coordinate rescaling

The screenshot scale is the ratio between the image sent to the model and the real screen it was taken from. Coordinate rescaling is dividing a model-given coordinate by that ratio before sending it to the real screen, so a click lands where the model actually saw it. Example: a screenshot shrunk to 0.5 of the real screen means a model coordinate of (412, 307) must become (824, 614) on the real screen — skip that division and the click lands at exactly half the intended distance from the origin.

To test the rule, the example clicked the middle of a 32-pixel-high button after taking screenshots at four scales:

Scale of the screenshot Coordinates scaled back Coordinates not scaled back
1.0 100% right 100% right
0.75 100% right 14% right, 62% on a neighbouring button, 23% on nothing
0.5 100% right 0% right, 100% on nothing
0.35 100% right 0% right, 100% on nothing

The bug is silent at full size and catastrophic below it, and at 0.75 it clicks the wrong button more often than the right one. The planted defect no_rescale is caught by check C4.

The model's aim is off. Real models can miss or invent coordinates, and the documentation says so, suggesting zoom and smaller steps. It is modelled as an error of a fixed number of image pixels in a random direction, clicking the middle of buttons of different heights with small gaps (6 pixels) between them:

Button height 4 px error 8 px error 16 px error
12 px 100% right 57% right (43% on nothing) 24% right, 38% on a neighbour, 38% on nothing
20 px 100% 100% 48% right (52% on nothing)
32 px and 48 px 100% 100% 100%

Small, close controls fail first. When they fail, the click often lands on a neighbour, which is worse than a miss because it does something. On the whole purchase task (20 seeds per error size), success was 100% up to 16 pixels of error, 15% at 32, 0% at 64. The rows are 32 pixels high, so the cliff is where the error passes the half-height. With a small icon button the cliff is at the same place. Numbers depend on the layout, which is the point. Target size and spacing decide how much aim you need, and a page designed for fingers is easier for an agent.