CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 3 of 12 · 2 min read

Coordinates are fragile; choosing among marked elements isn't

Asking a model to output precise pixel coordinates turns grounding into a hard regression problem; asking it to pick a numbered box turns the same problem into an easy multiple-choice one.

Think of asking someone to point at exactly pixel (412, 88) on a photo. Compare that with asking them to say "box number 3" when the boxes are already drawn and numbered. The second is just easier to get right, for the same underlying target.

In plain words: click precision falls off sharply as a target shrinks or coordinate error grows, which is exactly the failure mode set-of-mark prompting — drawing numbered boxes on the screenshot and asking the model to answer with a mark number instead of raw coordinates — is designed to avoid.

A worked example. With 4 pixels of coordinate error, a small 11px button is hit only 69 percent of the time, while a larger 32px button is hit 100 percent of the time. Push the error to 16 pixels, and even the 32px button's hit rate drops to 47 percent. This is the exact mechanism set-of-mark prompting sidesteps: instead of asking a model to output a precise (x, y) that has to land inside a small region, you ask it to choose among a handful of pre-identified, already-labeled candidates — turning a hard regression problem into an easy classification one.

The key lines. From code/mini-multimodal/grounding.py:

# from code/mini-multimodal/grounding.py
def hit_probability(size, sigma):
    """P(a click centred on the target with Gaussian error sigma (pixels, per axis) lands inside it)."""
    if size <= 0:
        raise ValueError("target size must be positive")
    if sigma <= 0:
        return 1.0
    p = erf(size / (2 * sigma * sqrt(2)))
    return p * p

Read it as: the hit probability depends on the ratio of target size to coordinate error, squared across both axes. This is why small targets are disproportionately punished by coordinate noise. The formula's sensitivity to size is compounding, not linear.

Set-of-mark prompting, action space

Set-of-mark prompting draws numbered boxes on a screenshot so the model answers with a mark number rather than raw coordinates. The action space is the set of actions the agent may take (click, type, scroll, select) made available to its planner.

Example: the gap between a 69-percent hit rate at 4px error and near-100-percent set-of-mark accuracy (choosing correctly among a few clearly numbered boxes) is the entire reason production computer-use agents favor marks over raw coordinate output.