Part 2 of 12 · 2 min read
Grounding a click gets harder as targets shrink and resolution drops
Finding "the Save button" in a screenshot is a search problem, and shrinking the image or the button both make that search fail more often.
Think of trying to spot a specific person's face in a crowd photo. It is easy at full resolution. It is much harder once the photo is scaled down to a thumbnail. It is hardest of all for the smallest faces in the crowd.
In plain words: grounding — mapping an instruction like "click Save" to an exact screen location — depends on template matching finding a strong, unambiguous match, and both small buttons and low-resolution screenshots make that match weaker and less reliable.
A worked example. At full resolution, every button size in this toy (from 11 to 32 pixels) is found correctly nearly 100 percent of the time. Shrinking the screenshot 4x drops the smallest (11px) buttons' success rate to just 0.23, while the largest (32px) buttons stay at 1.00. Push the shrink to 8x and even the 18px buttons fall to 0.26 — small targets and aggressive downscaling compound against each other, and this is the best case: a matcher given the exact template at the same resolution, not a real multimodal model guessing from context.
The key lines. From code/mini-multimodal/detect.py:
# from code/mini-multimodal/detect.py
def locate(img, label, m, bank, f=1, phases=True):
"""Best match over all f x f sub-pixel phases; returns (x, y) centre in ORIGINAL pixels and its score.
With phases=False only the phase-0 template is tried (a matcher that assumes alignment).
"""
Read it as: the function searches over sub-pixel phases to find the best-scoring match, not just one fixed alignment. This sub-pixel search is itself a meaningful chunk of the accuracy. Dropping it (phases=False) is exactly the difference the next part measures.
Grounding, normalised cross-correlation (NCC)
Grounding is mapping an instruction to the exact element or screen location it refers to. Normalised cross-correlation is a score from -1 to 1 for how well a template matches an image window, ignoring differences in brightness and contrast.
Example: the 0.23 success rate for 11px buttons at 4x shrink above is grounding failing in its purest form — the template genuinely can no longer be matched reliably once the image has lost that much detail.