Part 8 of 12 · 2 min read
Answering multimodal-agent questions in an interview
DOM, accessibility tree or screenshots for a web agent?
"The DOM is cheap and gives exact selectors, but it misses visual context and some widgets; screenshots work everywhere but cost tokens and lose fine detail. I filter the DOM to interactive elements and fall back to an annotated screenshot when the target is missing. In my arithmetic a filtered DOM was 848 tokens against 1,334 for a full screenshot, and a 50 percent DOM coverage cut DOM-only success to 0.49 while a hybrid held 0.93." Follow-up: "And on a desktop app?" Usually only screenshots or an OS accessibility API are available.
Why do computer-use agents fail, and how do you improve reliability?
"Errors compound: 0.95 per step is 0.60 after ten steps. Failures come from grounding (wrong element or coordinates), stale state and pop-ups. I narrow the choice with marks, verify each action's effect, retry once, and ask the user when confidence is low. In the example's toy, a check that caught 80 percent of failures lifted a 20-step task at 0.95 per step from 0.36 to 0.79." Follow-up: "What do you measure?" Success per step and per task, and which step types fail.
How do you control the cost and latency of a screenshot agent?
"Capture only when needed, shrink screenshots to the smallest size that still resolves the targets, drop old observations, cache the stable prefix and use a smaller model for routine steps. Keeping every screenshot makes tokens grow quadratically: 73,370 against 13,340 in my ten-step example." Follow-up: "What does shrinking cost?" In the example's experiment 18 px buttons stayed at 1.00 at 2x shrink but fell to 0.88 at 4x and 0.26 at 8x.
What are the security risks of a computer-use agent?
"Prompt injection through page content, credential exposure, third-party extensions, and screenshots that capture private data. I sandbox the browser, redact sensitive fields before storing captures, use short-lived least-privilege credentials, rate-limit, log actions, and require confirmation for sensitive actions." Follow-up: "Can the model be trusted to ignore injected text?" No; defences must be outside the model, such as permission checks on actions (earlier levels on guardrails).