CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • A multimodal model predicts the next token from text and image tokens. A computer-use agent has an action planner, an action executor and an interface representation.
  • Three ways to show the interface: text (DOM or accessibility tree: cheap and precise), image (works anywhere, costs tokens, loses small text) and hybrid, the usual production choice. Marks turn coordinates into a choice.
  • Screenshot readers break on small targets and low resolution first, and click precision falls with target size; marks, crops and element lists avoid the problem.
  • Cost is a design constraint. Screenshots cost far more tokens than a filtered DOM, and keeping every earlier observation in context makes input grow quadratically; keep only the latest.
  • Errors compound across steps, so verification after each action with one retry is worth more than a slightly better model. Fall back to a screenshot when the DOM is incomplete, and treat security as part of the design.