CSE Roadmap

Specialist tier · about 8 hours · 12 parts

Part 9 of 12 · 6 min read

The weakest layer, and what is unmeasured

The detector is the weakest layer, and why

The detector is a pattern list: it removes lines containing "assistant:" or "AI agent:" and phrases such as "ignore all". It is a stand-in for a trained classifier, and it shows why one pattern list is not one.

  • It catches all 6 canonical wordings and none of the 6 paraphrases ("Note to AI", "Dear AI model", "Attention AI" instead of "Assistant:"). With the detector as the only layer, all 180 harming cases happen against the paraphrases.
  • It flags 5 of 12 ordinary lines: "Customer assistant: Priya will help you at the desk", "Please ignore all the noise in the video", a book title containing "AI agents:", and two more. A defect that makes it broader (any "note", "ai" or "ignore") is caught by a check because it removes ordinary text.
  • A defect that makes it case-sensitive is caught only because the sample attack starts with a capital letter. Nothing was expected to catch it. A lower-case "assistant:" would get through.

Real classifiers are models and do much better against rewording, and Anthropic reports using one on content before it reaches the model. They still miss things, and a classifier is a probability, not a wall. The hygiene, host and contract layers do not depend on understanding the attacker's sentence, and that is why they are the layers to build first.

What the model-side defences add, and what has not been measured

Everything above is outside the model. Anthropic's research page describes three layers inside and around the model: training the model on injections (reinforcement learning), classifiers over untrusted content, and human red-teaming. It reports attack success rates against an adaptive attacker (a best-of-N search with 100 attempts per environment, about 2,200 to 3,200 attempts per figure). These are their numbers on their test, not the example's:

Model Model only Old safeguards Improved safeguards
Sonnet 4 13.8% 10.7% 2.0%
Sonnet 4.5 13.8% 10.8% 1.9%
Haiku 4.5 12.0% 11.1% 2.4%
Opus 4.5 8.0% 7.6% 1.4%

Three honest readings. A model that resists is far from one that always obeys (the stand-in is the second). The figures are single-digit and low-double-digit percentages against a persistent attacker, falling to about one to two percent with extra safeguards. Even the best number is not zero. The page says plainly that no browser agent is immune and that the web is adversarial. And the numbers are for those models, those tests and that date. That is why a harness that holds even when the model fails is the right thing to build. One percent of a million tasks is ten thousand incidents.

Does the suite work? Planted defects

The harness has 30 checks. (One more compares the page model with Chromium and needs a Chromium installed. All 31 pass on the correct harness.) 31 realistic defects were switched on, one at a time. They include a view that lists hidden elements, clicks that skip the scale-back, a host check that accepts suffixes, a contract that ignores parameters, a confirmation that shows the agent's words, and a vault that fills anywhere. There are more.

  • 28 of 31 were caught; 28 matched the prediction exactly (25 caught as predicted, and the 3 escapes below, which were predicted).
  • Three escaped (they were thought of after the checks were written, and no check was expected to notice them): hygiene_drops_aria_hidden (hygiene throws away text that is drawn on screen), confirm_total_wrong (the order prompt shows a total of zero) and step_cap_off_by_one (the cap allows one step too many). These are real holes in the suite. Closing them is exercise Easy 3.
  • One prediction was wrong: detector_case_sensitive. It was expected to escape; check C22 caught it, for the accidental reason in part 10.
  • Two had an extra failure: a form that forgets prefilled values (form_drops_prefill) and a mail prompt without the recipient (confirm_hides_recipient) both also failed the check that a legitimate reply asks the person once, because the prompt or the form is part of that legitimate path.

One more finding came from a check that was written wrongly. The first test for "an agent that repeats a failing click stops" used an agent that ignored a cookie banner. It did not loop: it gave up straight away, because it could not read the prices under the banner. It was replaced with a plan that repeats one action thirty times. A check has to produce the situation it claims to test.

What the example does not do

About this example What the example does not do
  • No real page structure. No frames, shadow DOM, pop-up windows, redirects, downloads, file dialogs, drag and drop, keyboard shortcuts, or time. Any of them can change the results.
  • No CAPTCHAs, no two-factor sign-in. A real agent should stop and hand these to the person; bypassing bot detection is not something to build.
  • A scripted agent. No model reasons, mis-clicks for its own reasons, or recovers from a surprise. The conclusions about a real model's behaviour are assumptions.
  • A perfect reader of screenshots. Real models misread small text, which makes the screenshot results friendlier than they would be.
  • One worst-case attacker. It obeys every sentence it can parse. It does not adapt to the layers, it does not try to split an instruction across pages, and it does not use images with embedded text beyond one canvas.
  • A "person" that is a script. The fatigue result is a model of behaviour, not a measurement.
  • Desktop applications. Computer use on a native app has no page source, and the accessibility interface differs by operating system. The example is a browser world. The computer-use documentation checked says to run it in a dedicated virtual machine or container with minimal privileges, and Claude Code's own documentation (read for module 8) says computer use runs on your actual desktop, not isolated, which is the opposite of the sandbox of module 8. Putting a browser agent inside the module 8 sandbox is a good idea and is not drawn here.

How to build it so it holds up

  1. Judge an action by the request it would send, never by the agent's own description of it — because a page that can steer the model's words can steer its account of what it is doing just as easily.
  2. Build layers that do not depend on understanding the attacker's sentence — because hygiene, host checks and task contracts stop attacks a classifier would miss on a rewording.
  3. Fix the writes a task may make before reading any page, and give read-only tasks none — because a contract decided after the page is read can be shaped by what the page says.
  4. Keep credentials out of the model's context entirely — because a secret the model never holds cannot be leaked by any instruction, however convincing.
  5. Scale screenshot coordinates back to the real screen every time, and test it at every scale you use — because the bug is silent at full size and catastrophic below it.
  6. Ask the person to confirm only irreversible actions, and build the prompt from the request, never from the agent's words — because confirmation that happens often teaches the person to say yes without reading, and a prompt built from the agent's cover story approves exactly what it should block.
  7. Plant defects and predict which checks should fail before running them — because a passing suite proves nothing about a hole nobody thought to test for.