CSE Roadmap

Common tier · about 6 hours · 10 parts

Part 7 of 10 · 5 min read

The suite, limits and checklist

Testing against the protocol, the limits of the example, and a checklist.

The suite: 41 checks, each tied to a rule

The conformance suite (conformance.py) is a list of functions with an id, the rule it comes from, and a title. They run against a server built from make_world(bugs).

Group Checks What they pin
Discovery (D) 3 card valid at the well-known path, ETag and 304, every listed interface answers
Authentication (A) 6 no token and the challenge, wrong audience, missing scope, another user's task looks absent, listing is scoped, joining someone's context
Version (V) 2 a missing version is treated as 0.3 and refused; an unsupported one is refused and the supported ones listed
Send (S) 11 blocking vs returnImmediately, direct message answers, question and answer on one task, duplicate message ids, server-made task ids, context mismatch, finished tasks refuse messages, malformed messages, media types, no speaking in the agent's voice
Tasks (T) 4 history length, list order and paging, cancel and release, a finished task never moves
Streaming (ST) 6 task then ordered updates, subscribe starts with a snapshot, two streams agree, close at an interrupted state, message-only streams, refusal when streaming is off
Push (PN) 6 ordered delivery with the credential, internal address refused at registration, name that later resolves inside blocked at delivery, secret not echoed, cleanup, capability checked
Extended card (E) 2 premium skill only for the right scope; an agent without one says so
Equivalence (Q) 1 both bindings agree

Does the suite work? Plant defects

Module 6's lesson applies again: a test is only worth what it would catch. The example plants 26 realistic defects. Each is a switch in the server, such as a missing version check, a task id accepted from the client, a stream that drops its last event, or a webhook sent to a private address. The example wrote down before running which checks were expected to fail.

  • On the correct server: 0 failures of 41.
  • 24 of 26 defects were caught, and for 24 the checks that failed were exactly the ones predicted.
  • Two predictions were wrong, both by being too narrow: blocking_returns_early also failed T2 (list order is by status-change time, not creation time, so a half-finished task moves in the order), and stream_drops_last_event also failed ST3. Neither was a missed defect; the effects had been under-counted.
  • Two defects escaped: working_message_dropped (a message sent to a working task is ignored instead of recorded) and no_push_retry (a failing webhook is not retried). Both are in the code as honest holes. A real check for the first needs to define what the agent should do with the message; for the second it needs a receiver that fails on purpose and a count of attempts. The test file asserts that both escape, so if someone closes the hole the test reminds them to update the counts.

Why 'predicted' matters

If you write down which checks should fail before you run the defect, a surprise tells you something about your understanding of the system, not only about the tests. Both surprises were small effects that had not been thought through.

What the example does not do

About this example What the example does not do
  • No language model anywhere. The agents are scripts. The conclusions about what a model would do are assumptions, and the tool-wrapper table is the weakest on that point.
  • No gRPC. The third binding is named, not built.
  • HS256 stands in for public-key signatures. It shows the checks; it does not show key distribution.
  • No real discovery. The router ranks cards that are handed to it; the specification leaves registries and curation out of scope, and so does the example.
  • The concierge cannot pass an authorization request upward (exercise Medium 3).
  • Time is virtual, so there is no real clock skew, no real timeouts, and no concurrency.
  • The choices where the specification is silent: the transition table, the notification token header, redaction of push secrets on reads, appending messages to working tasks, InvalidParams for another user's context id, and the HTTP status of a JSON-RPC version error.
  • The specification is young. It was read on 24 October 2026; check it before you rely on a detail.

How to build it so it holds up

  1. Treat every field in a card as a claim, and verify a signature against keys you provisioned yourself — because a card fetched from the network can always be forged by whoever serves it.
  2. Write the task's transition table down and enforce it in one place — because the specification does not publish one, so an implementer's own choices are the only thing stopping an illegal move.
  3. Scope every operation to the caller, and answer "forbidden" the same as "missing" — because telling the two apart tells an attacker which ids are real.
  4. Check a push URL against private and loopback ranges once, then use the checked address — because resolving twice lets the name change between the check and the call.
  5. Carry the user's identity through a delegation chain with an exchanged, narrowly scoped token, never the original or a shared service token — because only the exchange keeps results owned by the right user and keeps an audit trail of who acted for whom.
  6. Give a tool wrapper around an agent its own timeout, shorter than the task's, and surface a pause as a pause, not as an answer — because a wrapper that blocks until the task settles loses the question, the permission request and the task id all at once.
  7. Plant defects in your own conformance suite and predict which checks fail before running them — because a surprise then tells you something true about your understanding, not just about the tests.