CSE Roadmap

Core tier · about 9 hours · 13 parts

Part 7 of 13 · 3 min read

End-to-end and transport tests

Testing the server through a reader, and over the real transport.

End-to-end tests with a stand-in reader

The unit and conformance layers check that the server is right. This layer checks that a reader of the advertised text can use it. The reader is a script (part 3's warning applies). It does two things. It chooses a tool for a goal, by counting the words the goal shares with each tool's name and description. Rare words count for more, so a word that appears in every tool, like "ticket," says nothing. And it recovers from an error by following the hints in it.

Tool selection. The example uses eight goals, one per tool, such as "mark ticket 7 as solved" or "hand ticket 4 to a staff member":

Descriptions Right choices
the full descriptions 8 of 8
ticket_close described only as "Tickets." 7 of 8 (it chose ticket_assign)
names only, no descriptions at all 3 of 8

Warning: I adjusted the stand-in, and you should know how

The first selector counted raw shared words and chose the right tool for 6 of 8 goals on the clean server, because the word "ticket" appears in nearly every description. It was changed to weight rare words more. After the change the clean server scored 8 of 8. So 8 of 8 is not evidence that the descriptions are good; it shows that the harness can see a difference when a description is removed or emptied. The goals were also written after the descriptions. A real selection test needs goals written by someone else, ideally taken from real conversations (module 9).

Recovery. Eight scenarios (part 3), each run through the stand-in with at most three repairs, passing when the final database state is right and no more than five calls were made. All eight pass on the correct server. One further check is a multi-step triage task, run as scripted calls: find open urgent tickets that nobody owns (one: ticket 11), assign it, comment on it, read it back, and check the server logged no unexpected exception.

Transport tests

A server can pass every in-process test and still fail the moment it is started, because the transport adds failure modes that no function call has.

Standard input and output. The stdio_main.py server reads one JSON message per line on stdin and writes one per line on stdout, and writes nothing else on stdout. Logs go to stderr. The debugging page says it plainly: local servers should not log to stdout, because it interferes with the protocol; stderr is captured by the host. The test starts the server as a child process (a real one), runs the full connect, list and call sequence, and checks that stderr received the "ready" log. The defect is the classic one: a friendly start-up banner printed to stdout. What the host sees:

stdout carried a line that is not a JSON-RPC message: 'helpdesk server v1.0 starting up...'

No in-process test can catch that, because the in-process pipe never uses a real standard output. Only a test that starts the real program can.

HTTP on a real socket. The test serves module 5's protected server on 127.0.0.1, sends a real HTTP request with a token, and checks 200, a JSON content type and the structured result; then sends the same request without a token and checks for 401 with a resource_metadata challenge.

The debugging page also lists problems that only appear when a host launches the server. The working directory can be undefined, so use absolute paths. A stdio server inherits only a limited set of environment variables, so credentials must be passed explicitly. A test of "does the server start from a directory it did not expect?" is a cheap addition (Easy exercise 3).