Part 6 of 7 · 5 min read
Exercises
Easy
Easy 1: Run it and break it by hand (30 minutes)
In code/mini-build, run pytest, then python evaluate.py and python demo.py. Then open service.py, comment out the status = ? filter in search by hand, and run pytest -x. Note which check fails first and what its message says. Restore the file.
Done when: you list the seven layers with their check counts (18, 2, 1, 26, 3, 3, 6), say which one failed for your edit and why that layer and not another, and explain why your manual edit failed one unit check while the other layers stayed green.
Hint
use pytest -x -q; the failing check's name tells you the layer's check list.
Easy 2: Kill the escapee (30 minutes)
Two defects escaped every layer: tie_order and comment_no_touch. Write one new check for each, in layers.py, in the layer where it belongs. Run it against the correct server and against the defect.
Done when: each new check passes with no defect and fails with its own defect (show both outputs), and python evaluate.py now reports 25 of 25 defects caught.
Hint
for tie_order, urgent tickets 4 and 11 differ only in their updated value; for comment_no_touch, read updated before and after a comment.
Easy 3: Rewrite three error messages (40 minutes)
Pick three failures in service.py whose hint you think is weak (or add a failing case of your own, such as assigning a closed ticket). Rewrite them. Add each to PROBES in layers.py and make error_texts_say_what_to_do_next pass.
Done when: you show each message before and after, say what the model should do next after reading it, and the lint passes without you changing a lint rule.
Hint
the lint's three error rules are in lint.py; the cheapest fix is usually a sentence that names the next tool in backticks.
Medium
Medium 1: A ninth tool, through every layer (2 hours)
Add ticket_merge(source_id, target_id), which moves the comments of one ticket onto another and closes the source with a resolution that names the target. Build it service first. Then add everything the suite expects: tool description and schemas, an annotation, a scope, unit checks, conformance coverage, an end-to-end intent, a new approved snapshot, and two new defects for it in mutants.py with your predicted layers.
Done when: all 59-plus checks pass, the matrix shows your two new defects, you compare your prediction with the measurement, and your commit message explains why approved_tools.json changed.
Hint
Env.client(...) and layers.check are all you need for a new check; run python -c "import app, snapshot; snapshot.approve(app.Env().client().list_tools())" only after reviewing the diff.
Medium 2: Put a ceiling on results (2 hours)
Give desk_export a limit and a cursor. Then add a conformance check that asserts no tool result is larger than 12,000 characters in its worst case (create enough tickets to make that true for every tool). Add a defect that removes the cap.
Done when: the check fails on your defect and passes on the fix; you report the size of an export page and the number of pages for 1,000 tickets; and you explain what a model should do when next_cursor is present.
Hint
copy search's limit and cursor handling; to measure "worst case", create 1,000 tickets in one Env.
Medium 3: Rate limiting (2 hours)
Add a token-bucket limit per user (for example 20 calls per minute) in the HTTP layer, using the injectable clock. Return 429 with a Retry-After header. Write checks for a burst, for refill over time, and for two users not affecting each other. Decide what the client should do on 429.
Done when: the three checks pass using the fake clock (no sleeping), a defect that shares one bucket among all users is caught, and you state in two sentences what the specification asks of servers about rate limits and what your design leaves out.
Hint
the layer to change is protected.py; pass the clock in as world.Clock.
Hard
Hard 1: Use the official Inspector as the second client (3 hours)
Run the Inspector's command-line mode against stdio_main.py (it needs Node 22.19 or newer): list the tools with --strict, call tickets_search, and call ticket_close on a closed ticket. Record the commands, outputs and exit codes. Then switch on the stdout_banner defect and run it again. (I did not run this, so your results are the ground truth.)
Done when: you show the three exit codes (including the one for the tool error), what the Inspector reports for the banner defect compared with this suite's message, whether --strict found any schema problem in the 8 tools, and what you would add to your own checks because of what it found.
Hint
--cli is the first token on the line, then the server command; the isError exit code is the one that a shell && chain stops on.
Hard 2: Port to an official SDK and keep the suite (4 hours)
Re-implement the adapter layer with an official SDK (for example FastMCP) and keep db.py, service.py and the layer checks. Run the conformance, snapshot and transport layers against the SDK-based server. (I did not run this.)
Done when: you list every check that failed at first and say whether the SDK or your check was wrong, say which of my "choices" in section 7 the SDK made differently, regenerate and review the snapshot, and give an honest estimate of how many lines you deleted.
Hint
keep DeskError and map it to the SDK's error result yourself; start with the transport layer, which will tell you most.
Hard 3: A real model on the same tasks (4 hours)
Replace the scripted reader with a real model through a host or SDK you control. Use the eight goals and eight failure scenarios, written afresh by someone other than the person who wrote the descriptions. Run each 10 times with the three error styles. (I did not run this.)
Done when: you report selection accuracy and recovery for each error style with the model's name and the date, compare them with the stand-in's 8 of 8, 3 of 8 and 3 of 8, say which of the stand-in's predictions held and which did not, and state what that means for how much to trust section 3's table.
Hint
freeze the goals and failure scenarios before you read any model output, and use a fresh database per run.