Failure cases, and what to test first: Testing

Why the failure paths hold the real bugs, and the order in which to test an agent project.

The project goal for this level is: every endpoint has a test for its failure case. For the summarise endpoint, the example project covers:

Failure	Expected behaviour	Test
Missing or empty field	422, no upstream call	`test_empty_text_is_rejected_by_validation`
Whitespace-only text	422	`test_whitespace_only_text_is_rejected`
Upstream returns 503	502 to our caller	`test_upstream_failure_becomes_502`
Upstream times out	502	`test_upstream_timeout_becomes_502`

(422 means "the request is wrong" and 502 means "the service behind me failed". You met these codes in module 3.) Here is one of them:

def test_upstream_failure_becomes_502(api):
    client, _ = api(lambda req: httpx.Response(503))
    assert client.post("/summarise", json={"text": "hello"}).status_code == 502

The api fixture runs the real application with its outgoing client replaced by a fake that always answers 503. The test checks that our own endpoint turns that into a clean 502 for its caller. Failure tests are where real bugs live, because the happy path is the one you already tried by hand.

What to test first in an agent project

Order	Test	Why first
1	Input validation and parsing (JSON, tool arguments)	Cheap, and bad input is the most common production problem
2	Each tool function on its own	Tools are normal code; a bug here looks like "the agent is dumb"
3	Error handling and retries with a fake service	You cannot reproduce rate limits on demand with a real one
4	The agent loop with a scripted fake model	Tests your control flow: stops, step limits, tool-call routing
5	A few realistic end-to-end runs	Catch wiring mistakes
6	Evaluations of the real model's behaviour	A separate discipline (Level 9), run less often

A scripted fake model (step 4) is a stand-in for the model that returns pre-written replies, one after another. It is just a function that returns the next item from a list, so your test decides in advance what the "model" will say: a tool call, then a final answer, or a never-ending loop. It is the most useful trick in agent testing. You will build one in the exercises and again in Level 5.