A fixed set of checkable cases, run against every version, with a gate that blocks regressions.
Regression cases, and comparing versions
Regression case
A fixed input plus a checkable expectation, kept for good. When a prompt changes, every case runs again. A case that used to pass and now fails is a regression.
Case("bill-1","I was charged twice for my invoice",label="billing",max_reason_words=8,tags=("billing",))
Guidelines for a good case set:
Check things code can check: the label, valid JSON, a word limit, a forbidden phrase. Avoid "sounds friendly" unless you build a rubric (Level 9 covers evaluation).
Cover each category and the tricky ones. The case bug-3 ("Checkout is broken and my payment fails") is a deliberate trap that stays red for every version.
Add a case for every bug you fix. The set grows from real failures.
Tag cases so you can see results by group (report.by_tag()).
Keep them separate from the examples in the prompt (module 2). Never test on your own examples.
Give every case a unique id. The comparison matches cases by id, so a duplicate would hide one of them. The runner now refuses duplicate ids.
The check also copes with a reply that is valid JSON but not an object, such as [1, 2] or null. It used to crash with an AttributeError, which would have stopped the whole run on one odd reply.
Compare, don't just score
A single pass rate hides what moved. The comparison gives the review its facts.
Twelve cases against three prompt versions. Real output of the example code (imitation model).
Version
Change
Pass rate
What happened
v1
first version
50%
no other label, no word limit: six cases fail
v2
add other, add word limit
92%
five fixed, none regressed; bug-3 still fails
v3
tidy the label list
75%
two regressions: acct-1, acct-2
v3 looks like a harmless tidy-up, but it dropped the account label by accident. The comparison names exactly which cases broke, which is how a reviewer finds the cause in seconds.
The gate is the rule a pipeline can enforce: a minimum pass rate, and no regressions unless someone explicitly allows them. v1 to v2 passes the gate. v2 to v3 is blocked with two reasons: the pass rate (75%) is below the minimum (90%), and there are two regressions (acct-1, acct-2).
For the curious How a case you have just added is counted
A case that exists in the new report but not the old one counts as a regression if it fails. The comparison cannot tell "newly broken" from "newly added and failing", so it takes the strict view. When you add a case for a bug you have not fixed yet, either fix the bug in the same change, or tag the case as a known failure and allow it explicitly. Do not quietly raise the allowed number of regressions.
Treat failing-but-known cases honestly
bug-3 fails in every version, and the gate still passes v2 because 92% is above 90%. Record known failures on purpose (a tag like known-failure) so the number is honest, and fix them in a later change. Do not delete a case just to turn the report green.
What a prompt test cannot tell you
A pass means the model did the right thing on these inputs, not on all inputs.
A real model can be non-deterministic. Run important cases several times, or use a low temperature (Level 1, module 4), and look at the pass rate.
A prompt that passes on one model may not pass on another, so store the model name with the results.
Cases drift, because real traffic changes. Sample real inputs into the case set every so often.
Twelve cases is a demonstration. One case is more than 8 percentage points, so small differences are noise. Use more cases for real decisions (Level 9).
What you will use daily at work
Practice
What it looks like
Prompts in the repository
Prompt files sit next to the code, and changes go through pull requests.
A test job in CI
A pipeline runs the case set for every prompt change and applies the gate.
A prompt registry or tracing tool
Each production call is logged with the prompt version, so a bad reply can be traced to the exact text.
Cases mined from production
Replies that users mark as bad become new cases.
Draft: confirm which prompt-management and evaluation tools your team uses, and how they record versions, before relying on this.
Interview angle
"How do you test a prompt change before shipping it?" Answer with the loop: prompts live in version control. A fixed case set has code-checkable expectations. Run old and new, compare case by case, and block on regressions. Add a case for each production failure, and watch real traffic after release. Mention non-determinism and what you do about it.