Part 5 of 7 · 2 min read
Measure it, and work like a professional
The one number that tells you whether this works, and the habits teams add on top.
Measure it
The target for Level 2 is at least 95% valid structured output on a test set of about 50 cases. The function valid_rate(call, inputs, model, max_attempts) returns the share of inputs that end with a valid object.
Measure it twice:
- With
max_attempts=1, to see how good the model is on its own. - With
max_attempts=3, to see what your users actually get.
In the tests, an imitation model that fails about half the time on its first try reaches above 95% once it is given a retry budget. The gap between the two numbers is the value of your retry loop. Also keep a list of inputs that still fail after retries. They are the best material for improving your prompt.
Working like a professional
The method above works with any model. In a real project you will usually combine it with provider features and libraries.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Use the provider's structured output or forced tool call | Pass your schema to the API so the shape is guaranteed, then still validate in your code. |
| Keep one Pydantic model per output shape | The same class is the schema you send and the check you run. |
| Put a version on the schema | Name fields carefully. Adding an optional field is safe; renaming one breaks every stored reply. |
| Log failures with their kind | Count truncated, schema, bad_json separately, and alert when a rate jumps. |
| Keep a regression set | Fifty or so real inputs with known answers, run on every prompt or model change. |
| Use a library when it fits | Several open-source libraries wrap this retry-and-validate pattern for you. Know the pattern first so you can debug them. |
Draft: to be checked against the current documentation of your chosen provider before this page goes live.
Interview angle
Answer in your head first, then open it.
How do you make model output reliably parseable?
Give a schema, use the provider's constrained or structured output where it exists, extract and validate with a typed model, and retry with the error message under a cap.
JSON mode versus a schema-constrained output?
JSON mode ensures valid JSON; a schema-constrained mode also restricts the structure to your schema. Neither guarantees the content is right.
What do you do when parsing fails?
Classify the failure. Fix harmless slips in code, send content errors back to the model with the exact message, cap the attempts, then escalate.
Why not fix a bad enum value yourself?
A guessed value looks valid and hides the error. Ask the model again, or reject it.
How do you know your structured-output prompt is good?
Measure the valid rate and the correctness rate on a fixed test set.