Part 4 of 6 · 3 min read
Constrained decoding and other controls
How a model can be made to always produce valid JSON, and the other knobs you will meet.
Asking "reply with JSON only" works most of the time. Structured output features make it work every time by constraining decoding itself: at each step, tokens that would break the format are given zero probability.
Constrained (structured) decoding
Filtering the next-token candidates so that the finished text is guaranteed to match a grammar or a JSON schema, for example by removing any token that would make the JSON invalid at that point.
This is the mechanism behind "JSON mode" and schema-enforced tool calls. It guarantees the shape of the output, not that the content is true. You still validate with Pydantic (Level 0, module 1), and Level 2 shows how to combine them.
Other ideas you will meet
- Stop sequences: strings that end generation early (for example
</answer>). - Max tokens: a hard limit on output length. A cut-off answer is a common bug, so check the finish reason in the response.
- Repetition and frequency penalties: reduce the logit of tokens already used.
- Best-of-n and self-consistency: sample several answers and pick the best or the most common one, trading cost for accuracy. In a quick simulation (a pretend model that is right 60% of the time and wrong in one of three ways otherwise), majority vote over n answers gave 61% for 1, 71% for 3, 81% for 5 and 92% for 9. It only works because the wrong answers differ. If the model made the same mistake every time, voting would not help.
- Beam search: keeps several candidate sequences at once. Common in translation, rare in chat models.
For the curious How constrained decoding can be built
A program tracks where the output is in the grammar (for example, "inside a string, after a key"). At each step it works out which tokens could legally come next, sets the logits of all others to minus infinity, and then samples as usual. Exercise Hard 3 builds a character-level version for a simple pattern.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Low temperature for tools and extraction | Set temperature near 0 for steps whose output a program reads, so loops are stable. |
| Schema-enforced output | Use the provider's structured-output or tool-call feature, then validate with Pydantic anyway. |
| Properties in tests | Tests check that the answer parses and contains the right facts, never an exact string from the model. |
| Record and replay | Real model responses are saved once and replayed in unit tests. |
| Checking the finish reason | Code looks at why generation stopped (length, stop sequence, tool call) and handles a cut-off. |
| Self-consistency for hard questions | For questions where accuracy matters more than cost, several samples are voted on. |
Draft: to be checked against the current documentation of the providers the course uses (parameter names, which models ignore sampling settings, structured-output limits) before this page goes live.
Interview angle
Answer in your head first, then open it.
What does temperature do?
It rescales logits before softmax: low sharpens, high flattens; it never changes the ranking.
Top-k versus top-p?
Top-k keeps a fixed count; top-p keeps a fixed probability mass, so it adapts to how confident the model is.
Is temperature 0 deterministic?
Mostly, but not guaranteed: floating-point order, batching and model updates can change outputs.
How do you get reliable JSON from a model?
Schema-constrained decoding or tool calling, then validation. Prompts alone are not a guarantee.
How would you test code that calls an LLM?
Assert on properties, use recorded responses for unit tests, and run a separate evaluation set.