Part 4 of 7 · 1 min read
Constrained decoding
How a model can be made to always produce valid JSON.
Asking "reply with JSON only" works most of the time, but "most of the time" is not good enough for software.
The idea. Imagine typing on a keyboard that only lets you press keys that keep your text valid: if the next character must be a quote, every other key is dead. Structured output features work like that. At each step, tokens that would break the format are given zero probability, so the model cannot produce invalid output.
Constrained (structured) decoding
Filtering the next-token candidates so that the finished text is guaranteed to match a grammar or a JSON schema (a description of the allowed fields and types), for example by removing any token that would make the JSON invalid at that point.
This is the mechanism behind "JSON mode" and schema-enforced tool calls. It guarantees the shape of the output, not that the content is true. A form can be filled in neatly with a wrong answer. So you still check the content with Pydantic (Level 0, module 1), and Level 2 shows how to combine the two.
For the curious How constrained decoding can be built
A program tracks where the output is in the grammar (for example, "inside a string, after a key"). At each step it works out which tokens could legally come next, sets the logits of all others to minus infinity, and then samples as usual. Exercise Hard 3 builds a character-level version for a simple pattern.