Constrained decoding (guaranteed JSON)
We'll cover the following
An LLM writes one token at a time. Ask for JSON in the prompt and it is only *likely* to be valid: you still get stray prose, a missing brace or a cut-off object. Constrained decoding makes invalid output impossible instead of unlikely.
At every step the model scores every token in its vocabulary. A checker that tracks a grammar (a JSON schema, regex or CFG) works out which tokens are legal next and sets the score of all the others to minus infinity (a logit mask). The model then samples only from legal tokens.
Worked example: {"age": <number>}
| Text so far | Model wanted | Checker allows | Result |
|---|---|---|---|
| (empty) | Sure! | only { | forced to { |
| { | name | only "age" | forced to "age" |
| {"age": | thirty | only digits | forced to a digit, e.g. 4 |
| {"age": 4 | (anything) | digits, . or } | model picks 2, then } |
The model chose the value (42). The grammar fixed the shape.
Prompt and retry vs constrained decoding
| Prompt + validate + retry | Constrained decoding | |
|---|---|---|
| Valid syntax | Likely, not guaranteed | Guaranteed |
| Failure cost | Extra full calls (latency and money) | None |
| Setup | Nothing | Needs an engine or API that supports it |
What it does not guarantee
- Truth. {"age": 7} for a 40-year-old is valid and wrong. You still need evals.
- Reasoning quality. If the answer field comes first, the model cannot think before answering. Put a
reasoningfield before theanswerfield in the schema. - Completion. If max_tokens cuts the output mid-JSON, it is still broken. Check the finish reason.
- Tokens are not characters. Grammars work on characters but the model emits multi-character tokens, so engines (Outlines, XGrammar, llguidance) precompute the legal tokens per grammar state. Self-hosted: vLLM and SGLang guided decoding. Hosted: "structured outputs" / JSON-schema mode.
Check yourselfThe model returns valid JSON matching your schema, but the value is wrong. Did constrained decoding fail?
No. It guarantees shape, not truth. Wrong values are an eval or grounding problem.
Summary: say it in the interview
- "I do not rely on the prompt for structure. I use schema-constrained decoding so the syntax is guaranteed, put reasoning before the answer field, check for truncation, and still run semantic evals."