Constrained decoding (guaranteed JSON)

Lesson 4.4 · 4 min read · by Vinay C

We'll cover the following

An LLM writes one token at a time. Ask for JSON in the prompt and it is only *likely* to be valid: you still get stray prose, a missing brace or a cut-off object. Constrained decoding makes invalid output impossible instead of unlikely.

At every step the model scores every token in its vocabulary. A checker that tracks a grammar (a JSON schema, regex or CFG) works out which tokens are legal next and sets the score of all the others to minus infinity (a logit mask). The model then samples only from legal tokens.

Worked example: {"age": <number>}

Text so farModel wantedChecker allowsResult
(empty)Sure!only {forced to {
{nameonly "age"forced to "age"
{"age": thirtyonly digitsforced to a digit, e.g. 4
{"age": 4(anything)digits, . or }model picks 2, then }

The model chose the value (42). The grammar fixed the shape.

Prompt and retry vs constrained decoding

Prompt + validate + retryConstrained decoding
Valid syntaxLikely, not guaranteedGuaranteed
Failure costExtra full calls (latency and money)None
SetupNothingNeeds an engine or API that supports it

What it does not guarantee

  • Truth. {"age": 7} for a 40-year-old is valid and wrong. You still need evals.
  • Reasoning quality. If the answer field comes first, the model cannot think before answering. Put a reasoning field before the answer field in the schema.
  • Completion. If max_tokens cuts the output mid-JSON, it is still broken. Check the finish reason.
  • Tokens are not characters. Grammars work on characters but the model emits multi-character tokens, so engines (Outlines, XGrammar, llguidance) precompute the legal tokens per grammar state. Self-hosted: vLLM and SGLang guided decoding. Hosted: "structured outputs" / JSON-schema mode.
Check yourselfThe model returns valid JSON matching your schema, but the value is wrong. Did constrained decoding fail?

No. It guarantees shape, not truth. Wrong values are an eval or grounding problem.

Summary: say it in the interview

  • "I do not rely on the prompt for structure. I use schema-constrained decoding so the syntax is guaranteed, put reasoning before the answer field, check for truncation, and still run semantic evals."