1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
How does constrained decoding guarantee structured outputs such as JSON?
30-second answerSay your answer out loud first, then reveal.
How it works
- Convert the JSON schema (or regex or grammar) into an automaton.
- Track the automaton's state as tokens are generated.
- Precompute, for each state, which vocabulary tokens are allowed (tricky, because tokens span multiple characters).
- Apply a mask to the logits, then sample normally among valid tokens.
Example. After
{"status": ", with status an enum of ["open","closed"], only tokens that continue towards open or closed are allowed.Benefits: a 100% syntactically valid, schema-conformant output; no retry loops for parse errors.
Caveats
- Valid ≠ correct: the content can still be wrong or hallucinated within a valid structure.
- Quality impact: forcing structure can push the model into low-probability tokens. Field order matters (put a
reasoningfield before theanswerfield so the model can think first). - Performance: grammar compilation and masking overhead (modern engines make this small).
- Complex schemas (deep recursion, many optional fields) may not be fully supported.
Alternative: JSON mode (valid JSON, any shape) + schema validation + retry, which is simpler but not guaranteed.
Related
- Previous: Q30. After fine-tuning on your domain data, the model got better at your task but worse at general instructions and reasoning. What happened, and how do you fix it?
- Next: Q32. What is speculative decoding, and why does it speed up inference without changing outputs?
- LLM Fundamentals
- LangChain
Little by little, you're building something great.