TL;DR: Schema-constrained decoding makes malformed output impossible at the token level, but it only guarantees shape, not truth. Validate-and-retry works with any model but spends tokens and latency on failures. For most production pipelines the right answer is both: constrain where your provider or inference engine supports it, then validate business rules and retry a bounded number of times on whatever the schema cannot express.
If you have ever had a nightly extraction job fall over because a model wrapped its JSON in a markdown fence, dropped a closing brace, or invented a field name, you already know why this topic matters. Downstream code expects a contract. Language models produce text. The gap between the two is where a surprising share of LLM pipeline incidents live.
There are two broad ways to close that gap. You can constrain generation so the model can only emit tokens that keep the output valid against a schema, or you can validate and retry: let the model write freely, check the result, and ask again with the error attached when it fails. This article compares them as an engineer would before committing a design: what each guarantees, what each costs, where each breaks, and how to combine them. It is written for teams choosing an approach for API-backed or self-hosted models, and it cites vendor documentation throughout so you can verify details against the version you actually run.
Table of Contents
- What "structured output" actually guarantees
- Approach A: schema-constrained decoding
- Approach B: validate-and-retry
- Architecture: layering both
- Trade-off table
- Working code: a bounded validate-and-retry loop
- Common mistakes
- Decision checklist
- FAQ
What "structured output" actually guarantees
Three different things get called "structured output", and mixing them up causes most of the bad design decisions.
Prompted JSON. You ask for JSON in the prompt. Nothing is enforced. Modern models are good at this, but "good" is a probability, and at pipeline scale the tail matters.
JSON mode. The provider guarantees syntactically valid JSON. OpenAI's documentation is explicit about the limit: while JSON mode and Structured Outputs both ensure valid JSON, "only Structured Outputs ensure schema adherence", and the docs recommend Structured Outputs over JSON mode when possible (OpenAI, Structured outputs). Valid JSON with the wrong keys is still a production bug.
Schema-constrained output. The decoder is restricted so the output conforms to a JSON Schema you supply. This is the strongest guarantee available, and it is what the rest of this article means by "constrained decoding".
Whichever you use, keep one distinction in view. A schema can say a field is a string, an integer, or one of five enum values. It cannot say the invoice total equals the sum of the lines, that the cited quote actually appears in the source document, or that the diagnosis code exists. OpenAI's own guidance tells applications to still handle refusals, incomplete responses and API errors, and to "validate any business rules that the schema does not express." Constraint buys you structure. Correctness is still your job.
Approach A: schema-constrained decoding
How it works
At each generation step a language model produces a probability distribution over its vocabulary. Constrained decoding computes, from the schema (compiled into a grammar or state machine), which tokens are legal next, and masks the rest before sampling. The model can still choose among legal tokens, so quality depends on the model, but it cannot emit a token that would break the grammar.
You meet this in two places: hosted APIs that run it for you, and inference engines you operate yourself.
Hosted APIs
Both major providers now expose it as a first-class feature.
OpenAI describes Structured Outputs as constraining "completed, non-refusal model output" to your JSON Schema. For structuring a reply to the user you pass a json_schema format with strict set to true; for connecting the model to tools you use function calling. The docs' own comparison shows the enabling syntax as text: { format: { type: "json_schema", "strict": true, "schema": ... } } in the Responses API (OpenAI).
Anthropic's Claude API offers two features that can be used independently or together: JSON outputs via output_config.format, and strict tool use via strict: true on a tool definition. The documentation states the tool input strictly follows the input_schema and the tool name is always valid (Anthropic, Structured outputs; strict tool use). A minimal call, following the shape shown in Anthropic's docs:
import json
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="<your-model-id>",
max_tokens=1024,
messages=[{"role": "user", "content": "Extract the contact: John Smith ([email protected]) wants an Enterprise demo next Tuesday at 2pm."}],
output_config={
"format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"plan_interest": {"type": "string"},
"demo_requested": {"type": "boolean"},
},
"required": ["name", "email", "plan_interest", "demo_requested"],
"additionalProperties": False,
},
}
},
)
text = next(b.text for b in response.content if b.type == "text")
contact = json.loads(text)Both vendors also provide SDK helpers that take a Pydantic model and return a parsed object, which is the ergonomic route for Python teams.
Self-hosted inference
If you run open-weight models, constraint support lives in the serving engine. vLLM's documentation says it supports structured outputs using xgrammar or guidance as backends, selectable with --structured-outputs-config.backend (default auto), and the OpenAI-compatible server accepts constraints such as json, regex, choice, grammar and structural_tag as extra parameters (vLLM docs). Two details from that page matter in practice. First, changing backends "can change which tokens are allowed and therefore the generated output", so treat a backend switch as a change that needs regression testing. Second, the older guided_json family of fields was removed in vLLM v0.12.0 in favour of the structured_outputs parameter, so tutorials written before that point may not run on current versions. Check the docs for the version you deploy.
What constraint costs and where it bites
The documented limits are worth reading before you design around "guaranteed" output.
- Schema subset. Providers support a subset of JSON Schema. Anthropic's docs state
additionalPropertiesmust be false for objects, and thatminimum,maximum,minLengthandmaxLengthare unsupported; its SDKs strip those constraints and move them into field descriptions, while the SDK validation helpers still check them client-side. Anthropic also documents limits on strict tools per request (20), optional parameters (24) and union-typed parameters (16) (Anthropic). OpenAI notes some schema features "are unavailable either for performance or technical reasons". Read the supported-schema section of your provider's page rather than assuming Pydantic's full output will pass. - First-request latency. Anthropic states that the first use of a schema adds latency while the grammar compiles, compiled grammars are cached for 24 hours from last use, and changing the schema structure invalidates the cache. A schema that varies per request therefore pays compilation repeatedly. Prefer a small set of stable schemas.
- Refusals and truncation break the guarantee. Anthropic documents that a response with
stop_reason: "refusal"may not match the schema because the refusal message takes precedence, and thatstop_reason: "max_tokens"can leave incomplete output. OpenAI's docs similarly say refusals and incomplete responses do not necessarily follow your schema. Always check the stop reason or status before parsing. - Enum casing. Anthropic notes enum and const strings may come back with different capitalisation than the schema, and recommends case-insensitive comparison.
- Compatibility. Anthropic lists citations as incompatible with JSON outputs (a 400 error), and message prefilling as incompatible too.
- Privacy of the schema itself. Anthropic notes only the schema is cached (up to 24 hours) under its zero-data-retention terms, and for HIPAA-eligible use "PHI must not be included in JSON schema definitions" such as property names, enums or patterns. Regulated teams should treat schema text as a data-handling surface.
The most important point is semantic. A model forced into a schema will still fill required fields even when the source has no answer. If date_of_birth is required and the document lacks one, the decoder will help the model produce something that looks like a date. Make fields nullable or add an explicit "not_found" option so the legal output space includes honesty.
Approach B: validate-and-retry
How it works
You call the model, parse the output, validate it with a schema validator (typically Pydantic) plus any business rules, and if it fails, send the error back and ask for a correction. Libraries package this. Instructor's documentation describes the loop as capturing validation errors, formatting them as feedback, adding them to the prompt context and asking the model to try again (Instructor, retry mechanisms).
Where it shines
- Any model, any provider. No dependency on a vendor feature or serving-engine backend.
- Arbitrary rules. A Pydantic validator can check that a quote appears in the source, that line items sum to a total, or that an ID exists in your database. These are checks no JSON Schema can express.
- Actionable errors. The model sees exactly what was wrong, which often fixes the problem in one more turn.
Where it hurts
- Cost and latency are variable. Every retry is another full call. Instructor's retry documentation explains that
max_retrieson the library call counts validation retries after the first attempt, and separately that the underlying SDK's ownmax_retriesgoverns HTTP-level retries; the page warns against simply multiplying the two to predict call counts (Instructor, retrying). Budget for the worst case, not the average. - Some errors never converge. If the input genuinely lacks the information, the model cannot repair it by being told off. The same Instructor docs recommend bounded stop conditions and not unbounded retries, with a suggested 2 to 3 attempts for validation errors.
- The tail is real. When a pipeline processes large volumes, the rare failure that exhausts retries must have a defined destination: a dead-letter queue, a human review step, or a safe default.
Architecture: layering both
The practical pattern is not either/or. Constrain first so structural failures disappear, then validate semantics and retry only for what remains.
flowchart TD
A[Source document or user input] --> B[Prompt + stable schema]
B --> C{Provider or engine supports constrained output?}
C -- Yes --> D[Constrained generation]
C -- No --> E[Free generation, JSON requested]
D --> F{stop_reason / status OK?}
E --> G[Parse JSON]
F -- refusal or truncated --> H[Route: raise max tokens, rephrase, or escalate]
F -- OK --> G
G --> I[Schema validation, Pydantic]
I --> J[Business-rule validation]
J -- pass --> K[Downstream system]
J -- fail --> L{Attempts left?}
L -- Yes --> M[Retry with error feedback]
M --> B
L -- No --> N[Dead-letter queue / human review]A few design notes that follow from the diagram.
Keep schema and validator in one source of truth. Define the Pydantic model once and derive the JSON Schema from it. Where your provider strips unsupported constraints, the client-side validator still enforces them, so nothing is lost; it just moves from decode time to validate time.
Make retries cheap and rare. With constraint on, retries are for semantic failures only. That keeps the retry rate low enough that a cap of two or three attempts is rarely hit.
Log every failure with its reason. Failure categories (refusal, truncation, schema violation, rule violation, retry exhausted) are your best signal for prompt changes, schema redesign or model selection.
Treat the schema as a versioned interface. Changing it invalidates provider-side grammar caches and changes downstream contracts. Version it like an API. This is the same discipline described in our write-up on API-first architecture for fintech startups.
Trade-off table
| Dimension | Schema-constrained decoding | Validate-and-retry |
|---|---|---|
| Structural validity | Enforced at decode time (excluding refusal/truncation) | Probabilistic; fixed after failure |
| Semantic and business rules | Not expressible beyond schema keywords | Fully expressible in code |
| Provider/model dependency | Needs vendor feature or engine backend | Works with any model |
| Latency profile | Possible first-request grammar compile; otherwise predictable | Variable; each retry is a full call |
| Cost profile | Predictable; possible extra input tokens for injected instructions | Grows with failure rate |
| Schema flexibility | Limited to a supported subset | Anything your validator can check |
| Failure handling | Must still handle refusal and truncation | Must handle retry exhaustion |
| Portability | Differences across providers and backends | High |
| Best fit | High-volume extraction, tool arguments, agent plumbing | Rule-heavy outputs, multi-provider setups, models without constraint support |
Working code: a bounded validate-and-retry loop
The loop below is provider-agnostic. You pass in a call_model function that takes a message list and returns the raw text. It validates with Pydantic, applies a cross-field business rule the schema cannot express, appends the failed output and the error to the conversation, and stops after a fixed number of attempts. It was run locally with a stubbed model to confirm the retry path: a first answer with a lowercase currency code fails, a second with mismatched totals fails the business rule, and a third passes.
import json
from pydantic import BaseModel, Field, ValidationError, field_validator
class Invoice(BaseModel):
invoice_number: str
currency: str = Field(pattern=r"^[A-Z]{3}$")
total: float = Field(gt=0)
line_totals: list[float]
@field_validator("line_totals")
@classmethod
def not_empty(cls, v):
if not v:
raise ValueError("line_totals must contain at least one value")
return v
def extract(call_model, document, max_attempts=3):
messages = [{"role": "user", "content": f"Extract the invoice as JSON.\n\n{document}"}]
last_error = None
for attempt in range(1, max_attempts + 1):
raw = call_model(messages)
try:
invoice = Invoice.model_validate_json(raw)
if abs(sum(invoice.line_totals) - invoice.total) > 0.01:
raise ValueError("line_totals do not sum to total")
return invoice
except (ValidationError, ValueError) as exc:
last_error = exc
messages += [
{"role": "assistant", "content": raw},
{"role": "user", "content": f"That output failed validation:\n{exc}\nReturn corrected JSON only."},
]
raise RuntimeError(f"gave up after {max_attempts} attempts: {last_error}")In production, wrap call_model so it checks the provider's stop reason or status first and raises a distinct exception for refusals and truncation. Those should not be fed back as "please fix your JSON"; a truncated response needs a larger token limit, not a scolding. If you use a library such as Instructor instead of hand-rolling, note its docs flag version differences in where max_retries is configured, so verify against the version you install.
Common mistakes
- Treating "valid JSON" as "correct data". JSON mode and even strict schemas verify shape. Add semantic checks.
- Ignoring stop reasons. Parsing a refusal or a truncated response as if it satisfied the schema is the classic way a "guaranteed" pipeline fails.
- Requiring fields the source may not contain. This invites plausible fabrication. Allow null or an explicit not-found value.
- Generating a new schema per request. On providers that compile and cache grammars, this pays compile latency repeatedly and defeats caching.
- Unbounded retries. A bad input becomes a runaway cost loop. Cap attempts and define the failure path.
- Conflating SDK retries with validation retries. Transport retries and semantic retries are different mechanisms with different budgets.
- Copying a tutorial across versions. Parameter names change, as vLLM's removal of
guided_jsonshows. Pin versions and read the docs for yours. - Putting sensitive data in schema text. Property names, enums and patterns can be cached by the provider; keep identifiers and personal data out of them.
- Case-sensitive enum comparison where the provider documents capitalisation drift.
- No regression suite when switching backends or models. Different decoding backends can produce different outputs for the same schema.
Decision checklist
Use this before you commit.
- [ ] Does your provider or serving engine support schema-constrained output for your model? If yes, enable it as the first layer.
- [ ] Have you read the supported-schema subset and tested your real schema against it, including optional-field and union limits?
- [ ] Is the schema stable enough to benefit from grammar caching?
- [ ] Can every required field genuinely be filled from the input? If not, make it nullable or add a not-found value.
- [ ] Do you check stop reason or status before parsing, with separate handling for refusal and truncation?
- [ ] Are business rules encoded in validators (totals, referential checks, quote-in-source checks)?
- [ ] Is the retry cap set (commonly two or three for validation errors) and is retry-exhaustion routed somewhere visible?
- [ ] Do you log failure category, attempts and token cost per request?
- [ ] Is the schema versioned and covered by a regression set that you re-run on model or backend changes?
- [ ] Have compliance requirements been checked for schema text and prompt retention?
How this applies to real projects
Which approach carries more weight depends on the workload. Extraction pipelines with high volume and simple structure lean on constraint, because retries at scale are costly. Rule-heavy outputs, such as anything feeding a financial or clinical workflow, lean on validation, because the important failures are semantic. Agent and tool-calling systems benefit from strict tool schemas because a malformed tool call is a hard failure for the whole loop. Teams building AI features for regulated sectors can see the same layering in our discussions of AI-driven fraud detection build-versus-buy and HIPAA-compliant AI clinical decision support architecture, where output contracts and auditability matter as much as model quality.
Syslabs works with engineering teams on exactly this layer: designing the contract between an LLM and the systems around it, building evaluation sets, and wiring validation, retries and observability into production pipelines. We do not claim one approach beats the other universally; the right mix depends on your model, volume, and risk tolerance.
Conclusion
Constrained decoding and validate-and-retry solve different halves of the same problem. One makes malformed output nearly impossible; the other lets you enforce what your business actually means. Start with constraint where it is available, keep your schema small and stable, validate semantics in code, cap retries, and give every failure somewhere to go. Then measure failure categories and adjust.
If you are designing or hardening an LLM pipeline and want a second set of engineering eyes, book a meeting with the Syslabs team.
References
- OpenAI. Structured model outputs. https://developers.openai.com/api/docs/guides/structured-outputs
- Anthropic. Structured outputs. https://platform.claude.com/docs/en/build-with-claude/structured-outputs
- Anthropic. Strict tool use. https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use
- vLLM. Structured outputs. https://docs.vllm.ai/en/latest/features/structured_outputs/
- Instructor. Retry logic. https://python.useinstructor.com/concepts/retrying/
- Instructor. Retry mechanisms. https://python.useinstructor.com/learning/validation/retry_mechanisms/
