Core Concepts

Structured outputs

JSON schemas, tool calls, and validation loops — making LLMs produce machine-checkable results.

40 min

Also in guided path: Structured output

Why free-form text is not an API

Chat UIs love prose. Product backends need objects: { "intent": "refund", "order_id": "..." }. Structured outputs close that gap with schemas, constrained decoding, or tool/function calling.

If you paste model text into JSON.parse without a contract, you will ship intermittent outages dressed as “the model was creative today.”

flowchart LR
  User[User / upstream] --> Prompt[Prompt + schema]
  Prompt --> Model[LLM]
  Model --> Parse[Parse / validate]
  Parse -->|ok| App[Application]
  Parse -->|fail| Retry[Repair loop]
  Retry --> Model

Mental model: three layers

  1. Contract — JSON Schema / Pydantic / Zod as the source of truth.
  2. Generation — prompt, JSON mode, constrained decoding, or tool args.
  3. Gate — validate before side effects; repair or fail closed.

Never skip the gate. Provider “JSON mode” reduces errors; it does not replace application validation.

Three common techniques

Technique How it works Tradeoff
Prompted JSON “Reply with JSON only” Fragile; easy to break with markdown fences
JSON mode / schema Provider constrains tokens to valid JSON / schema Best when available; still validate
Tool / function call Model emits named args matching a schema Great for agents + APIs; natural for side effects

Prefer schema-constrained generation when the provider supports it. Keep a validator as the source of truth — never trust the model alone.

flowchart TD
  Need[Need machine object] --> Cap{Provider schema support?}
  Cap -->|Yes| Schema[JSON schema / strict tools]
  Cap -->|No| Prompt[Prompted JSON + strict parse]
  Schema --> Val[Validate in app]
  Prompt --> Val
  Val -->|fail| Repair[Bounded repair]
  Val -->|ok| Use[Use object]

Step-by-step: production pattern

  1. Define the schema in code (shared with OpenAPI if you expose it).
  2. Ask the model via the strongest constraint available (strict tools / response_format).
  3. Parse bytes → object; run schema validation.
  4. On failure: log raw output + error paths; repair ≤N times with the validator message.
  5. On persistent failure: fallback (smaller model, rules, human) — never infinite retry.
  6. Only then call payment, ticket, or DB write APIs.

Minimal repair loop (shape)

for attempt in range(3):
    raw = llm.complete(prompt, schema=OrderIntent)
    try:
        return OrderIntent.model_validate_json(raw)
    except ValidationError as e:
        prompt = repair_prompt(prompt, raw, e)
raise StructuredOutputError("exhausted repairs")

How this shows up in agents

Tool arguments are structured outputs. The same rules apply: schema → validate → execute. Final answers that must feed another system should also be schema’d (final_answer tool with typed fields), not free prose. See Agents and ReAct.

Tools today (2025–2026)

Layer Examples
Providers OpenAI structured outputs / tools, Anthropic tool_use, Gemini schema
Validators Pydantic, Zod, JSON Schema, TypeBox
Frameworks Instructor-style wrappers, LangChain output parsers (still validate!)
Serving open-weight Outlines, guidance, llama.cpp grammars, vLLM guided decoding

Exact API names shift; the contract + gate pattern does not.

Failure modes

Symptom Cause Fix
Markdown-wrapped JSON Weak prompting / no constraint Strip fences; prefer schema mode
Extra keys / wrong types Loose schema additionalProperties: false; strict types
Infinite repair No attempt budget Cap retries; fail closed
Valid JSON, wrong business meaning Schema too weak Enums, ranges, cross-field checks
Silent truncation Max tokens too low Raise limit; detect incomplete JSON
Schema drift Prompt says one thing, code another Single schema source in repo

Tradeoffs

  • Strict schemas — reliable integrations; less creative formatting.
  • Prompt-only JSON — fast to prototype; high operational tax.
  • Tools vs response_format — tools shine when calling APIs; response_format shines for pure data extraction.

When to use

  • Any LLM output that drives code paths, DB writes, or UI state machines.
  • Extraction from tickets, emails, receipts (pair with Multimodal basics).
  • Agent tool I/O and final machine-readable answers.

Prefer prose when the only consumer is a human reading a chat bubble.

Glossary

Term Meaning
Constrained decoding Token generation restricted to a grammar/schema
Repair loop Re-prompt with validation errors
Strict tool calling Args must match declared JSON Schema
Fail closed On invalid output, do not execute side effects

Micro-project

Define a schema for {intent, order_id, confidence}. Generate from messy user text; validate; implement a 2-attempt repair; log one forced failure.

Talk to models in the real world — Structured output, chat roles, and prompt versioning with evals. Pair with Prompt engineering and Agents and ReAct.

Project checklist0/3 done