Structured output

Why products need objects, not vibes

A model that "usually" returns JSON is not shippable. Downstream code expects parseable fields — dates, enums, nested lists — every time. When the model drifts into markdown fences, trailing commentary, or a missing key, your pipeline crashes or silently corrupts data.

Structured output is the discipline of constraining generation so the model emits data your application can validate before use. This lesson covers three layers that work together:

1. Prompt contract — tell the model the exact schema in the system message.
2. Provider features — JSON mode, response_format, or tool/function calling that nudges syntax.
3. Application validation — Pydantic or jsonschema that rejects bad payloads before side effects.

Skip any layer and you will debug production incidents at 2 a.m.

Callout — validate after every call: Never trust model output because it "looked fine" in testing. Parse, validate, and handle validation errors as first-class outcomes — retry, fallback, or escalate to a human.

Prompt-level structure

Start with an explicit schema in the system prompt. For small objects, inline JSON Schema or a TypeScript-like shape works:

Return ONLY valid JSON matching this schema:
{
  "title": string,
  "due_date": "YYYY-MM-DD or null",
  "assignee": string or null,
  "priority": "low" | "medium" | "high"
}
No markdown, no preamble, no trailing text.

Few-shots should demonstrate edge cases: null dates, empty assignee, ambiguous priority resolved consistently. The model learns format from examples as much as from instructions.

Common failure modes to address in prompts:

• Wrapping JSON in ```json fences → forbid fences explicitly.
• Adding "Here is the JSON:" preamble → repeat "ONLY valid JSON."
• Hallucinating enum values → list allowed values verbatim.

Provider mechanisms

Major APIs offer syntax-aware generation:

OpenAI supports response_format: { "type": "json_object" } and structured outputs tied to JSON Schema on supported models. Anthropic supports tool use with input schemas. Local servers vary — assume prompt + validation as the portable baseline.

Provider features reduce syntax errors; they do not guarantee semantic correctness. A valid JSON object can still contain a wrong date or invented assignee. That is why application validation stays mandatory.

When choosing between JSON mode and tool calling for extraction:

• JSON mode: Best for batch extractors and ETL-style pipelines.
• Tool calling: Best when extraction is one step in a multi-tool agent loop (agents module).

For this lesson, implement JSON mode or raw completion + parse — keep the validation layer identical either way.

Validation with Pydantic

Pydantic models turn JSON into typed Python objects and produce clear errors:

from pydantic import BaseModel, Field
from typing import Literal
from datetime import date

class ActionItem(BaseModel):
    title: str = Field(min_length=1)
    due_date: date | None = None
    assignee: str | None = None
    priority: Literal["low", "medium", "high"] = "medium"

def parse_action_item(raw: str) -> ActionItem:
    import json
    data = json.loads(raw)  # raises JSONDecodeError
    return ActionItem.model_validate(data)  # raises ValidationError

Wrap calls in a small retry policy: on validation failure, optionally re-prompt with the error message ("You returned priority 'urgent'; allowed values are low, medium, high"). Cap retries at two — infinite retry loops burn budget.

For non-Python stacks, use jsonschema with the same schema document shared across services.

Callout — failing fixtures are features: Maintain fixtures/invalid/ with malformed model outputs your parser must reject. Regression tests on bad JSON matter as much as happy-path cases.

Building a validated extractor

Your micro-project is a small extractor service — input text in, validated object out. Architecture:

m4/extractor/
  schema.py          # Pydantic models
  prompt.md          # system + schema instructions
  extract.py         # call model, parse, validate
  fixtures/
    valid/*.json     # expected inputs + gold outputs
    invalid/*.txt    # bad model outputs for unit tests
  test_extract.py

Pipeline steps:

1. Load prompt pack and user content.
2. Call chat API with JSON-oriented settings.
3. Strip fences if present (defensive normalize step).
4. json.loads → Model.model_validate.
5. Return object or structured error { "error": "validation", "details": ... }.

Log raw model text even on failure — you will tune prompts using those logs.

Handling partial extraction

Real documents exceed context or contain multiple entities. Patterns:

• One-shot full doc: Works for short emails; breaks on long PDFs.
• Chunk then merge: Extract per chunk; dedupe in code (RAG module covers chunking deeply).
• List wrapper schema: { "items": [ ... ] } with max length in prompt.

Pick one scope for the micro-project; document why.

Testing strategy

Unit tests should not call the live API in CI. Structure:

Add at least three invalid fixtures: invalid JSON syntax, wrong enum, wrong type for a field. Your extractor should never throw an unhandled exception on bad model output.

Engineering problem (staff framing)

Products need JSON/schema, not prose. Enforce structure with schemas + repair + validation.

Diagram — Structured decode path

flowchart LR
  Prompt --> Model --> Draft[JSON draft]
  Draft --> Val[Schema validate]
  Val -->|fail| Repair[Repair / retry]
  Repair --> Val
  Val -->|ok| App[Typed object]

Precise definitions & mental model

JSON mode / constrained decoding; Pydantic/Zod validation; tool_call arguments as structured I/O.

Tradeoffs — when to use what

Constrained decode (reliable, provider-specific) vs validate-and-retry (portable).

Failure modes (interview + on-call)

Trailing commas; schema drift; validating only in happy demos.

Production & OSS practices

Contract tests for schemas; version schemas with prompts.

Micro-project: Validated JSON extractor

Ship in m4/extractor/ under your course-portfolio monorepo:

1. Choose a extraction task (meeting notes → action items, job posting → structured fields, support email → category + urgency).
2. Define a Pydantic model and prompt that requests JSON only.
3. Implement extract.py with validation and bounded retry on validation errors.
4. Add ≥5 valid fixtures and ≥3 invalid raw outputs tested without API calls.

Document run instructions and one example failure you observed during development.

Checklist

• Schema defined in code and reflected in prompt
• Invalid outputs handled without crashing
• Unit tests pass without network
• Raw model responses logged for debugging