Multimodal in → text

Multimodal is another I/O path

Vision-capable models accept images alongside text in the same chat API you already use. The product pattern is unchanged: construct messages, call the model, validate structured output. What changes is the content block format — images arrive as base64 blobs, URLs, or file IDs depending on provider — and the failure surface expands to blur, glare, cropping, and ambiguous handwriting.

Treat multimodal features as a pipeline branch, not a separate product category:

Image upload → preprocess (resize, rotate) → vision model → JSON validation → ledger DB

Same engineering habits apply: prompt packs, versioning, golden fixtures — but fixtures now include PNG/JPG files and expected JSON sidecars.

Callout — pixels are untrusted input: Users can upload adversarial images, screenshots of instructions, or unrelated photos. Scope the system prompt narrowly and validate extracted fields before financial actions.

Message format for images

OpenAI-style APIs use content arrays:

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "Extract expense fields from this receipt."},
        {
            "type": "image_url",
            "image_url": {"url": "data:image/jpeg;base64,..."}
        }
    ]
}]

Anthropic and Google use analogous structures with different keys. Abstract behind a helper:

def user_message(text: str, image_path: pathlib.Path) -> dict:
    b64 = base64.standard_b64encode(image_path.read_bytes()).decode()
    ...

Keep image bytes out of git for real user data; commit only synthetic or redacted sample receipts in fixtures/images/.

Receipt → expense JSON task

Receipt extraction is an ideal teaching task because fields are constrained and errors are measurable:

Reuse your Pydantic extractor from lesson 4.2. The vision model replaces OCR + LLM text parsing in one step — but you still validate totals and currency in code.

System prompt essentials:

• List exact JSON schema.
• Instruct: "If a field is unreadable, use null; do not guess."
• Specify default currency only when locale is known.

Preprocessing that actually helps

Before sending images to the API:

Resize large photos to max dimension ~2048px — sufficient detail, lower upload latency.

Auto-rotate using EXIF orientation so text is upright.

Crop optionally when UI provides a scanner overlay (mobile apps); skip aggressive crop in batch pipelines.

Format: JPEG for photos, PNG for screenshots. Avoid sending 10 MB RAW files.

Log preprocessing parameters per run — they are part of reproducibility.

Failure modes to document

Build a failure gallery in fixtures/hard/:

Run your extractor on each hard image and record outcomes in FAILURES.md. Interviewers and future teammates learn from documented failure more than from demo happy paths.

Callout — eval with real scans: Synthetic crisp receipts overstate accuracy. Include at least two messy phone photos in your golden set.

Cost and latency notes

Vision calls cost more tokens than text-only — image tiles convert to token charges on many providers. Batch offline extraction for expense reports; do not block UI on synchronous multi-image uploads without progress indicators.

Cache results keyed by image hash + prompt version so re-uploads do not re-spend.

Batch vs interactive extraction

Two product shapes drive different architectures:

Interactive (user uploads one receipt): Optimize for latency — single image, streaming optional, show spinner with preprocessing step visible. Cache by image hash for accidental re-submission.

Batch (nightly expense folder): Optimize for throughput — queue jobs, rate-limit API calls, write results to CSV/DB. Failures go to dead-letter queue with image path for human review.

For batch, add a confidence or needs_review boolean in schema when any field is null or totals do not reconcile (subtotal + tax ≈ total within epsilon). Downstream workflows route low-confidence rows to accountants instead of auto-posting.

Vision model choice matters: smaller vision models cost less but fail on faded thermal paper; document which model your eval used so upgrades are intentional.

Privacy and retention

Receipts contain PII and payment hints. For the course project, use fake or redacted samples. In production:

• Encrypt at rest.
• Set retention TTL.
• Avoid logging full base64 in plaintext telemetry — log hash + extracted fields only.

Engineering problem (staff framing)

Images/audio expand threat + cost surface. Tokenize media, bound size, eval modality slices.

Diagram — Multimodal input path

flowchart LR
  Img[Image/Audio] --> Enc[Provider encode]
  Enc --> Ctx[Joint context]
  Txt[Text] --> Ctx --> Model --> Out[Text out]

Precise definitions & mental model

Vision tokens, resolution tradeoffs, safety on image inputs.

Tradeoffs — when to use what

High-res detail vs token cost/latency.

Failure modes (interview + on-call)

Huge images; prompt injection via screenshots; no modality-specific evals.

Production & OSS practices

Max bytes, virus scan, strip metadata, separate rate limits.

Micro-project: Receipt → expense JSON

In m4/receipt_extractor/:

1. Implement a script that accepts an image path, calls a vision-capable chat model, returns validated expense JSON.
2. Include ≥5 fixture images with gold JSON sidecars.
3. Document ≥3 failure modes you observed with screenshots or paths in FAILURES.md.
4. Wire prompt version logging consistent with lesson 4.3.

Optional stretch: CLI flag for --dry-run that prints raw model JSON before validation.

Checklist

• Images sent via documented API format
• Output validated with Pydantic (or equivalent)
• Hard fixtures included beyond pristine samples
• No real user receipts committed to git