Core Concepts

Multimodal basics

Images, audio, and documents into LLMs — when modality helps, how APIs expose it, and failure modes (OCR, resolution, cost).

40 min

Also in guided path: Multimodal in → text

Beyond text-only

Modern frontier models accept images (and sometimes audio) alongside text. Product pattern: vision for documents/UI screenshots; keep structured outputs for downstream systems (Structured outputs).

Multimodal is not “add a picture and quality magically rises.” It is another input channel with its own cost, failure modes, and eval needs.

flowchart LR
  Img[Image / PDF page] --> Enc[Model vision encoder]
  Text[Instructions] --> LLM[Multimodal LLM]
  Enc --> LLM
  LLM --> JSON[Validated JSON / text]

Mental model

  1. Encode pixels (or audio frames) into embeddings the language model can attend to.
  2. Jointly condition on text instructions + modality tokens.
  3. Generate text (or tool calls) as usual.
  4. Validate anything that drives backends.

Providers differ on resolution limits, page counts, and whether PDFs are native vs “render to images.”

When to use multimodal vs OCR pipeline

Approach Pros Cons
End-to-end multimodal Fast to prototype; layout-aware Costly; harder to debug
OCR → text RAG Cheaper retrieval; classic RAG tooling OCR errors; lost layout
Hybrid OCR for search; vision for hard pages More moving parts 
flowchart TD
  Doc[Document / photo] --> Q1{Need search over many docs?}
  Q1 -->|Yes| OCR[OCR / parse → index]
  Q1 -->|No| VLM[Vision LLM on pages]
  OCR --> Q2{Hard layout / handwriting?}
  Q2 -->|Yes| VLM
  Q2 -->|No| TextRAG[Text RAG]
  VLM --> Schema[Structured extract]
  TextRAG --> Schema

Step-by-step product pattern (receipts / forms)

  1. Capture image; compress/resize to provider limits.
  2. Prompt with field list + JSON schema.
  3. Run vision model; validate with Zod/Pydantic.
  4. Bounded repair on schema failures.
  5. Human review queue for low confidence / failed validation.
  6. Log modality token usage separately for cost.

Engineering checklist

  • Resize/compress images; cap pages per request
  • Put untrusted image-derived text behind the same prompt-injection defenses as user text
  • Eval with real scans, not clean screenshots only
  • Log modality token usage separately for cost
  • Prefer schemas over free-form prose for expenses, KYC fields, UI asserts
  • Watch EXIF / PII in images before sending to vendors (Privacy and data for AI)

Tools today (2025–2026)

Layer Examples
Hosted multimodal APIs GPT-4o-class, Claude vision, Gemini
OCR / parse Classic OCR engines, PDF text extractors
Document AI stacks Vendor document AI + LLM cleanup
Open-weight VLMs Vary quickly — check evals on your docs

Failure modes

Symptom Cause Fix
Missed small print Resolution / crop Higher res; tile pages
Wrong totals Arithmetic via pixels Extract fields → compute in code
Works on screenshots only Domain shift Eval on phone photos / scans
Cost spikes Full PDF every turn Cache; OCR index; send hard pages only
Injection via image text Untrusted OCR/VLM text Delimit; treat as data

Tradeoffs

  • Native vision — better layout understanding; $$ and vendor lock-in risk.
  • OCR-first — cheaper retrieval; brittle on handwriting/tables.
  • Hybrid — usually wins for large corpora.

Glossary

Term Meaning
VLM Vision-language model
Modality Input type (text, image, audio, video)
OCR Optical character recognition → text
Tile / crop Splitting large pages for resolution

Micro-project

Build receipt/image → JSON expense fields with a schema validator. Include one messy real photo in the golden set.

Multimodal in → text. Pair with Structured outputs and RAG building blocks for document corpora.

Project checklist0/3 done