Multimodal agents

Agents that see still need tools and stops

Multimodal models accept images (screenshots, photos, diagrams) alongside text. Multimodal agents combine vision with tool loops — e.g. screenshot → identify UI element → call automation tool. Vision errors are first-class failure mode: misread button label, miss small text, confuse chart legend.

Architecture parallels text agents; evals must separate perception from action.

Vision + tool loop shape

User image + goal → VLM describes relevant regions
                  → Plan tool calls (click, type, query DB)
                  → Tool results → final answer

Stop conditions unchanged: max steps, verify gate (Airbnb lesson applies to screenshot verify).

Choose API with vision (GPT-4o, Claude 3, Gemini) or open VLMs — document model id in eval logs.

Callout — Perception ≠ reasoning: Log perception_error when ground truth image label differs from model description — even if final answer accidentally correct.

Tool design for visual tasks

Examples:

• crop_region(bbox) — focus next call
• ocr_region(bbox) — delegate text to specialized model
• click_ui element_id — in sandbox automation fixture
• compare_screenshot baseline — verify step

Return structured JSON from tools; VLMs parse poorly over raw pixel dumps in text.

Small visual eval set

Create ≥12 cases in evals/vision/:

• Chart reading (value at bar X)
• UI screenshot (which button enables export)
• Document photo (extract table cell)
• Adversarial: low contrast, cropped text

Labels: expected_answer, optional required_regions.

Metrics:

• Perception accuracy — description matches label
• Task success — correct action/answer after tools
• Tool appropriateness — called OCR when needed?

Logging perception failures separately

Span attributes:

{"perception_check": "fail", "expected": "Submit button blue top-right", "model": "..."}

Component eval report — do not hide vision miss behind E2E judge pass.

Safety and privacy

Images may contain PII — redact before sending to vendor; warn users on upload; retention TTL on image blobs.

Image preprocessing pipeline

Resize, compress, strip EXIF (GPS leaks) before sending to VLM — preprocessing span in trace. Document max megapixels accepted at API boundary.

Fallback when vision fails

If perception check fails, ask user for text description or manual crop tool — graceful degradation path in agent policy skill.

Multimodal cost

Vision tokens often dominate bill — log image token equivalent in cost_log; compare multimodal route vs human OCR + text-only agent for ROI note in eval report.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Multimodal agents add tools for vision/audio with new injection and cost profiles.

Diagram — Multimodal agent

flowchart LR
  Sense[Vision/audio tools] --> Agent --> Act[Actuation]

Precise definitions & mental model

Toolified sensors, grounding, sync issues.

Tradeoffs — when to use what

End-to-end multimodal model vs tool collage.

Failure modes (interview + on-call)

Blind trust OCR; huge media in context.

Production & OSS practices

Media quotas; modality evals; HITL for actuation.

Micro-project: Vision + tool agent

Ship:

1. Agent accepting image + text; ≥2 vision-related tools.
2. Eval set ≥12 with perception + task metrics.
3. eval report separating perception vs task failures.
4. One trace with logged perception_failure.
5. Capstone note: multimodal in scope or explicit non-goal.

Acceptance: report shows at least one case failed perception but would pass naive E2E text-only check.

Checklist

• Vision + tool agent runnable
• Visual eval set ≥12 labeled
• Perception failures logged separately
• Component metrics in eval report
• Image privacy note in README