Airbnb — verify-heavy workflows

Verification as a first-class workflow stage

Airbnb engineering writeups on migration and harness-heavy workflows emphasize verify before commit — automated checks that outputs match expectations (DOM structure, screenshots, API responses) beyond "the model said it worked." UI and migration agents fail silently when verify is weak: wrong page rendered, partial migration, broken layout on mobile.

Add an explicit verify stage after agent action, before promotion — extending your harness with objective UI or HTML checks.

Reading focus

From verify-heavy migration/harness posts, extract:

• Stages — plan, apply, verify, rollback
• Artifacts — screenshots, HTML snapshots, diff reports
• Human escalation — when verify fails repeatedly
• Idempotency — migrations safe to retry

Notes in industry/airbnb/notes.md.

Callout — Verify is not eval: Evals score quality offline on golden set; verify gates this run's artifact before side effects become permanent.

Screenshot/HTML verify step

Minimal implementation for portfolio:

1. Agent modifies static HTML or small web app in sandbox.
2. Headless browser (Playwright) loads page.
3. Verify checks:
   • Selector exists (#checkout-total)
   • Text contains expected string
   • Optional screenshot hash vs baseline (pixel diff tolerances)

Output verify_report.json:

{"status": "fail", "checks": [{"name": "selector", "ok": false, "detail": "missing #submit"}]}

Orchestrator blocks merge/deploy on fail — feed structured failures back to model once.

False positives and false negatives

Document risks in verify_risks.md:

Production teams accept verify flake budgets — track flake rate separately from agent quality.

Integration with durable jobs

Long migrations checkpoint after verify passes each stage — tie to workflow durability lessons. Failed verify → no checkpoint advance → retry or human.

Verify ordering in DAG

Verify can run after each step or only at end — trade latency vs early fail. Migration graphs often verify after every destructive step to minimize rollback scope. Document your capstone choice; Airbnb-style heavy verify favors per-step for long migrations.

Rollback coupling

When verify fails, automated rollback script restores previous artifact version — verify + rollback tested together in chaos drill. Rollback failure escalates to human immediately — worse than original fail.

Accessibility in visual verify

Screenshot diff alone misses screen-reader regressions — note limitation in verify_risks.md; add DOM assertion on aria labels for one demo page if UI capstone.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Study public engineering patterns around Airbnb: Verify-heavy workflows. Rebuild the idea, not scraped proprietary text — focus on transferable architecture.

Diagram — Airbnb pattern (conceptual)

flowchart LR
  Draft-->Verify-->Human-->Ship

Precise definitions & mental model

Extract 3 transferable patterns from Airbnb's public tech narrative on Verify-heavy workflows; map each to a ShipAI module artifact.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Cargo-culting brand names without metrics; inventing fake citations; cloning UI not architecture.

Production & OSS practices

Write a lab README: hypothesis, architecture diagram, eval, cost model, what you'd ask their eng in an interview.

Micro-project: Agent + verify step

Ship:

1. Airbnb reading notes.
2. Agent task changing HTML or UI fixture.
3. Playwright (or equivalent) verify step in harness.
4. verify_risks.md with ≥4 risks and mitigations.
5. Demo pass and fail traces with screenshots attached.

Acceptance: intentional bad agent change fails verify with actionable report.

Checklist

• Reading notes on verify-heavy workflows
• Automated verify step blocking promotion
• verify_risks.md documented
• Pass and fail demos with artifacts
• Verify integrated in harness trace spans