Harness thinking

Harnesses turn hopeful agents into engineered workflows

A bare model with tools is a capability. A harness is the engineering wrapper that makes capability shippable: validate inputs, constrain execution environment, verify outputs, and block promotion when checks fail. Production coding agents (Stripe Minions-class), migration agents (Airbnb verify-heavy flows), and merchant assistants (Shopify Sidekick-class) differ from demos less in model size than in harness depth.

The recurring pattern:

Validate → (optional Plan) → Execute in sandbox → Verify → Ship or retry

Your agent module stop conditions were the seed. This lesson closes the agentic workflow milestone by wrapping your best workflow in a full harness and tying it to evals plus a failure isolation test from prior lessons.

Validate: refuse garbage early

Pre-flight checks before any model call or tool side effect:

• Schema — User upload size, JSON shape, allowed MIME types.
• Authz — Actor may invoke this workflow on this resource.
• Policy — Prompt injection heuristics on pasted content (deepen in security lessons).
• Budget — Token and cost ceiling for run.

Validation failures return structured errors to user — never invoke tools with partial invalid state.

Example:

{"blocked": true, "stage": "validate", "reason": "FILE_TOO_LARGE", "limit_mb": 10}

Callout — Stripe-style harness: Separate "propose diff" from "apply diff." Validation runs on proposal; apply only after verify gates green.

Sandbox: limit blast radius

Execution environment constraints:

• Filesystem — chroot or temp dir; allowlist paths.
• Network — egress allowlist or none for coding tasks.
• Commands — prefix allowlist for shell tools.
• Secrets — inject via env at runtime; never in model context.

MCP servers are natural sandbox boundaries — crash or compromise isolates to subprocess.

Airbnb-style migration agents add environment parity checks before apply: schema version, feature flags, dry-run mode.

Verify: objective gates beat model self-report

Never ask "are you done?" without external checks:

Verification produces artifacts (verify_report.json) stored on the run log. Orchestrator promotes output only if verify.status == pass.

False positive/negative risks: flaky tests, visual diff noise — document in runbook and tune thresholds.

Retry policy inside the harness

When verify fails:

1. Feed structured failure to model (failed tests only, not whole log).
2. Increment attempt; stop at max.
3. Escalate to human with full artifact bundle.

Avoid infinite "fix loop" — each attempt costs money and may drift scope.

Industry patterns at high level (not copy-paste)

• Stripe / coding agents — Sandboxed repo, test/lint verify, human review for merge.
• Shopify / merchant agents — Tool allowlists per merchant role, LLM-judge on tone/policy in eval loop.
• Airbnb / migration — Verify-heavy: run transformation, validate output schema, compare snapshots before commit.

You are not rebuilding these systems — you are mapping their harness stages to your portfolio architecture diagram.

Close Milestone 9: evals + failure test

Milestone acceptance:

1. Workflow with router or orchestrator pattern (lesson 9.1).
2. Message protocol + event log (9.2).
3. Agentic RAG or documented skip with reason (9.3).
4. Checkpoint resume proven (9.4).
5. Worker kill or tool chaos test (9.5).
6. Harness validate → sandbox → verify on primary path (this lesson).
7. Small eval set (≥10) on verify pass rate + one end-to-end golden case.

Write milestone-9.md linking traces, eval scores, and harness stage timings.

Human gates in the harness

Some verify passes still require human approval — merge to main, refund above threshold, publish merchant discount. Harness stage human.approval with timeout and audit: {approver, decision, timestamp}. Model proposes; human disposes; never conflate LLM judge score with legal sign-off.

Document which capstone paths require human stub (console y/n) vs fully automated verify.

Measuring harness overhead

Span timings for validate/sandbox/verify as fraction of total trace. If verify is 80% of latency, invest in parallel tests or slimmer fixture — harness correctness should not destroy UX. Report harness_ms in eval_report for capstone.

Engineering problem (staff framing)

The harness (driver, tools, evals, policies) often matters more than the model. Invest here.

Diagram — Harness surrounds model

flowchart TD
  H[Harness: tools/evals/policy/obs] --> M[Model]
  M --> H
  H --> User

Precise definitions & mental model

Harness engineering: make models reliable with software, not vibes.

Tradeoffs — when to use what

Bigger model vs better harness — often harness wins unit economics.

Failure modes (interview + on-call)

Model-shopping without fixing tools/evals; no regression suite.

Production & OSS practices

Harness checklist in design docs; model is a replaceable component.

Deep dive (FAANG / OSS bar)

Harness checklist (steal this for design docs)

1. Tool schemas + authz
2. Stop/budget policies
3. Offline eval suite
4. Tracing/redaction
5. Degrade modes (model downgrade, disable tools)
6. Cost accounting keys

If a team only upgrades model size without this list, unit economics usually worsen.

Micro-project: Add validation loop around agent

Ship:

1. Implement three harness stages around your main workflow entrypoint.
2. At least two verify checks (tests, schema, or screenshot stub).
3. Run eval: 10 cases with verify pass/fail breakdown.
4. Re-run failure isolation test — harness should fail closed, not partial ship.
5. Diagram harness stages vs. Stripe/Airbnb/Shopify bullets in README.

Acceptance: verify failure blocks promotion; eval table in milestone doc.

Checklist

• Validate, sandbox, verify stages implemented and logged
• Verify uses external checks, not model self-assessment
• Milestone 9 doc with eval + failure test links
• Retry cap on verify loop documented
• Module README runnable smoke path for stranger