Architecture and eval plan

No build without eval gates and a defensible diagram

Capstone implementation starts only when architecture and eval plan exist. The diagram shows trust boundaries — where data crosses VPC, where authz enforced, where vendor models invoked. The eval plan shows how you will prove the capstone works before the demo narrative — golden set, canaries, component metrics, red-team subset.

This lesson turns proposal promises into engineering artifacts reviewers can critique without reading code.

Architecture diagram requirements

Include boxes for:

• Client / API (FastAPI + SSE from production module)
• Agent or workflow orchestrator
• Model gateway or direct API (Netflix lab pattern if used)
• Tools / MCP servers with authz boundary (Uber lab)
• Retrieval / vector store (if applicable)
• Harness stages (validate, sandbox, verify — Stripe/Airbnb)
• Observability sinks (traces, cost logs — no PII)
• External vendors (dashed boundary)

Mark trust boundaries with labeled edges: TLS, redaction, auth tokens.

Store as capstone/architecture.png or architecture.md ASCII — both OK if readable.

Callout — Diagram lies if authz missing: Every mutating tool edge crosses a boundary with actor identity labeled.

Eval plan structure

capstone/eval_plan.md:

1. Golden set — target count by bucket (happy, tool fail, policy, injection)
2. Component metrics — route, retrieval, verify as applicable
3. Judge rubric — criteria + calibration status
4. Canaries — 5–12 IDs blocking release
5. Red-team — critical ASR targets
6. CI commands — make eval, make canary, make redteam-smoke
7. Promotion policy — thresholds from eval module

Tie each metric to capstone success metrics from scope freeze.

Industry pattern mapping

Table mapping diagram/eval elements to industry labs:

Capstone element	Pattern source	Simplification
Model gateway	Netflix lab	two backends only
MCP authz	Uber lab	mock JWT
Verify gate	Airbnb lab	HTML check only
Merchant judge loop	Shopify lab	10 cases

Honest simplification column required — graders reward self-awareness.

Trust boundary checklist

• User PII redacted before vendor LLM
• Tool credentials not in model context
• Audit log on mutating tools
• Eval data contains no production secrets
• Staging vs prod keys separated

Link privacy_sketch.md from synthesis module.

Pre-build gate

Self-certify before lesson 15.3 coding marathon:

• Architecture reviewed against scope freeze — no orphan boxes
• Eval plan covers every success metric
• At least 15 golden cases written (may fail until built — that's OK)

Threat modeling walkthrough

One-hour exercise: STRIDE each trust boundary on architecture diagram; file issues for unmitigated threats — feeds capstone hardening in lesson 15.3. Link top issues to red-team cases.

Eval debt tracking

Cases you plan but haven't written — list in eval_plan as debt with target date. Honest debt list beats missing buckets silently.

Staging vs production eval parity

Run same canaries against staging before prod deploy — config drift between environments causes classic false confidence. Document env-specific golden overrides if unavoidable (different mock tools).

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Architecture without evals is fiction. Pair every component with a metric.

Diagram — Arch ↔ evals

flowchart LR
  Comp[Component] --> Metric
  Metric --> Gate

Precise definitions & mental model

Critical path, failure domains, eval ownership.

Tradeoffs — when to use what

Complex graphs vs measurable simplicity.

Failure modes (interview + on-call)

Orphan components with no metric.

Production & OSS practices

Architecture review checklist includes eval map.

Micro-project: Architecture + eval plan

Ship:

1. architecture diagram with trust boundaries.
2. eval_plan.md complete with CI commands.
3. Industry mapping table ≥2 patterns.
4. Golden case list (JSONL stub rows OK) ≥15 IDs defined.
5. pre_build_signoff.md checklist signed.

Acceptance: eval plan explains what fails capstone if canary cap-03 fails.

Checklist

• Architecture diagram with vendor and authz boundaries
• eval_plan.md with buckets and thresholds
• Industry pattern mapping table
• ≥15 golden cases defined in JSONL
• pre_build_signoff completed