Build and harden

Bar: runnable + honest eval report — not a slide deck

This lesson is the implementation marathon: vertical slice from API/demo UI through agent/workflow to tools/models, plus hardening — failure tests, cost logging, traces, guardrails smoke, runbook stub. The bar is a stranger clones repo, follows README, runs demo script, and sees metrics that match your eval plan — not polished slides hiding broken paths.

Vertical slice beats horizontal layers: one end-to-end user journey complete before secondary features.

Vertical slice definition

From scope freeze demo script, implement:

1. Entrypoint (API route or CLI) for primary flow
2. Orchestration (agent loop or workflow graph)
3. Required tools/models for that flow only
4. Harness verify on critical path (if promised)
5. SSE or progress output if user-facing

Defer BACKLOG items — resist "while I'm here" refactors.

Hardening checklist (minimum)

Production module patterns applied to capstone:

Hardening without eval green is theater — run make eval after each hardening add.

Callout — Honest eval report: Publish pass rates including failures fixed during build. Document known gaps vs. scope freeze metrics.

README for strangers

README sections (top of capstone folder):

1. What this is (one paragraph)
2. Prerequisites (API keys, Python version, Ollama optional)
3. Setup commands (copy-paste block)
4. Demo command (exact)
5. Eval commands with expected pass summary
6. Architecture link
7. Known limitations

Stranger test: someone not you runs setup + demo without DMs.

Failure tests to include

Pick ≥1 from workflow module:

• Kill mid-run → resume from checkpoint
• Tool timeout → structured error, no hang
• Verify fail → no promotion, bounded retry
• Guardrail block → user-visible message

Automate where cheap; document manual chaos steps otherwise.

Avoid capstone traps

• Secret keys in git — use .env.example
• "Works on my machine" — pin dependencies lightly
• Eval only after everything built — run partial evals daily
• Giant refactor week 4 — freeze scope

eval_report.md

Commit current metrics:

## Capstone eval report (YYYY-MM-DD)
- Golden pass: 18/20 (90%)
- Canaries: 6/6
- Red-team critical ASR: 0/5
- p95 latency: 6.2s
- Mean cost: $0.031/req
## Known failures
- case refund-eu-02: retrieval miss (BACKLOG)

Update through build; final version in lesson 15.4.

Daily integration habit

During build week: run smoke eval every morning before new features — catches regressions when fatigue rises. Commit eval_report date stamps showing trend.

Dependency pinning note

requirements.txt or lockfile with major versions pinned — README states Python version tested. Reduces stranger clone friction — part of hardening.

Feature flags for capstone demo

Hide incomplete BACKLOG features behind CAPSTONE_V1=1 code paths — demo script only exercises v1 surface; judges don't wander into broken WIP routes.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Harden: tests, guardrails, load, threat model — after the slice works.

Diagram — Harden loop

flowchart LR
  Slice --> Tests --> Guards --> Load --> Fix --> Slice

Precise definitions & mental model

Threat model, rate limits, degrade modes.

Tradeoffs — when to use what

Ship date vs harden depth — timebox.

Failure modes (interview + on-call)

Security bolted after demo recording.

Production & OSS practices

Go/no-go checklist before public demo.

Micro-project: Runnable vertical slice

Deliver:

1. End-to-end primary demo working from README commands.
2. Hardening checklist items implemented or explicitly waived with reason.
3. eval_report.md with current numbers vs. targets.
4. One failure test automated or scripted.
5. Stranger test log (name/date who ran successfully, or self with fresh clone path).

Acceptance: demo script completes; eval report honest about gaps.

Checklist

• Vertical slice matches scope freeze demo
• Cost + trace instrumentation on primary flow
• Failure test documented and executed once
• README stranger-tested
• eval_report.md with pass rates and known failures