Runbook

Serving platforms vs training stacks

Netflix-class serving problems: routing models, latency SLOs, caching, gradual rollouts, observability — what this module covers. Meta-class training problems: thousand-GPU efficiency, checkpoint sharding, data pipelines — different team, different runbooks. Your ops checklist should reflect inference and agent serving, not pretend you operate a pretraining cluster.

A runbook is the document a tired engineer follows at 2am — commands, not architecture essays.

Runbook sections

Minimum RUNBOOK.md:

1. Service overview — what it does, owner, on-call rotation (can be "you").
2. Dependencies — model provider, vector DB, MCP servers, secrets location.
3. Health checks — /health, canary command, expected latency.
4. Deploy — steps, flags, smoke test after deploy.
5. Rollback — previous image/tag, prompt version revert, feature flag off.
6. Incidents — symptom → diagnosis → mitigation tables.
7. Key rotation — provider API keys, JWT secrets, without downtime plan.
8. Cost emergency — disable route, lower max_steps, switch model tier.

Callout — Rollback prompt version: Git revert is not enough if prod loads prompts from object storage — document prompt artifact versioning and pointer rollback.

Incident playbooks (examples)

Symptom	Likely cause	First actions
p95 latency 3×	slow tool, cache cold	check waterfall traces
429 spike	retry storm	enable circuit breaker, reduce concurrency
canary fail after deploy	prompt regression	rollback prompt pointer
cost alert	runaway loop	disable feature flag, inspect max_steps

Link each to dashboards/log queries (grep trace_id, canary_history.jsonl).

Key rotation without drama

Procedure stub:

1. Issue new key in provider console.
2. Dual-write secrets manager entries API_KEY_PRIMARY, API_KEY_SECONDARY.
3. Deploy reading primary; validate traffic.
4. Swap primary; revoke old after 24h.

Document for each secret your service uses.

Close Milestone 11

Milestone acceptance — deployed service (local docker counts if documented):

1. FastAPI + SSE API (11.1)
2. Per-request cost logging + rollup (11.2)
3. Cache experiment with benchmarks (11.3)
4. JSONL traces + waterfall (11.4)
5. Resilient client (11.5)
6. RUNBOOK.md (this lesson)
7. milestone-11.md — one-page ops summary + demo URL or curl

Industry comparison note: your mini gateway parallels Netflix-style model routing at toy scale — cite in milestone doc, do not overclaim.

On-call readiness without on-call

Solo portfolio still benefits from runbook: future you is on-call. Include "first day with repo" section: where secrets live, how to get read-only prod logs, who to email for vendor outage (status page links).

Post-incident template

After any sev-worthy bug (data leak near-miss, day-long outage in prod if you have one), append postmortem to incidents/YYYY-MM-DD.md: timeline, root cause, action items with owners. Course capstone can use tabletop incident as practice postmortem — still valuable.

Capacity planning stub

Document expected RPS and tokens/day at 10× current demo usage; note which component breaks first (API worker count, provider TPM, vector QPS). Connects Meta literacy (scale) to your serving reality without overbuilding.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

On-call needs runbooks: symptoms → checks → mitigations → communications.

Diagram — Incident loop

flowchart TD
  Alert --> Triage --> Mitigate --> Confirm --> Postmortem

Precise definitions & mental model

SLO burn, rollback, degrade modes (smaller model, disable tools).

Tradeoffs — when to use what

Auto-rollback vs human confirm.

Failure modes (interview + on-call)

No owners; tribal knowledge; no degrade path.

Production & OSS practices

Page-worthy alerts only; quarterly game days.

Micro-project: Ops checklist for your service

Ship:

1. Complete RUNBOOK.md with all sections above.
2. Execute tabletop exercise: simulate canary fail — write timestamps as if real.
3. Document rollback you actually tested (revert one commit or flag).
4. milestone-11.md linking cost sample, trace waterfall screenshot, runbook.
5. README "Production readiness" checklist derived from runbook.

Acceptance: stranger follows runbook deploy + smoke sections successfully.

Checklist

• RUNBOOK.md complete with rollback and key rotation
• Tabletop incident exercise documented
• Milestone 11 doc with cost + traces evidence
• Deploy/smoke steps tested once
• Module README points to on-call owner and health URL