Canaries

Canaries catch silent quality cliffs

A canary is a tiny set of high-signal eval cases run on every deploy — stricter than smoke, smaller than full golden set. They exist because "small" prompt edits cause discontinuous quality cliffs: retrieval route flips, tool description drift, judge rubric ambiguity. Canaries fail fast before traffic hits a broken build.

Think release canaries in ops — 1% traffic — but for quality, not availability.

Designing canary cases

Properties of good canaries:

• Historically brittle — broke before, likely to break again.
• Business critical — refund policy, authz, PII handling.
• Fast to run — <2 min total with mocks.
• Deterministic or tight judge band — low flake.

Size: 5–12 cases, not 500. Each should map to an owner and a regression story ("2024-03-14 prompt shrink dropped EU citation").

Separate from full golden set — canaries ⊆ golden, tagged "canary": true.

Callout — Canaries are not a substitute for full eval: They guard cliffs; nightly golden guards coverage.

Deploy gate integration

Flow:

Build → offline smoke → canary suite → deploy staging → (optional online canary %) → prod

Implement make canary or CI job eval-canary failing non-zero on any canary fail. Block merge to main or block deploy script — pick one and document.

Store canary results time-series (canary_history.jsonl) to spot slow drift even when pass/fail binary.

When canaries fail

Runbook:

1. Identify failing case ID and diff prompt_version / model since last green.
2. Bisect with component metrics — route vs retrieval vs generation.
3. Fix or revert — never weaken canary assertion without review.
4. If behavior change intentional, update canary + golden in same PR with PM sign-off.

Close Milestone 10 on your best agent

Milestone checklist:

1. Golden set in CI (10.1)
2. Calibrated judge optional but documented (10.2)
3. Component + E2E report (10.3)
4. Red-team baseline + guardrails (10.4–10.5)
5. Canary gate on best agent/workflow (this lesson)
6. milestone-10.md summarizing metrics: golden pass %, ASR, canary history screenshot or log

Your "best agent" is the workflow from agentic modules with harness — evals apply to the whole stack, not a toy chatbot.

Canary vs feature flags

Pair canaries with model/prompt flags — deploy code with new prompt behind flag, run canaries against flag on staging before enabling in prod.

Canary ownership rotation

Each canary case has named owner in JSONL metadata — when fail, owner gets pinged first. Prevents orphaned cases nobody understands after team churn.

Canary budget

Cap canary runtime at 2 minutes wall clock — if exceeded, split suite or mock slow tools. Slow canaries get skipped in practice, defeating purpose.

Relationship to feature flags

Canaries validate behavior; flags control exposure. Deploy new prompt behind flag=new_prompt_staging, run canaries against flag-on environment, then promote pointer only when green. Never run canaries against stale config while prod serves different prompt — log config_hash in canary results.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Canaries detect silent quality/safety regressions in prod.

Diagram — Canary flow

flowchart LR
  Traffic --> Canary[Canary prompts]
  Canary --> Score --> Alert

Precise definitions & mental model

Synthetic probes, seed sets, alert thresholds.

Tradeoffs — when to use what

Coverage vs probe cost/privacy.

Failure modes (interview + on-call)

Canaries in train data; ignoring alerts.

Production & OSS practices

Own canary dashboard; page on sustained fail.

Micro-project: Regression gate

Ship:

1. Tag ≥5 canary cases in golden set.
2. make canary (or CI job) blocking on failure.
3. Simulate one canary catch (break prompt, show fail, revert).
4. milestone-10.md with full eval stack summary.
5. README badge or status line "canary: passing" with date.

Acceptance: deploy/merge policy documented; canary run completes <2 min in README timing.

Checklist

• ≥5 canary cases tagged and documented
• Canary gate wired to CI or deploy script
• Simulated regression caught by canary
• Milestone 10 doc with eval metrics
• Module README lists canary ownership and update policy