ShipAI

Real-world examples

Evals as release gates: shipping LLM changes without vibes

Treat prompt and model changes like code — golden sets, graded rubrics, and CI gates that block regressions before customers notice.

11 minPattern inspired by mature LLM product orgs
  • evals
  • ci
  • quality
  • release

Framed from public engineering talks, blogs, and OSS patterns. Not confidential internals or invented quotes.

Vibes don’t scale

If anyone can edit a production prompt without a test, you don’t have a product — you have a stage play. Mature teams (publicly described across AI labs and SaaS) put eval suites on the release path the way unit tests gate backend merges.

Gate model

flowchart LR
  Change[Prompt / model / retriever PR] --> Unit[Unit: schemas + tools]
  Unit --> Offline[Offline golden eval]
  Offline --> Canary[Canary traffic]
  Canary --> Full[Full release]
  Offline -->|fail| Block[Block merge]
  Canary -->|guardrail trip| Rollback[Rollback]

Building a golden set that earns trust

Size: start ~50–100 labeled tasks; grow with production failures.

Slices that matter:

  • Happy path
  • Adversarial / injection
  • Empty retrieval
  • Multilingual if you serve it
  • High-severity policy cases

Labels: expected answer keys, rubrics, or pairwise preferences — document the rubric in-repo.

Scoring without self-delusion

Method Use Pitfall
Exact / regex Structured outputs Brittle to wording
Rubric + human High stakes Expensive
LLM judge Scale Family bias; calibrate
Embedding similarity Rough drafts Misses factual errors

Combine: deterministic checks for contracts; judges for open-ended; humans for calibration samples.

CI wiring

on: pull_request paths: [prompts/**, rag/**, agents/**]
job: eval
  - spin small model or pinned API stub
  - run suite
  - fail if metric < threshold or delta < −ε

Pin model versions in eval config. Flaky float thresholds create alert fatigue — use absolute floors + max allowed regression.

Online canaries

Offline ≠ production mix. Canary on 1–5% traffic with guardrails:

  • Spike in negative feedback
  • Tool error rate
  • Cost per task
  • Safety classifier hits

Auto-rollback beats heroics.

Failure modes

  • Teaching to the test (overfit golden set)
  • Judges that prefer verbose answers
  • Secrets in fixtures committed to git
  • Gates so slow nobody runs them → culture death

What to ship

  1. evals/ directory with fixtures + runner
  2. PR check on prompt paths
  3. Dashboard: last eval vs production feedback rate
  4. Weekly “add 5 failures from prod” ritual
  5. Ownership: who can change thresholds

Punchline

An LLM feature is only as shippable as the tests that can fail. Make failure cheap in CI so failure stays rare in production.