Offline vs online evals

Evals before demos — now with automation

Shipping an agent without evals is shipping without a test suite. Offline evals run fixed inputs in CI before merge; online evals sample live traffic after deploy. Confusing the two causes either false confidence (offline-only) or slow feedback (waiting for production regressions to appear in dashboards).

Offline answers: "Did this PR break known good cases?" Online answers: "Is the deployed system drifting on real users?" Both need different datasets, metrics, and promotion rules.

Offline eval architecture

Components:

1. Golden set — Curated (input, expected_behavior) cases; version controlled.
2. Runner — Invokes agent/ workflow with frozen config (model, prompt hash, tool mocks where needed).
3. Scorers — Deterministic checks + optional LLM-as-judge (next lesson).
4. Gate — CI fails if metrics below threshold.

Golden cases should be stable — avoid assertions on exact prose when paraphrase is OK. Prefer:

• Tool was called with correct args
• Output contains required fields
• Judge score ≥ threshold on rubric

Store golden set as JSONL:

{"id": "refund-01", "input": "...", "expect": {"tools": ["lookup_order"], "judge_min": 0.8}}

Callout — Freeze config in CI: Log model, prompt_version, skill_versions on every eval run. Non-reproducible evals are worthless for bisect.

What belongs in CI vs. nightly

PR gates must finish in minutes. Heavy evals belong in scheduled jobs with trend dashboards.

Online monitors (high level)

Production signals:

• Task success rate — user accepted outcome or verify passed
• Tool error rate — spikes indicate dependency or schema drift
• Judge drift — sampled LLM-judge scores vs. baseline week
• Latency/cost — not quality per se but catches config mistakes

Online rarely blocks individual PRs; it triggers rollbacks and golden set additions when new failure modes appear.

Separate alerting from gating. Gate on offline; alert on online anomalies.

Promotion criteria

Define explicit rules before debates:

Example policy:

• Offline smoke: 100% pass on blocking cases, ≥95% on full set.
• No regression >2% on judge average vs. main branch.
• New failures require labeled wontfix or golden update in same PR.
• Online: 24h canary with success rate within 1% of control before 100% traffic.

Document who can override (tech lead + PM) and required artifacts (eval diff report).

Building your first golden set

Sources:

• Bugs you fixed (regression tests for agents!)
• Support tickets anonymized
• Synthetic edge cases from red-team (later lesson)
• Hand-labeled "hero" demos that must never break

Aim for coverage buckets: happy path, tool failure, ambiguous input, policy boundary, injection attempt (stub until red-team lesson).

Start with 15 cases, grow weekly from production misses.

Anti-patterns

• Eval only happy path — Green CI, red production.
• Brittle string match — "I'm sorry" exact match fails on rephrase.
• Flaky tools in PR gate — Mock or record/replay external APIs.
• No version pinning — Model update Friday night breaks Monday CI mysteriously.

Golden set hygiene

Rotate cases out when product intentionally changes behavior — stale goldens become noise. Each case needs owner and last_verified date in JSONL metadata. Quarterly audit: delete duplicates, merge cases differing only in phrasing, add cases from top production failures.

Version golden set semver; eval runner records golden_version in report for bisect when CI flips.

Connecting offline to deploy hooks

Map git branches: PR runs smoke; merge to main runs full offline; tag release runs full + red-team smoke. Document in promotion.md exactly which command gate blocks deploy script — avoids "we thought CI ran that."

Engineering problem (staff framing)

Offline catches regressions pre-ship; online catches drift. You need both with clear ownership.

Diagram — Eval loops

flowchart LR
  PR --> Off[Offline suite] --> Ship
  Ship --> On[Online metrics]
  On --> Off

Precise definitions & mental model

Golden sets, shadow traffic, interleaving, metric windows.

Tradeoffs — when to use what

Offline speed vs online realism.

Failure modes (interview + on-call)

Only vibe QA; online without privacy review.

Production & OSS practices

Gate PRs on offline; weekly online review.

Micro-project: Golden set in CI

In portfolio:

1. Create evals/golden.jsonl with ≥15 cases across ≥3 buckets.
2. Implement runner producing eval_report.json with pass/fail per case.
3. Wire GitHub Actions or local make eval failing on regression.
4. Write promotion.md with offline gate thresholds and online monitor wishlist.
5. Demonstrate one intentional break + fix in git history or doc screenshot.

Acceptance: CI (or documented command) fails when a golden case regresses.

Checklist

• Golden set ≥15 cases in version control
• Eval runner logs config versions
• CI or make target gates on smoke set
• promotion.md with explicit thresholds
• Module README explains offline vs online split