LLM-as-judge

Useful when calibrated, dangerous when not

Many agent behaviors lack deterministic oracles — tone, helpfulness, policy compliance nuance, reasoning quality. LLM-as-judge uses a separate model call with a rubric to score outputs. Shopify-class production systems use judges in eval loops; they also know unc calibrated judges optimize for length and confidence.

Treat judges as noisy instruments — calibrate against human labels, measure agreement, and never use as sole safety gate without deterministic checks.

Building a judge prompt

Structure:

1. Role — impartial grader, not advocate for the agent.
2. Rubric — 3–6 criteria with 1–5 scale definitions per level.
3. Input bundle — user message, agent trace summary, final answer (not always full tool logs — budget).
4. Output schema — JSON only:

{
  "scores": {"correctness": 4, "citation": 5, "tone": 3},
  "overall": 0.82,
  "rationale": "one paragraph",
  "fail_reasons": []
}

Criteria examples for support agent:

• Correctness — factually aligned with tool results
• Policy — no unauthorized promises
• Citation — sources cited when claiming doc facts
• Clarity — actionable for user

Anchor each score level with concrete examples in the rubric ("5 = cites doc section and matches tool output").

Callout — Position bias: Judges favor longer answers. Randomize order when comparing two answers; use pairwise comparison for A/B model picks.

Calibration workflow

1. Sample 30–50 production or golden outputs.
2. Two humans label (or one expert if budget tight) on same rubric.
3. Run judge on same set.
4. Compute Pearson/Spearman on overall; per-criterion confusion matrix.
5. Iterate rubric where judge diverges systematically.

Target: ≥0.7 correlation on overall for CI gating; lower thresholds only for trend monitoring.

Document known blind spots — judges bad at counting, bad at verifying math, gullible to confident hallucinations.

Measuring agreement

Report:

• Human–human agreement (ceiling)
• Judge–human agreement
• Judge–judge stability (same input, two calls — should be tight with temperature 0)

If judge–judge unstable, tighten schema and lower temperature before trusting CI.

Use adversarial calibration cases — answers that sound good but violate policy — judges must fail them.

When not to use judges

• Binary safety (PII leak) — use regex + block lists.
• Exact API payloads — schema validation.
• Math — code execution or calculator tool ground truth.

Judges complement; they rarely replace.

Cost and latency

Judging doubles model cost for eval runs. Cache judge results keyed by (output_hash, rubric_version). Run full judge only nightly; PR gate uses subset or deterministic checks only.

Pairwise comparison mode

When choosing between prompt A and B, pairwise judge ("which answer better satisfies rubric?") often beats absolute scoring — reduces scale drift. Run double-blind: swap order A/B in half of cases to detect position bias; report inconsistency rate.

For capstone A/B, require ≥60% pairwise wins before promoting prompt B.

Judge model selection

Smaller judge model saves cost but may diverge from human labels — re-calibrate when switching judge model. Keep judge on stable temperature 0; note in rubric version header which judge model calibrated.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Judges scale eval but inherit bias. Calibrate against humans; version judge prompts.

Diagram — Judge pipeline

flowchart LR
  Out[Model output] --> J[Judge model]
  Gold[Human labels] --> Cal[Calibrate]
  J --> Score
  Cal --> Score

Precise definitions & mental model

Pairwise vs pointwise; position bias; rubrics.

Tradeoffs — when to use what

Human quality vs judge scale/cost.

Failure modes (interview + on-call)

Judge shares blind spots with candidate; unblinded references.

Production & OSS practices

Agreement metrics; separate judge vendor when possible.

Micro-project: Calibrate small human set

Ship:

1. judge/rubric.md + prompt template with JSON schema output.
2. Label ≥20 cases human overall score (spreadsheet OK).
3. Run judge; produce agreement report (calibration.md with correlation).
4. Adjust rubric once based on worst disagreements; re-run.
5. Wire optional judge score into eval runner from lesson 10.1.

Acceptance: calibration.md shows before/after agreement; rubric version bumped.

Checklist

• Judge rubric with anchored score levels
• JSON schema output enforced
• ≥20 human labels compared to judge
• Agreement metrics in calibration.md
• Documented judge blind spots and gating policy