Eval vs base / teacher

Fine-tunes without report cards are vibes

Training loss going down does not mean deployment wins. Lesson 6.4 builds a report card comparing three baselines on the same held-out test set:

1. Base model — open weights, no adapter, minimal or zero-shot prompt.
2. Adapter model — your LoRA checkpoint merged or loaded.
3. Teacher / prompt-only — frontier or strong API model with your best prompt pack from the prompt engineering module.

Ship only if adapter beats the baseline that matters for your worksheet decision — usually prompt-only on accuracy and meets latency/cost targets locally.

Callout — same test set, same metric script: Baselines compared unfairly invalidate the entire module milestone.

Metrics for narrow tasks

Pick metrics aligned with task type:

Run eval.py that loads test.jsonl, runs each model backend, writes predictions.jsonl.

for row in test:
    pred_base = generate(base_model, row)
    pred_adapter = generate(base + adapter, row)
    pred_teacher = call_api(prompt_pack, row)

Parse outputs to canonical form before scoring (strip whitespace, lowercase labels).

Report card structure

m6/eval/report_card.md:

# Ticket classifier — 2026-08-11

## Summary
| Model | Accuracy | Macro-F1 | p50 latency (ms) | Cost / 1k |
|-------|----------|----------|------------------|-----------|
| Base 0-shot | 0.62 | 0.58 | 35 | local |
| LoRA v1 | 0.91 | 0.89 | 38 | local |
| GPT-4o-mini prompt | 0.88 | 0.86 | 420 | $0.15 |

## Decision: SHIP LoRA v1
Beats teacher on F1; 10x lower latency; fails on 2/50 spot checks (see failures.md).

## Failures
- id 044: sarcasm misclassified as bug
...

Include confusion matrix image or ASCII table for classification.

Spot checks beyond numbers

Automatic metrics miss:

• Tone violations
• Subtle safety issues
• Format noncompliance when JSON parser is lenient

Review 20 random test rows + all failures where adapter ≠ teacher. Tag failure modes for next data iteration.

Ship / no-ship criteria

From worksheet kill criteria, example:

• Ship if: adapter accuracy ≥ teacher AND ≥ base + 0.25 absolute AND no safety regression on red-team rows.
• No-ship if: adapter within noise of teacher but adds ops burden — stay prompt-only until volume justifies.

Honest no-ship reports are portfolio-positive — they show engineering judgment.

Regression fixtures

Save failing inputs to m6/eval/fixtures/regression.jsonl for CI later. When you train v2 adapter, same script must not regress on those ids.

Callout — merge vs on-the-fly adapter: Document whether eval used merged weights or PEFT load — latency differs.

Statistical significance on small test sets

With 50–200 test rows, accuracy differences of 1–2 points may be noise. Report:

• Exact counts correct/total per model
• Wilson score interval or bootstrap CI if you implement it
• McNemar-style paired comparison on per-row pass/fail (did adapter fix rows teacher missed while teacher fixed adapter misses?)

Do not ship on 0.5 point lift without examining which rows flipped — aggregate accuracy hides catastrophic regressions on billing class.

Calibration and confidence (optional stretch)

If model outputs probabilities or you derive confidence from token logprobs, plot reliability diagram — overconfident wrong answers are worse than low-confidence wrong answers for HITL routing later.

Serving cost comparison column

Extend report card with projected monthly cost at expected QPS:

Ship decision weighs accuracy and unit economics — adapter wins on both or tradeoff documented ("2 points worse but 8× cheaper").

Report card as living document

Version report cards per adapter release:

m6/eval/report_cards/
  2026-08-11_v1.md
  2026-08-18_v2.md

Diff v1 vs v2 on same test set — interview story about iterative improvement. Include "what we tried and rejected" section for failed hyperparameter or data experiments.

Baseline fairness checklist

Before claiming adapter wins:

• Teacher uses best prompt pack version from prompt engineering module, not empty system prompt.
• Base model uses same chat template as adapter inference.
• Test set never seen during training or early-stop tuning.
• Metrics script shared across all three backends — no bespoke parsing per model.

Error analysis workflow

1. Bucket errors (label ambiguity, OOD phrasing, train gap).
2. Count buckets — top bucket guides data collection.
3. Optional: add 50 rows targeting top bucket, retrain v2, re-run report card.

Engineering problem (staff framing)

Measure deltas vs base and teacher; otherwise you cannot tell if FT helped.

Diagram — Eval triangle

flowchart TD
  Base --> Metrics
  Student --> Metrics
  Teacher --> Metrics

Precise definitions & mental model

Paired eval sets; win-rate; task metrics not vibes.

Tradeoffs — when to use what

Automatic metrics vs human/LLM judge cost.

Failure modes (interview + on-call)

Train-set contamination; reporting only wins.

Production & OSS practices

Gate merge on eval budget + regression suite.

Micro-project: Report card

In m6/eval/:

1. Implement unified eval.py for base, adapter, teacher backends.
2. Run on held-out test split; produce report_card.md + predictions.jsonl.
3. Explicit SHIP or NO-SHIP decision with rationale.
4. ≥5 spot-check notes on disagreements.

Checklist

• Three baselines on identical test set
• Metrics automated and reproducible
• Decision stated with numbers
• Failure examples documented