Component vs E2E

Component metrics find the broken stage faster

End-to-end (E2E) evals ask: did the user get a good outcome? Component evals ask: which stage failed — route, retrieval, tool, generation, verify? E2E alone is insufficient for agent pipelines because a strong generator can mask retrieval failure (lucky guess) and a fixed retriever can still lose to bad synthesis.

Ship both. E2E guards product outcomes; components guard debuggability and iteration speed.

Stages to eval separately

Typical RAG + agent stack:

Log stage outputs in eval runner so one golden case produces a stage report card.

Case study: E2E pass, retrieval fail

Scenario:

• User asks refund eligibility for order EU-8842.
• Retriever returns US policy chunk (wrong region, high lexical overlap).
• Generator confidently cites US 30-day rule — wrong for EU user.
• E2E judge scores 4/5 on "helpful tone" — false pass.

Component eval catches:

• retrieval_recall@5: gold doc eu-refund-v3 missing.
• citation_match: cited chunk region != order region metadata.

Fix retrieval routing or metadata filter — E2E may barely move until edge case retested.

Document this pattern in your portfolio as e2e_hidden_failure.md — required artifact for micro-project.

Callout — Goodhart's law: Optimizing only retrieval MRR can hurt generation if chunks are relevant but unreadable. Watch both; don't collapse to one number.

Wiring into eval reports

Unified report shape:

{
  "case_id": "refund-eu-01",
  "e2e_pass": true,
  "components": {
    "route": {"pass": true},
    "retrieval": {"pass": false, "gold_in_top_k": false},
    "generation": {"judge": 0.85}
  },
  "diagnosis": "e2e_false_pass_retrieval_miss"
}

CI policy options:

• Strict — fail if any blocking component fails, even when E2E passes.
• Pragmatic — warn on component fail, fail only on E2E for PR; block release on component trends.

For learning portfolios, prefer strict on labeled regression cases.

Synthetic negatives

Build cases designed to fail one stage:

• Retrieval poison — corpus has misleading doc.
• Tool trap — correct answer requires tool model skipped.
• Verify trap — answer sounds fine, schema wrong.

These accelerate harness and eval maturity faster than random user logs.

Component eval cost

Cheaper than full E2E when tools mocked — run hundreds of retrieval cases without generation. Use layered CI: retrieval suite on index changes; full E2E nightly.

Stage ownership in on-call

When E2E fails at 2am, component report routes page: retrieval fail → search on-call; tool schema fail → agent platform; generation fail → prompt owner. Build runbook links from component name to dashboard — reduces finger-pointing.

Synthetic component tests

Generate retrieval-only fixtures without generation: given query, assert gold doc IDs in top-k. Runs in milliseconds — run on every index deploy. Separates "index broken" from "model dumb."

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Component evals localize faults; e2e measures user value. Balance both.

Diagram — Eval pyramid

flowchart TD
  E2E[E2E tasks]
  Comp[Component: retrieve/tool/parse]
  Unit[Unit schemas]
  E2E --> Comp --> Unit

Precise definitions & mental model

Fault localization, contract tests, task success rate.

Tradeoffs — when to use what

E2E flaky/expensive vs component blind to integration.

Failure modes (interview + on-call)

Only e2e → slow debug; only unit → false confidence.

Production & OSS practices

Pyramid in CI timings; quarantine flakes.

Micro-project: Retrieval ≠ answer eval

Ship:

1. Extend golden set with ≥5 cases with gold retrieval labels (doc IDs).
2. Implement retrieval metrics separate from generation judge.
3. Document one E2E pass / retrieval fail with diagnosis.
4. Combined eval report JSON for all golden cases.
5. README section: which metric you trust for index vs. prompt changes.

Acceptance: report shows at least one case where components disagree with E2E headline pass.

Checklist

• Retrieval (or tool) component metrics implemented
• e2e_hidden_failure.md case study committed
• Combined report includes per-stage pass/fail
• CI or make documents gating policy on components
• Module README links to sample report JSON