Agentic RAG

Sometimes the right retrieval decision is to not retrieve

Naive RAG always embeds, always retrieves, always stuffs chunks — even when the user says "thanks" or asks a question answerable from one clarifying follow-up. Agentic RAG adds a routing layer: retrieve, answer from model knowledge, or ask the user — with logging and evals for each path.

The agent treats retrieval as a tool with cost and failure modes, not a mandatory preprocessor. That shift improves latency, reduces hallucinated citations, and makes evals interpretable.

Three-way routing

Router inputs: user message, session summary, retrieval confidence preview (optional cheap keyword hit), tool availability.

Example router output:

{"route": "retrieve", "reason": "mentions refund policy", "query": "EU refund policy 30 day"}

Callout — Ask-user is a feature: Products hide uncertainty; agents should ask when retrieval score spread is flat or top chunk contradicts session facts.

Implementing retrieve as a tool

Instead of automatic pipeline:

1. Model or small router chooses search_docs(query).
2. Tool returns ranked chunks + scores + metadata.
3. Model synthesizes with citation requirements.

Skills encode citation format: [doc_id:section]. Guardrails block answers when no chunk exceeds score threshold — force ask-user route.

Log every decision:

{"route": "retrieve", "query": "...", "top_score": 0.71, "chunks_used": 3}

Eval against always-retrieve baseline

Build a labeled set of ≥30 queries with gold route labels (answer | retrieve | ask). Metrics:

• Route accuracy — did we pick the right path?
• End-to-end correctness — given route, is final answer acceptable?
• Latency/cost — token and ms vs. baseline

Expect agentic RAG to win on latency for simple turns and on precision for ambiguous ones. It loses if router is untested — always ship route evals before answer evals.

Show one case where E2E passes but route was wrong — proves you need component metrics alongside end-to-end scores.

Failure modes

• Retrieve spam — Model calls search every turn. Cap retrieves per session; penalize in router training prompts.
• Answer when retrieve needed — Compliance bug. Block answer route for regulated intents via rules overlay.
• Ask loops — Model keeps asking without retrieving. Max asks per run; escalate to human.

Chunk quality still matters

Routing does not fix bad indexing. Agentic RAG assumes retrieval tool returns useful signal. Monitor zero_result_rate and low_score_rate separately from route accuracy.

Combining with skills and memory

Session memory may already contain retrieved facts — router should check memory before re-retrieving duplicate policy text. Skills define when re-fetch is mandatory (policy version changed, user cites new doc ID).

Router features beyond raw text

Improve route accuracy with cheap structured features before another LLM call: keyword hits on policy corpus, metadata filters (user region, product SKU present in message), session flags (docs_already_loaded), and retrieval dry-run score from lightweight BM25. Combine into score vector; rules layer overrides ML when compliance demands retrieve for regulated intents regardless of score.

Log feature vector with route decision — essential when debugging "why didn't it retrieve?"

Cost accounting for routes

Answer path saves embedding + retrieval + extra context tokens; retrieve path adds vendor and index cost. Tag cost_log.route from production module — compare mean $/request by route monthly. Product may cap retrieves per session when users spam search-like messages without new information.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Agentic RAG lets the model decide retrieve/rewrite/stop — powerful and easy to thrash.

Diagram — Agentic RAG loop

flowchart TD
  Q --> Decide{Retrieve?}
  Decide -->|yes| Search --> Rank --> Decide
  Decide -->|answer| Gen
  Decide -->|abstain| Abs

Precise definitions & mental model

Adaptive retrieval, evidence thresholds, tool-formed queries.

Tradeoffs — when to use what

Fixed top-k RAG (predictable) vs agentic (flexible, costlier).

Failure modes (interview + on-call)

Retrieve forever; ignore evidence; citation drift.

Production & OSS practices

Max retrieve calls; faithfulness judge; cache queries.

Deep dive (FAANG / OSS bar)

Thrash detector

If the same query embedding is retrieved twice, or evidence score does not improve after a rewrite, stop. Agentic RAG without a progress metric is a spend loop.

Faithfulness vs answerability

• Answerable — enough evidence exists.
• Faithful — claims supported by cited chunks.

Evaluate separately; optimizing only BLEU/ROUGE misses hallucinations.

Micro-project: Retrieve vs answer vs ask

Ship in portfolio:

1. Implement three-way router over your existing RAG stack (or toy corpus).
2. Log route + retrieval scores on every turn.
3. Label 30 cases; report route accuracy vs. always-retrieve baseline on latency and correctness.
4. Document one failure you fixed by adding ask-user.

Acceptance: markdown table with route metrics; trace showing intentional non-retrieve turn.

Checklist

• Three routes implemented with structured logging
• Labeled eval set ≥30 with route accuracy computed
• Comparison to always-retrieve documented
• Citation or ask-user behavior on low confidence
• Module README summarizes when retrieve loses to answer