Production RAG: chunking, hybrid search, rerank, and eval gates

Why demos lie

A weekend RAG demo: dump PDFs → embed → cosine top-k → stuff into a prompt → “wow.” Production RAG fails on identifiers, negation, stale docs, chunk boundaries, and confident wrong citations.

Pattern inspired by public write-ups from teams building internal search + assistant products: treat RAG as a retrieval system with an LLM renderer, not a prompt trick.

End-to-end pipeline

flowchart TD
  Src[Sources: docs, tickets, wikis] --> Ingest[Ingest + normalize]
  Ingest --> Chunk[Chunk + metadata]
  Chunk --> Emb[Embed]
  Chunk --> Lex[Lexical index]
  Emb --> Vec[Vector index]
  Q[User query] --> Rew[Optional rewrite]
  Rew --> Hybrid[Hybrid retrieve]
  Lex --> Hybrid
  Vec --> Hybrid
  Hybrid --> Rerank[Rerank top-N]
  Rerank --> Ctx[Context packer]
  Ctx --> LLM[LLM + citations]
  LLM --> Ans[Answer + spans]
  Ans --> Eval[Offline + online eval]

Chunking that survives contact with users

Goals: each chunk answers something alone; metadata enables filters; overlaps preserve boundary sentences.

Ship rule: store doc_id, section, updated_at, acl, source_url on every chunk. Retrieval without metadata is a toy.

Hybrid search (lexical + dense)

Dense embeddings miss SKUs, error codes, exact names. BM25/sparse misses paraphrase. Production systems fuse both:

flowchart LR
  Q[Query] --> D[Dense top-k]
  Q --> S[Sparse top-k]
  D --> F[Fusion / RRF]
  S --> F
  F --> Cand[Candidate set]

Reciprocal Rank Fusion (RRF) is a strong default when scores aren’t calibrated. Learn to tune k and candidate pool size before buying a fancy reranker.

Reranking

Cross-encoders / late-interaction models score (query, passage) jointly — better precision, higher latency/cost.

Pattern: retrieve 50–100 cheaply → rerank to 5–10 → pack context.

Failure modes: reranker trained off-domain; latency blows TTFT; rerank on already-wrong candidate set (garbage in).

Context packing and citations

Packing is an engineering problem:

• Token budget for system + tools + history + retrieved text
• Prefer diverse docs over 5 chunks from one PDF
• Require span-level citations mapped to chunk ids
• Refuse or hedge when top scores are weak

if max(score) < τ or citation_coverage < ρ:
  return "insufficient evidence" path

Eval gates (non-optional)

Offline set (≥50–200 gold questions):

• Retrieval: recall@k, MRR, nDCG
• Answer: groundedness / faithfulness, citation precision, task accuracy
• Regression: every chunker or embedder change re-runs the suite

Online:

• Thumbs + “wrong citation” reasons
• Sampled LLM-as-judge with human calibration
• Slice by doc type and tenant

Without evals, every “improving the prompt” is folklore.

Tradeoffs cheat sheet

What to ship this month

1. Ingestion job with versioned chunker config
2. Hybrid retrieve + optional rerank behind a flag
3. Citation UI that links to source spans
4. Golden eval set in CI for retrieval recall@10
5. ACL filters applied before the LLM sees text

Design interview angle

Walk the pipeline left-to-right, name one failure at each stage, and say what metric catches it. That reads as production literacy — not framework name-dropping.