RAG building blocks

What RAG actually is

Retrieval-Augmented Generation is a systems pattern, not a checkbox feature: retrieve relevant evidence, pack it into the prompt under a token budget, generate an answer that stays faithful to that evidence, and cite what you used.

Demos hide the hard parts. Production RAG fails on identifiers, stale docs, chunk boundaries, ACL leaks, and confident wrong citations.

flowchart TD
  Src[Documents / tickets / wikis] --> Ingest[Ingest + normalize]
  Ingest --> Chunk[Chunk + metadata]
  Chunk --> Emb[Embed]
  Chunk --> Lex[Lexical index]
  Emb --> Vec[Vector index]
  Q[User query] --> Rew[Optional rewrite]
  Rew --> Hybrid[Hybrid retrieve]
  Lex --> Hybrid
  Vec --> Hybrid
  Hybrid --> Rerank[Rerank]
  Rerank --> Pack[Context packer]
  Pack --> LLM[LLM + citations]
  LLM --> Eval[Offline + online eval]

Mental model: five blocks you must own

Block	Job	Failure if skipped
Ingest	Normalize, ACL, version, delete	Serving deleted / private docs
Chunk	Atomic, metadata-rich units	Mid-sentence splits; no filters
Retrieve	Hybrid sparse + dense	Misses SKUs / paraphrases
Rerank / pack	Precision + token budget	Noisy context → hallucinations
Cite + eval	Grounding + regression gates	“Looks good” demos that rot

Frameworks (LlamaIndex, LangChain) orchestrate these blocks — they do not replace owning them.

Step-by-step: one query through the system

1. User asks — optionally rewrite/expand (“HyDE,” multi-query) for recall.
2. Retrieve — BM25 + vector ANN → candidate set (e.g. 50).
3. Filter — tenant, ACL, recency, doc type.
4. Rerank — cross-encoder or vendor rerank → top 5–10.
5. Pack — respect token budget; preserve structure; include citation IDs.
6. Generate — instruct model to answer only from context; refuse if insufficient.
7. Return — answer + citations the UI can open.
8. Log — query, chunk IDs, scores, answer, user feedback → eval fuel.

Chunking that survives users

Goals: each chunk answers something alone; metadata enables filters; overlap preserves boundary sentences.

Strategy	Best for	Watch-outs
Fixed tokens (512–1024) + overlap	General prose	Blind to headings
Structure-aware (headings, HTML, Markdown)	Manuals, RFCs	Parser quality
Parent–child	Long narratives	More index complexity
Sentence / window	High-precision FAQ	May lose section context

Ship rule: store doc_id, section, updated_at, acl, source_url on every chunk. Deletes and ACL changes must flow to the index.

Hybrid search, not “just vectors”

Dense embeddings miss SKUs, error codes, and exact legalese. BM25 misses paraphrase. Fuse both (RRF or weighted fusion), then rerank.

flowchart LR
  Q[Query] --> BM25[BM25]
  Q --> Dense[Dense ANN]
  BM25 --> Fuse[Fusion RRF]
  Dense --> Fuse
  Fuse --> Rerank[Rerank]
  Rerank --> Top[Top chunks]

Deep dive: Hybrid search and rerankers, Chunking and metadata.

Citations are a product requirement

Return chunk- or span-level citations the UI can highlight. If the model cannot point at evidence, treat the answer as untrusted — especially support, legal, finance, health-adjacent content.

Patterns:

• Force according to [doc_id] style in the prompt and validate citations exist in retrieved set
• Show quotes in the UI from your store, not model-invented quotes
• Prefer extractive highlights for high-risk answers

How to build (minimal path)

1. One corpus (e.g. your README + FAQs).
2. Chunk + embed + BM25.
3. Simple packer + “answer only from context” system prompt.
4. 20 golden questions with expected chunk IDs / answer keys (Evals).
5. Only then add query rewrite, rerank, agents.

Agentic RAG (retrieval as a tool inside a loop) comes after one-shot RAG is eval-gated — see Agents and ReAct.

Tools today (2025–2026)

Layer	Examples
Orchestration	LlamaIndex, LangChain/LangGraph, custom pipelines
Embeddings	Voyage, OpenAI, Cohere, open-weight sentence models
Lexical	Elasticsearch/OpenSearch, Postgres FTS, Typesense
Vectors	pgvector, Chroma, Qdrant, Weaviate, Pinecone
Rerank	Cohere rerank, bge-reranker, cross-encoders
Eval	Golden sets, RAGAS-style metrics, LLM-as-judge with caution

Failure modes

Failure	Symptom	Mitigation
Wrong chunk boundaries	Answer half-right	Structure-aware chunking
Stale index	Contradicts UI of record	Versioning + reindex jobs
Citation hallucination	Fake quotes	Validate IDs; UI from store
Context stuffing	Vague / contradictory answers	Rerank + tighter budget
ACL bug	Cross-tenant leak	Metadata filters + tests
Eval theater	Only happy demos	Hard negatives + online feedback

Tradeoffs and when to use RAG

Use RAG when answers must reflect your private or changing knowledge.

Prefer fine-tuning when you need style/format/domain behavior more than factual lookup (often combine both).

Prefer tools/SQL when the truth lives in transactional systems (“order status”) — retrieve rows, don’t embed the whole DB blindly.

Prefer agents when multi-step tool choice is required after retrieval basics work.

Glossary

Term	Meaning
RAG	Retrieve evidence, then generate with it in context
RRF	Reciprocal Rank Fusion — combine ranked lists
Faithfulness	Answer supported by retrieved evidence
Recall@k	Fraction of needed docs found in top-k
Context packer	Selects/orders/truncates evidence for the prompt

Micro-project

Sketch a RAG pipeline for one real doc set you care about. Label ingest → chunk → indexes → retrieve → pack → cite → eval. Circle the riskiest failure mode.

Related guided path

Build end-to-end in RAG (embeddings → chunking → vector DB → hybrid → rerank → citation failures). Pair with Embeddings, Data track, and Examples on RAG citation failures.