LlamaIndex

What LlamaIndex is (plain English)

LlamaIndex is a data framework for LLM apps: connectors (readers), node parsers (chunking), indexes, retrievers, query engines, and agent abstractions aimed at “connect LLM ↔ private data.”

Complementary positioning:

You can mix: LlamaIndex for retrieval, your own loop or LangGraph for agents.

Analogy: LlamaIndex is an ORM for retrieval — it maps documents → nodes → indexes the way an ORM maps rows → models. You still own the database (vector store), ACLs, and evals.

flowchart LR
  Docs[Documents] --> Load[Readers / connectors]
  Load --> Nodes[Nodes = chunks + metadata]
  Nodes --> Idx[Index / vector store]
  Idx --> Ret[Retriever]
  Ret --> QE[Query engine]
  QE --> LLM[LLM + citations]

Interview cue: Be ready to explain Document → Node → Index → Retriever without naming LlamaIndex. The framework should be optional clothing on that skeleton.

The problem it solves

RAG glue code explodes: load PDFs, chunk, embed, store, retrieve, pack citations, retry. LlamaIndex packages that pipeline so teams iterate on retrieval quality instead of rewriting loaders.

It does not replace your vector DB ops, ACL model, or eval suite — it sits above them.

Architecture: core objects

flowchart TB
  Q[User query] --> Ret[Retriever top-k]
  Ret --> Pack[Context packer]
  Pack --> Synth[Response synthesizer]
  Synth --> Out[Answer + sources]
  VS[(Vector store: Chroma / pgvector / ...)] --> Ret

Index types you will actually meet

Start with vector + metadata filters. Add hybrid + rerank when recall@k plateaus — see Hybrid search and rerankers.

How it fits LLM apps

flowchart TB
  Ingest[Batch ingest workers] --> LI[LlamaIndex readers + nodes]
  LI --> Store[(Vector DB)]
  API[Product API] --> QE[Query engine / retriever tool]
  QE --> Store
  QE --> LLM[LLM]
  Eval[Golden set CI] --> QE

How to use (shape)

# Shape only — APIs evolve; pin versions
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(docs)
qe = index.as_query_engine(similarity_top_k=5)
print(qe.query("What is our refund policy?"))

Production-shaped additions you should still own:

• Metadata filters (tenant_id, doc_type)
• Stable node IDs for upserts
• Hybrid retriever + rerank (see Data track)
• Tracing spans around retrieve vs synthesize
• Explicit embed model id in the collection name

Retriever-as-tool for agents

# Shape — expose retrieval to ReAct / LangGraph without dumping full docs
def search_knowledge(query: str, tenant_id: str) -> str:
    nodes = retriever.retrieve(query)  # with where tenant_id=
    # return ids + short excerpts, not novels
    return format_excerpts(nodes, max_chars=1200)

Pair with LangGraph and LangChain patterns or a hand-rolled loop from Agents and ReAct.

When to pick LlamaIndex

Retrieval quality loop (what actually matters)

Framework speed is worthless without a measurement loop:

flowchart LR
  Gold[Golden questions] --> Ret[Retriever]
  Ret --> Hits[Hit rate / recall@k]
  Hits --> Chunk[Tune chunk + metadata]
  Chunk --> Ret
  Hits --> Rank[Add hybrid + rerank]
  Rank --> Ret

1. Freeze a golden set (question → relevant node_ids).
2. Log retrieved IDs every change.
3. Only then tune synthesizer prompts.

Data track depth: Chunking and metadata, Hybrid search and rerankers, Vector databases.

Walkthrough: policy wiki bot

1. Ingest Markdown policies with tenant_id + doc_type metadata.
2. Build VectorStoreIndex over Chroma or Postgres + pgvector.
3. Query with where tenant_id=….
4. Return answer + source node ids.
5. Eval: recall@5 on 50 golden questions before prompt fiddling.

Ingest and upsert discipline

sequenceDiagram
  participant Job as Ingest job
  participant LI as LlamaIndex
  participant VS as Vector store
  Job->>LI: load + parse nodes
  LI->>VS: upsert by node_id
  Note over VS: same id replaces; new id duplicates

Alternatives

Production gotchas

• Default chunking is rarely optimal for your corpus — measure recall@k
• Hidden prompts in synthesizers — version them like code (Weights & Biases / MLflow for LLMOps)
• Embed model changes without reindex → silent quality cliff
• ACL leaks if metadata filters are optional “nice to have”
• Framework upgrades — pin versions; LI APIs move
• Agent tool dump — returning full nodes into ReAct context blows budgets; return ids + excerpts (Context engineering)
• Chat engine memory — unbounded history without summarization
• Evaluating only answers — miss retrieval failures that cause hallucinations

flowchart TD
  Bad[Wrong answer] --> Split{Split blame}
  Split -->|Missed nodes| R[Fix retrieval / chunk / filters]
  Split -->|Good nodes, bad prose| P[Fix synthesizer / model]
  Split -->|Wrong tenant| A[Enforce metadata filters]

Failure modes checklist

1. Golden questions never written — “vibes” shipping
2. tenant_id filter omitted on one code path
3. Re-embed forgotten after model upgrade
4. Query engine citations dropped in UI
5. Agent receives 20 full pages per tool call

Hands-on next steps

1. Index a Markdown folder; ask 10 golden questions; log retrieved node IDs.
2. Rebuild the same index hand-rolled with Chroma; compare misses.
3. Add metadata filters and a reranker from the Data track.
4. Expose the retriever as a single tool inside a hand-rolled or LangGraph agent.

Micro-project

1. Build a 30-question golden set with relevant node ids.
2. Index with LlamaIndex defaults; measure recall@5.
3. Tune chunk size once; remeasure.
4. Hand-roll the same corpus in Chroma; compare miss lists.
5. Write a one-pager: keep LI, hand-roll, or hybrid.

Interview whiteboard: Document → Node → Retriever

Draw the skeleton without framework names first. Then put LlamaIndex as a dashed box around readers/parsers/engines. Interviewers want to know you can rebuild it with raw embeddings + Chroma/pgvector.

Synthesizer modes (what “answer” means)

Pick based on corpus size and latency SLO. Always return source node ids to the product layer.

Metadata filter patterns

where tenant_id = X AND doc_type IN (...) AND not deleted

Optional filters are ACL bugs waiting to happen. Make tenant filters mandatory in your wrapper — not a caller courtesy.

Observability split: retrieve vs synthesize

Log separately:

• retrieve_ms, hit_ids, embed_model
• synthesize_ms, model_id, prompt_version
• End-to-end answer score in evals

Otherwise you will “fix the prompt” when the retriever is blind. Pair with OpenTelemetry for LLMs.

Tradeoffs summary

Checklist

• Golden set with node ids
• recall@k measured before prompt changes
• Stable node ids + embed model in collection name
• Tenant filters mandatory in wrappers
• Synthesizer prompts versioned
• Agent tools return excerpts, not full nodes

Glossary

Debugging playbook (first hour)

Ship rule: split retrieve vs synthesize before you touch prompts. Most “LLM regressions” are retrieval regressions.

Anti-patterns

1. Framework = architecture — LI is clothing on Document → Node → Retriever.
2. Eval only final answers — you cannot tell miss vs hallucination.
3. Optional tenant filters — ACL bugs.
4. Default chunk sizes forever — measure recall@k on your corpus.
5. Dumping query engines into every agent step — budgets die.
6. Upgrading LI without pinning — silent API / default changes.
7. Chat engine as unbounded memory — summarize or truncate.

Security and privacy threat note

• Readers that pull SaaS sources can ingest secrets — scrub before index.
• Metadata often holds emails / ticket ids — treat as PII in logs.
• Multi-tenant: filter in the wrapper you own, not “hope the caller passes where.”
• Synthesizer prompts can leak system policy if logged raw — redact.

Interview prompts you should be able to answer

1. Draw Document → Node → Index → Retriever → synthesizer without naming LlamaIndex.
2. How do you measure retrieval quality separately from answer quality?
3. What breaks when you swap embedding models?
4. How would you expose retrieval to a ReAct agent without blowing the context window?
5. When do you keep LlamaIndex vs hand-roll Chroma/pgvector?

End-to-end: from folder to gated release

flowchart LR
  Src[Docs in git / drive] --> Ingest[Readers + node parsers]
  Ingest --> VS[(Vector store)]
  VS --> Ret[Retriever + filters]
  Ret --> Synth[Synthesizer]
  Synth --> API[Product API + citations]
  Gold[Golden set CI] --> Ret
  Gold --> Gate{recall@k OK?}
  Gate -->|yes| Ship[Ship prompt/retriever version]
  Gate -->|no| Tune[Chunk / hybrid / metadata]
  Tune --> Ingest

Ship only when recall@k (and a small answer faithfulness sample) beat baseline with lineage: embed_model, chunk_config, prompt_version, git SHA.

Production readiness checklist

• Pin llama-index* versions
• Collection / index name encodes embed model + dim
• Stable node ids + soft-delete
• Mandatory tenant filter in app wrappers
• Golden set with node ids in CI
• Separate retrieve vs synthesize metrics
• Agent tools return excerpts, not full nodes
• Synthesizer prompts versioned as artifacts

How it works end-to-end (request lifecycle)

1. Load documents via readers (local files, Notion, Slack, …).
2. Parse into nodes (chunks) with metadata and stable ids.
3. Embed nodes; upsert into the configured vector store.
4. On query: embed question → retrieve top-k (filters applied).
5. Synthesize with LLM (compact / refine / tree — pick deliberately).
6. Return answer + source nodes; log retrieve vs synthesize metrics separately.

sequenceDiagram
  participant App
  participant LI as LlamaIndex
  participant VS as Vector store
  participant LLM
  App->>LI: query
  LI->>VS: retrieve + filters
  VS-->>LI: nodes
  LI->>LLM: synthesize with context
  LLM-->>LI: answer
  LI-->>App: response + citations

Interface contract (staff bar)

Whether you use LlamaIndex or not, keep these seams:

Framework code should implement the seams — not own business authZ. Enforce tenant_id filters in your retrieve wrapper even if the index API is permissive.

Chunking and metadata (do not outsource judgment)

LlamaIndex node parsers are convenient; they are not a substitute for chunking and metadata strategy:

Ablate chunk sizes on a golden set before debating LlamaIndex vs LangChain.

Hybrid retrieval and rerank hooks

Production RAG often needs BM25 + vectors + rerank. LlamaIndex can compose retrievers; you can also retrieve in LI and rerank outside. Measure:

• recall@k vector-only vs hybrid
• nDCG after rerank
• added p95 latency

See Hybrid search and rerankers.

Agents vs query engines

Do not start with an agent if a query engine + citations solves the product. Agent loops multiply failure modes — pair with LangGraph when control flow needs cycles/HITL.

Observability fields

Worked walkthrough: internal wiki RAG

1. Reader loads docs/**/*.md.
2. Heading node parser; metadata path, section, tenant=internal.
3. Upsert to Chroma or pgvector.
4. Retriever top-8 with path prefix filter for space.
5. Compact synthesizer; require citations.
6. Golden 30 questions; gate release on recall@5 + faithfulness sample.

Memory, cost, and sizing intuition

FAQ (LlamaIndex)

Is LlamaIndex required for RAG?
No. It accelerates glue. Hand-rolled teaches the skeleton first.

LlamaIndex or LangChain?
Retrieval-heavy → LI often. Agent graphs → LangGraph. Mixing is fine.

Will LI lock me to one vector DB?
Not if you keep the interface contract and stable node ids.

Deep dive: evaluating synthesizers

Compact packs context once; refine iteratively rewrites; tree summarize hierarchical nodes. On long policies, refine can help — and burn tokens. Always A/B on your corpus with cost caps.

Putting what / why / how together

Architecture that survives production

flowchart TB
  subgraph ingest [Ingest workers]
    R[Readers]
    N[Node parsers]
    U[Upsert jobs]
  end
  subgraph online [Online path]
    Ret[Retriever wrapper + ACL]
    Syn[Synthesizer]
    GW[API gateway]
  end
  R --> N --> U --> VS[(Vector store)]
  GW --> Ret --> VS
  Ret --> Syn --> LLM[LLM provider]

Separate ingest from query processes. Nightly connector sync should not share thread pools with user Q&A. Version the node parser with the index; a silent parser bump without re-embed creates split-brain retrieval.

Index types you will actually touch

Start with vector + metadata filters. Add complexity only when evals demand it.

Consistency and freshness

SLA the lag: “searchable within N minutes” is a product promise.

Cost model for LlamaIndex stacks

Costs hide in: embedding calls on re-ingest, synthesizer multi-calls (refine), oversized top-k, and connector polling. Dashboard monthly $ by embed vs synthesize vs agent_tools. Cap refine iterations in config.

Interview whiteboard: replace LlamaIndex boxes

Draw Document → Node → Index → Retriever → Pack → LLM. Then redraw with LI class names. If you cannot erase the framework and keep the diagram, you do not understand the system yet.

Failure story bank

1. Connector token expired → empty index → model invents policy.
2. Tenant filter omitted in one query engine path → cross-tenant leak.
3. Refine synthesizer × long context → latency SLO burn.
4. Re-chunk without id scheme → duplicate nodes, weird citations.

End-to-end lab checklist (do this once)

• Index a Markdown folder; query with citations
• Ablate chunk size on 20 questions
• Enforce tenant filter in wrapper tests
• Split retrieve vs synthesize timings in logs
• Compare one hand-rolled retriever on the same corpus

Production readiness checklist (LlamaIndex)

• Node id scheme documented and tested for re-chunk
• Embed model revision pinned; collection/index named accordingly
• ACL/tenant filters enforced outside “happy path” demos
• Retrieve vs synthesize metrics and traces
• Synthesizer mode chosen deliberately (not default-by-accident)
• Eval suite gates releases (retrieval + answer quality)
• Ingest lag SLO defined if async
• Secrets for connectors rotated; least privilege
• Runbook: empty retrieval, embed outage, vector store down

What “good” looks like in a design doc

Name the corpus, chunker, embed model, vector backend, retriever config, synthesizer, citation UX, ACL model, eval suite, and rollback. “We’ll use LlamaIndex” is a library choice, not a design.

Common interview traps

• Claiming LlamaIndex replaces evals or vector DB ops
• Unable to explain nodes without framework jargon
• Ignoring metadata filters in multi-tenant designs
• Equating query engine with a safe agent

Security threat note (retrieval layer)

Retrieved text can contain prompt injection. Treat nodes as untrusted evidence. Bound tool use from RAG answers; cite sources; prefer read-only tools when context is untrusted HTML/email.

Glossary addendum

Micro-project stretch

After the base folder index: add a second embed model collection, shadow-query 20 questions, and write a one-pager on migration cost — without changing the public retrieve() interface.

Related

Core: RAG building blocks, Embeddings. Data: Vector databases, Chunking and metadata. Key tech: Chroma, LangGraph and LangChain patterns. Guided RAG.