Vector databases — what, why, and how

What a vector database is (plain English)

A vector database stores embeddings — lists of numbers that represent meaning — and answers one question fast:

Which stored items are most similar to this query embedding?

That is it. It does not “understand” your documents. It does not replace an LLM. It is a specialized index for similarity search over high-dimensional vectors, usually with metadata filters (tenant, ACL, doc type, time) so products can retrieve safe and relevant snippets.

Analogy: a relational DB answers “rows where status = 'paid'.” A vector DB answers “chunks whose meaning is near this question,” then your app packs those chunks into an LLM prompt.

One-sentence definition you can defend in an interview

A vector database is a persistence + ANN search system for embeddings and payloads, optimized for filtered nearest-neighbor queries that feed retrieval-augmented generation and agent memory.

If you cannot say ANN, payload/metadata, and filtered search, you are describing a numpy array, not a production vector store.

Why LLMs need this (the problem)

An LLM only “knows” what is in its weights and its context window. Your company wiki, tickets, contracts, and PDFs are in neither — unless you retrieve relevant snippets and put them in the prompt. That pattern is RAG (retrieval-augmented generation).

Keyword search (BM25) finds exact tokens. Embeddings turn text into vectors so paraphrase still matches. A vector database (or vector-capable search engine) stores those vectors and runs: which chunks are nearest to this query embedding?

Without a vector store (or equivalent ANN index), you either scan every embedding in memory (fine for demos) or fall back to keyword search alone (misses paraphrase and multilingual near-matches).

flowchart TD
  Docs[Documents] --> Chunk[Chunk + metadata]
  Chunk --> Emb[Embedding model]
  Emb --> VDB[(Vector DB / ANN index)]
  Q[User question] --> QEmb[Embed query]
  QEmb --> ANN[ANN search + filters]
  VDB --> ANN
  ANN --> Pack[Top-k chunks]
  Pack --> LLM[LLM answers with citations]

Interview cue: “Vector DB” is the data plane for semantic retrieval. Wrong chunks → confident wrong answers. Evals must measure retrieval separately from generation.

Why not “just put all docs in the prompt”?

Context windows grew, but products still need retrieval because:

1. Cost — stuffing 200 pages every turn is expensive
2. Attention — models dilute on huge noisy contexts
3. Freshness — index updates beat prompt paste jobs
4. Access control — you cannot dump every tenant’s docs into one window
5. Latency — retrieve top-k then generate beats shipping megabytes of tokens

Long context complements RAG; it does not delete the need for a retrieval layer in multi-tenant SaaS.

What a vector database actually does

Software that earns the name typically:

1. Persists high-dimensional vectors + payload metadata (text, source ids, ACLs, timestamps)
2. Builds an approximate nearest neighbor (ANN) index for fast similarity search
3. Supports upsert / delete, concurrent queries, and usually filtered search
4. Often exposes hybrid search (vectors + BM25/keywords) or plays nicely with one
5. Offers backups, replicas, metrics so ops can treat it like a real datastore

It is not a replacement for the LLM. It retrieves candidates; the LLM still generates. It is also not magic memory — garbage chunks with a perfect index still poison the prompt.

Vector DB vs “just Postgres” vs “numpy in RAM”

Ship rule: treat the vector index as a derived artifact. Canonical chunk text + metadata live in object storage / Postgres; you can always rebuild the ANN layer.

How it helps LLMs (concretely)

Agents use the same primitive for long-term memory — different collection design, same ANN + metadata story. Caching layers (exact / semantic) often sit in front of both the LLM and the vector store; see Redis for AI caching.

Three product jobs that share one primitive

flowchart LR
  subgraph SameANN[Same ANN + metadata machinery]
    RAG[RAG over docs]
    Mem[Agent long-term memory]
    Rec[Similar-item / semantic recommend]
  end
  RAG --> Pack[Pack into LLM]
  Mem --> Pack
  Rec --> UI[UI / ranking]

Do not invent three unrelated systems. Share embedding versioning, id schemes, and eval habits.

Embeddings and distance — the math you actually need

An embedding model maps text → (\mathbb{R}^d) (common (d): 384, 768, 1024, 1536, 3072). Similar meanings land nearby under a chosen metric:

Ship rule: the query and document embeddings must use the same model + version + metric. Changing any of those requires a full re-embed.

Some models use asymmetric prefixes (e.g. query: vs passage:). If the model card says so, apply prefixes consistently at ingest and query — that is part of the model contract, not optional flavor text.

# Shape only — pin model id in your index name
import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    a = a / (np.linalg.norm(a) + 1e-9)
    b = b / (np.linalg.norm(b) + 1e-9)
    return float(a @ b)

# Collection naming that survives model swaps
COLLECTION = "docs_v3_text-embedding-3-small_1536_cosine"

Dimensionality and quality intuition

Start with a strong general embedding API; ablate on your eval set before fine-tuning an embedder.

Exact vs approximate nearest neighbor (ANN)

Exact nearest neighbor (brute force) is (O(n \cdot d)) per query. At millions of vectors that is too slow or too expensive for interactive RAG.

ANN algorithms trade a little recall for large speed / memory wins. You tune for recall@k on your eval queries — not vendor defaults.

flowchart LR
  Q[Query vector] --> Exact[Exact: scan all]
  Q --> ANN[ANN: search subset / graph]
  Exact --> R1[100% recall · slow]
  ANN --> R2[High recall · fast]

Recall@k = fraction of true top-k neighbors (from exact search) that ANN also returns. Product quality needs task recall (gold chunks in top-k), which correlates with but is not identical to ANN geometric recall.

Index families you will see in job interviews

HNSW mental model: layers act like a skip list over a neighborhood graph. Search starts coarse (upper layers), descends, then expands neighbors at the bottom. Higher ef_search → better recall, higher latency. Higher M / ef_construct → better graph, more RAM and build time.

IVF mental model: train centroids on a sample; each vector belongs to a list. At query time, probe nprobe nearest lists and rank within them. Cheap to build relative to HNSW; recall depends heavily on training quality and nprobe. Under-trained centroids → silent recall cliffs.

Product quantization (PQ): split the vector into subspaces; replace each with a codebook id. Great for memory; combine with IVF (IVFPQ) or HNSW variants. Always re-check task quality — compression can blur near neighbors.

flowchart TD
  subgraph HNSW
    L2[Layer 2 long jumps] --> L1[Layer 1]
    L1 --> L0[Layer 0 dense neighbors]
  end
  subgraph IVF
    C[Coarse centroids] --> Lists[Inverted lists]
    Lists --> Rank[Exact or PQ rank in nprobe lists]
  end

HNSW search — step by step (whiteboard version)

1. Pick an entry point on the top layer (often a fixed or random hub).
2. Greedy walk: move to the neighbor closest to the query until no neighbor is closer.
3. Drop one layer down; repeat with a larger candidate set.
4. At the bottom layer, expand a candidate heap of size ef_search; return the best k.

Insert assigns a random max layer (geometric distribution), links the new node to M neighbors per layer, and may prune edges. Build quality is dominated by ef_construct (how hard you search while linking). Query quality is dominated by ef_search (how wide you explore at query time).

Ship rule: treat ef_search as a runtime dial you can A/B; treat M / ef_construct as a rebuild decision.

IVF + PQ — step by step (whiteboard version)

1. Train: sample vectors → run k-means → get nlist coarse centroids.
2. Assign: each vector goes into the inverted list of its nearest centroid.
3. Optional PQ: train codebooks per subspace; store codes instead of float32.
4. Query: find nprobe nearest centroids → scan those lists (exact or asymmetric PQ distance) → return top-k.

Worked intuition: 10M vectors, nlist=4096 → ~2.4k vectors per list if balanced. nprobe=32 scans ~77k candidates — far cheaper than 10M, but only if centroids match the data distribution. Retrain after a major corpus shift.

Index selection decision tree

Need interactive RAG at <~1–5M vectors, heavy filters, simple ops?
  └─ Start HNSW (or pgvector HNSW) in RAM

RAM / cost exploding above ~10–50M?
  └─ IVF+PQ or disk-backed (DiskANN-class); accept latency

Already on Elastic/OpenSearch and hybrid is mandatory?
  └─ Engine kNN + BM25 in one cluster; still measure recall@k

Lab / unit tests / <100k vectors?
  └─ Exact or tiny HNSW; do not over-engineer

Multi-region write-heavy with managed SLA?
  └─ Dedicated vector service — but keep Postgres as SoT

There is no universally “best” index. There is only recall@k vs p95 vs $/month on your filters and QPS.

Filtered ANN (the product-critical part)

Real products rarely want “globally similar.” They want:

tenant_id = X AND doc_type = policy AND updated_at > … AND acl ∩ roles ≠ ∅

Engines apply filters as pre-filter, post-filter, or constrained graph walk. Post-filter after global top-k can return empty results under strict ACLs. Prefer engines/docs that support filter-aware search, and load-test with realistic selectivity.

Ship rule: ACL bugs in retrieval are security bugs. Automate a test: tenant A query never returns tenant B payloads.

How it works end-to-end (ingest + query)

flowchart TB
  subgraph Ingest
    Src[Sources: Drive / S3 / Wiki] --> Parse[Parse + clean]
    Parse --> Chunk[Chunker]
    Chunk --> Meta[Attach metadata + stable ids]
    Meta --> EJob[Embed batch job]
    EJob --> Upsert[Upsert to vector store]
  end
  subgraph Online
    User[User / agent] --> GW[API gateway]
    GW --> EmbQ[Embed query]
    EmbQ --> Search[Filtered ANN ± hybrid]
    Search --> Rerank[Optional reranker]
    Rerank --> Prompt[Pack context + citations]
    Prompt --> LLM[LLM]
  end
  Upsert -.-> Search

Stable IDs and idempotent upserts

Stable chunk IDs are non-negotiable. Prefer something like hash(doc_id + chunk_index + embedding_model_id) or content-addressed ids so re-ingest is idempotent.

# Conceptual upsert — batch for throughput
points = []
for chunk in chunks:
    points.append({
        "id": chunk.stable_id,
        "vector": embed(chunk.text),
        "payload": {
            "text": chunk.text,
            "doc_id": chunk.doc_id,
            "tenant_id": chunk.tenant_id,
            "updated_at": chunk.updated_at.isoformat(),
            "acl_roles": chunk.acl_roles,
            "content_hash": chunk.content_hash,
        },
    })
client.upsert(collection="docs_v3_…", points=points)  # batches of 100–500

# Conceptual filtered search
hits = client.search(
    collection="docs_v3_…",
    query_vector=embed(user_query),
    filter={"must": [
        {"key": "tenant_id", "match": tenant_id},
        {"key": "acl_roles", "any": user_roles},
    ]},
    limit=40,  # retrieve wide; rerank/pack later
)

Retrieve wide, pack narrow

Interactive RAG rarely wants k=5 from ANN alone:

1. Retrieve 20–50 candidates (vector ± BM25)
2. Optional rerank to 5–10
3. Pack under a token budget with citations

That pipeline is covered in hybrid search and rerankers and chunking and metadata.

sequenceDiagram
  participant App
  participant Embed as Embedding API
  participant VDB as Vector DB
  participant LLM
  App->>Embed: embed(query)
  Embed-->>App: qvec
  App->>VDB: search(qvec, filters, k)
  VDB-->>App: top-k payloads
  App->>LLM: prompt + chunks
  LLM-->>App: answer + cite ids

Memory, cost, and sizing intuition

Rough KV-free vector RAM (uncompressed float32):

[ \text{bytes} \approx n \times d \times 4 + \text{index overhead} + \text{payload} ]

Example: 10M vectors × 1536 dims × 4 bytes ≈ ~61 GB raw — before HNSW graph overhead and payloads. This is why PQ / disk indexes / tiering show up at scale, and why “just put it all in HNSW RAM” fails budget reviews.

Also budget embedding API cost for initial backfill and ongoing CDC. A 50M-chunk re-embed is a project with a calendar, not a Tuesday evening.

Ops concerns product engineers hit

• Dimension / metric mismatch — silent garbage neighbors
• Embedding train/serve skew — different models or prompts for query vs doc
• Idempotent upserts — unstable ids → duplicates forever
• Deletes / tombstones / GDPR — mark deleted + filter; compact later
• Reindex budget — model upgrade = full re-embed + dual-write cutover
• Hot vs cold — RAM indexes vs disk; cost curves
• Multi-tenancy — collection-per-tenant vs shared + mandatory filter
• Backup / restore — snapshot volumes before bulk reingest
• SLO — p95 query latency and recall@k on a frozen eval set
• Payload bloat — storing megabyte blobs in the vector node

Dual-write cutover (embedding model change)

1. Create docs_v4_newmodel_…
2. Backfill async; keep serving v3
3. Shadow-query both; compare recall@k / nDCG on eval set
4. Flip traffic; keep v3 read-only until rollback window ends

flowchart LR
  V3[docs_v3 serving] --> Shadow[Shadow compare]
  V4[docs_v4 backfill] --> Shadow
  Shadow -->|pass gates| Flip[Flip read traffic]
  Flip --> Keep[Keep v3 for rollback]

Multi-tenancy patterns

Start shared + mandatory filters for MVPs under ~50 tenants; graduate when compliance or noisy-neighbor requires harder walls. Decision detail: Choosing vector stores.

Architecture that survives production

flowchart TB
  subgraph SoT[System of record]
    PG[(Postgres: docs, ACL, chunk text)]
    S3[(Object store: raw files)]
  end
  subgraph Derived[Derived indexes]
    VDB[(Vector DB)]
    BM25[(Optional search engine)]
  end
  subgraph Online
    API[RAG API] --> Cache[Redis exact/semantic]
    Cache --> Emb[Embed]
    Emb --> VDB
    Emb --> BM25
    VDB --> Fuse[Hybrid + rerank]
    BM25 --> Fuse
    Fuse --> LLM[LLM]
  end
  PG --> VDB
  S3 --> PG

Contract:

Sharding, replication, and HA (what “production” adds)

Most dedicated vector engines expose patterns you already know from search databases:

Ship rule: the vector tier is a derived cache. If you lose it, rebuild from SoT + embed jobs — not from “hope the snapshot was recent.”

flowchart LR
  subgraph Write
    W1[Ingest workers] --> P[Primary shard]
  end
  subgraph Read
    P --> R1[Replica A]
    P --> R2[Replica B]
    R1 --> API[RAG API]
    R2 --> API
  end

Change data capture (CDC) — keeping the index honest

Treat ingest as an event stream, not a one-time dump:

doc.updated → parse → chunk → content_hash changed?
  yes → embed batch → upsert vectors + payload
  no  → skip embed (save $)
doc.deleted → tombstone SoT → delete vectors (or filter deleted_at)
acl.changed → update payload only (no re-embed if text unchanged)

Wire ingest lag metrics: now() - doc.updated_at at first successful search. Product teams notice “doc not found” before ops notice CPU.

How to evaluate retrieval (not vibes)

Hold a frozen set of queries with labeled relevant chunk ids (or answer spans).

Ship rule: never ship a new embedder, chunker, or index knob without a before/after on the same eval set. Generation quality alone hides retrieval rot.

Three measurable metrics (staff bar)

1. recall@10 on held-out labeled queries (target: set a floor, e.g. ≥ 0.85 for your corpus)
2. p95 ANN latency under production-like filters
3. Cross-tenant leak rate = 0 in automated isolation tests

Two degrade modes

1. ANN / embedder outage → fall back to BM25-only or cached FAQ; tell the user answers may be weaker
2. Empty filtered retrieval → refuse to invent; ask clarifying question or escalate — do not let the LLM freestyle policy

When you need more than vectors

Vectors alone miss exact IDs, error codes, SKUs, rare proper nouns. Production RAG usually adds:

1. Hybrid BM25 + vector (hybrid search article)
2. Rerankers on a shortlist
3. Good chunking + metadata (chunking article)

Security and privacy threat note

See also Privacy and data for AI.

Failure modes checklist

• Embedding train/serve skew (different models or prefixes)
• Chunks too large → diluted similarity; too small → no context
• Filter too strict → empty retrieval → model improvises
• Post-filter after ANN under ACL → empty / biased hits
• Evaluating only final answers → blind to retrieval rot
• No citation ids → un-debuggable hallucinations
• Unfiltered multi-tenant search → data leak into prompts
• Dimension mismatch after silent config change
• Re-embed without dual-write → partial corpus on new model
• Payload-only deletes without index compaction → zombie hits

Debugging playbook (first hour)

1. Confirm model id + dim + metric match between ingest and query logs
2. Run the failing query with filters disabled in a secure staging clone — is it filter or geometry?
3. Compare exact top-10 vs ANN top-10 on a sample (recall cliff?)
4. Inspect chunk text — parse/chrome garbage?
5. Check content_hash / doc version — stale index?
6. Verify citations map to real doc_ids in SoT

Production readiness checklist

• Collection name encodes model + dim + metric
• Stable chunk ids + idempotent upsert
• Mandatory tenant/ACL filters in code review checklist
• Frozen eval set with recall@k gate in CI or weekly job
• Backup/restore drill for the vector volume
• Re-embed / dual-write runbook documented
• p95 latency + empty-retrieval alerts
• Export path for payloads (exit plan)

Tradeoffs and when to use a vector DB

Decision matrix (staff bar)

When a dedicated vector DB beats pgvector

• ANN QPS >> what one Postgres primary can serve under mixed OLTP load
• Filter-aware ANN is first-class and you have sparse ACL tenants
• Managed snapshots / autoscaling beat babysitting HNSW bloat
• Multi-region read replicas without hand-rolling shard merge

When pgvector (or search engine kNN) beats a dedicated vector DB

• Postgres already owns docs, ACL, and chunk text
• Corpus and QPS fit one HA Postgres cluster
• You need transactional consistency between rows and vectors
• Hybrid can be tsvector + vector in one SQL query

Detail lives in Choosing vector stores and Postgres + pgvector — this page is the why and how, not a vendor pick list.

Interview prompts you should be able to answer

1. Exact vs ANN — what do you trade, and how do you measure it?
2. Why can post-filtering ANN results fail ACL-heavy products?
3. Walk through a safe embedding-model upgrade.
4. How do you size RAM for 5M × 768-dim float32 + HNSW?
5. What belongs in the vector DB vs Postgres vs object store?

How embeddings are produced (pipeline detail)

sequenceDiagram
  participant Src as Source system
  participant Worker as Ingest worker
  participant Emb as Embedding API
  participant VDB as Vector DB
  Src->>Worker: doc create/update/delete
  Worker->>Worker: parse + chunk + hash
  alt content_hash unchanged
    Worker-->>Worker: skip embed
  else changed
    Worker->>Emb: batch embed texts
    Emb-->>Worker: vectors
    Worker->>VDB: upsert points
  end

Batching and backpressure

• Batch 50–200 texts per embed call when the API allows — cuts HTTP overhead.
• Cap concurrent embed workers so you do not melt provider RPM or the vector store write path.
• On delete events: tombstone in SoT first, then delete vectors (or filter deleted_at).
• Dead-letter failed chunks; do not silently drop them or recall rot is invisible.

Query-time embedding tricks

Collection design patterns

Never mix embedding models in one collection. Prefer a new collection name over an in-place dimension change.

Observability for the vector data plane

Log every retrieval:

query_id, tenant_id, model_id, collection, k, filter_hash,
hit_ids[], scores[], ann_ms, embed_ms, empty=true|false

Alert on:

• Empty retrieval rate spikes (filter or ingest break)
• p95 ann_ms regressions after index rebuilds
• Embedding error rate
• Collection count / disk unexpectedly flat during backfill (stuck job)

Trace ids should join retrieval spans to the LLM generation span so “wrong answer” tickets can show which chunks were packed.

Worked walkthrough: support policy RAG

1. Ingest: Confluence export → HTML clean → heading chunks (~500 tokens) → embed text-embedding-3-small → upsert Qdrant/pgvector with tenant_id, acl_roles, heading, doc_id.
2. Query: User asks “How long do refunds take?” → embed → filter tenant+role → top 40 → optional rerank → pack 6 chunks with citations.
3. Eval: 40 labeled questions; gate recall@10 ≥ 0.9 before launch.
4. Ops: Weekly sample of empty retrievals; dual-write plan documented for next embed model.

Anti-patterns (vector DB edition)

• One vector per entire PDF “to keep it simple”
• Storing embeddings without chunk text (cannot cite or rebuild prompts)
• Sharing a collection across tenants without mandatory filters
• Tuning ef_search on vibes with no recall@k
• Treating Pinecone/Qdrant as SoT for document bodies
• Re-embedding in place without versioned collections

How ANN search feels in practice (timing)

A typical interactive RAG query budget for the retrieval slice:

If embed is remote and slow, cache query embeddings for identical strings (short TTL) — separate from answer caching.

Consistency models you will actually hit

Document freshness SLOs (“new doc searchable within N minutes”) and measure ingest lag — not just query latency.

Agent memory vs RAG collections

Ship rule: do not dump raw tool traces into the same collection as customer policy docs without type filters and separate evals.

Interview whiteboard: design a vector layer

Prompt: “Design retrieval for a 50-tenant SaaS with 5M chunks.”

A strong answer covers:

1. SoT in Postgres; derived ANN (pgvector first or Qdrant)
2. Mandatory tenant filters + isolation tests
3. Chunking + metadata schema
4. Hybrid for IDs; rerank optional
5. recall@k eval harness
6. Re-embed dual-write runbook
7. Redis exact cache in front
8. Graduation criteria to dedicated ANN

Weak answers only name a vendor.

Glossary

Hands-on next steps

1. Choosing vector stores — Pinecone / Weaviate / Chroma / pgvector / Qdrant
2. Postgres + pgvector — one-DB path
3. Guided Vector DB — Docker Compose + ingest + query
4. Hybrid search and Rerank / rewrite (vectors alone are not enough)

End-to-end lab checklist (do this once)

1. Spin up Qdrant or Postgres+pgvector locally.
2. Ingest 100 docs with stable ids + tenant metadata.
3. Run 20 labeled queries; compute recall@5 exact vs ANN.
4. Break it on purpose: wrong metric, missing tenant filter, huge chunks — observe symptoms.
5. Fix with versioned collection name + mandatory filter helper + chunk ablation.
6. Add Redis exact cache in front of one FAQ route.
7. Write a one-page runbook: re-embed + dual-write.

That sequence teaches what / why / how better than reading vendor pages alone.

What “good” looks like in a design doc

A staff-level vector design doc usually includes:

• Corpus size, growth, and delete rate
• Embedding model + dim + metric + versioning scheme
• Chunking strategy + ablation results
• Index type + knobs + recall@k gate
• Tenancy / ACL approach + isolation tests
• Hybrid / rerank plan (or explicit deferral with date)
• SLOs: ingest lag, p95 query, empty-rate
• Cost model: embed + storage + QPS
• Exit plan: rebuild from SoT in another store

If any bullet is missing, the design is not ready for production review.

Common interview traps

Putting what / why / how together

If you can teach those three rows with a diagram and a failure story, you are past buzzword RAG.

FAQ (vector databases)

Do I need a vector DB if I use a 1M-token model?
Usually yes for multi-tenant ACL, cost control, and freshness. Long context reduces how often you retrieve, not whether you need a retrieval layer.

Is FAISS a vector database?
FAISS is a library for similarity search. You still need persistence, filters, ops, and an API around it for most products.

Can I store vectors in S3 and scan them?
Only for offline batch jobs. Interactive RAG needs an ANN index in memory or a specialized engine.

What recall@k should I target?
Task-dependent. Start by measuring; many teams gate launch on recall@10 ≥ 0.85–0.95 on a frozen set — then watch for regressions.

How often do I re-embed?
When the embedding model changes, or when chunk text changes (content_hash). Not on a calendar alone.

Deep dive: filtered search under sparse ACLs

Imagine 1M vectors, but a user may only access 0.5% of them. If the engine runs global ANN top-50 then post-filters, expected surviving hits ≈ 0.25 — often empty. The model then freestyles.

Mitigations:

1. Prefer filter-aware ANN (constrained graph / prefilter) documented by the vendor.
2. Retrieve larger k when post-filter is unavoidable — still a blunt tool.
3. Maintain per-tenant collections for tiny tenants with ultra-strict ACLs.
4. Measure empty-rate by tenant — averages hide the sparse tenants.

Load-test with production-like selectivity, not uniform random filters.

Deep dive: embedding model upgrades without downtime

Treat the collection as immutable versioned artifacts:

1. docs_v3_modelA_1536_cosine serves production.
2. Build docs_v4_modelB_3072_cosine from SoT in a worker fleet.
3. Shadow 1–5% of queries to v4; log hit-id Jaccard vs v3 and eval recall.
4. Flip reads via config flag; keep v3 warm for 48–72h.
5. Only then delete v3 after backup.

Never “UPDATE embedding SET vector = …” in place across mixed dims.

Interface contract (staff bar)

Write this in the design doc before you pick a vendor:

If the store cannot state those four rows, you do not have a production data plane — you have a demo notebook.

Three metrics that prove the layer works

1. recall@k on a frozen labeled set (gate releases)
2. empty-hit rate by tenant / ACL selectivity (sparse tenants)
3. p95 retrieval latency (embed + ANN + hydrate), separate from LLM TTFT

Two degrade modes when the happy path fails

1. Keyword / hybrid fallback — serve BM25 hits when ANN is down or empty under filters; label the path in traces
2. Stale-safe cache / last-good pack — Redis exact-answer or last successful chunk pack with TTL + “may be stale” UI, never silent invent

Threat note: a poisoned document or prompt-injection in retrieved text can steer the LLM. Catch it at ingest allowlists, payload ACL filters, output citations, and eval / HITL gates — not inside the ANN index.

What / why / how — worked mini-lab (30 minutes)

Do this once so the abstractions stick:

1. What: embed three sentences with the same model; print cosine between (A, paraphrase-of-A) vs (A, unrelated). Confirm paraphrase is nearer.
2. Why: ask an LLM the same question with and without those two nearest chunks in context — watch hallucination disappear when evidence is present.
3. How: insert the three vectors into any store (Chroma local is fine); query with a paraphrase; assert top-1 id matches. Then add a tenant_id filter that excludes the true hit and confirm empty or wrong — that is your ACL lesson.

Ship the notebook + a one-page diagram. That portfolio artifact beats “I used Pinecone” on a resume.

Sizing worksheet (fill before buying hardware)

Refuse “we will tune later” as a sizing strategy. Later is an incident.

Vendor landscape (names, not a pick list)

Every product below implements the same contract: persist vectors + payloads, ANN search, filters, upsert/delete. Differences are ops model, hybrid story, and filter semantics — not magic retrieval quality.

Ship rule: pick on SoT fit, filter semantics, hybrid needs, exit plan — then run the same recall@k harness on two finalists. Marketing pages do not replace your eval set.

Micro-project

Draw (or Mermaid) the full path document → chunk → embedding → ANN → pack → LLM. Label at least five failure points (bad chunk, wrong model, filter miss, low recall, no citation). Bring that diagram to the RAG lab.