Chroma

What Chroma is (plain English)

Chroma is a developer-friendly vector database aimed at local-first and small-app RAG: collections of embeddings + documents + metadata, with similarity query and filters. You can run it embedded in-process or as a client/server.

It shines for labs, notebooks, and early products. Many teams later graduate to pgvector, Weaviate, Pinecone, or another store — but the RAG mental model transfers.

Analogy: SQLite for vectors. Perfect for learning and early apps. You graduate when HA, multi-tenant scale, or hybrid search ops demand a heavier store — not because Chroma “isn’t real.”

flowchart LR
  Chunks[Chunks + metadata] --> Emb[Embedding model]
  Emb --> Col[(Chroma collection)]
  Q[Query] --> QEmb[Embed query]
  QEmb --> Search[Similarity + where filter]
  Col --> Search
  Search --> LLM[LLM context]

Interview cue: Chroma teaches the collection / id / embedding / metadata / query contract. Be ready to redraw that on any vector store.

The problem it solves for LLMs

Without a vector store you either scan all embeddings in memory or rely on keywords alone. Chroma gives you:

1. Persist vectors across restarts (local path / server)
2. Top-k semantic retrieval for RAG
3. Metadata filters (tenant_id, doc_type, …)
4. A gentle API for learning ANN retrieval before ops complexity

It does not generate answers — it retrieves candidates. The LLM still synthesizes (and can still hallucinate if retrieval is empty or wrong).

Deep foundation: Vector databases, Embeddings, RAG building blocks.

Architecture

flowchart TB
  subgraph ingest [Ingest]
    D[Doc] --> C[Chunk]
    C --> E[Embed]
    E --> U[Upsert ids]
  end
  subgraph query [Query]
    Q[Question] --> EQ[Embed]
    EQ --> ANN[Query collection]
    ANN --> Top[Top-k docs]
  end
  U --> Store[(Chroma)]
  Store --> ANN

Embedded vs client/server

Ship rule: if your web server forks workers, use client/server — not N in-process DBs fighting one path.

How to use

# Shape only — pin chromadb version in labs
import chromadb

client = chromadb.Client()  # or HttpClient for server mode
col = client.get_or_create_collection("policies")

col.upsert(
    ids=["policy-refund-001"],
    documents=["Refunds within 30 days require order id ..."],
    metadatas=[{"tenant_id": "acme", "doc_type": "policy"}],
    embeddings=[[0.1, 0.2, 0.3]],  # or use an embedding function
)

hits = col.query(
    query_embeddings=[[0.11, 0.19, 0.29]],
    n_results=5,
    where={"tenant_id": "acme"},
)

Ship rule: never invent new IDs on every ingest — upsert by stable chunk keys or you duplicate forever.

Collection naming that survives model swaps

policies_v3_text-embedding-3-small_1536_cosine

Include embed model + dim + metric. Changing any of those ⇒ new collection + full re-embed.

How Chroma fits LLM apps

flowchart TB
  App[RAG / agent service] --> Ch[(Chroma)]
  Ingest[Ingest job] --> Ch
  App --> LLM[LLM]
  Ch --> App

Frameworks: LlamaIndex often sits above Chroma for readers/query engines — you can also call Chroma directly for clarity.

When to use Chroma vs graduate

See Data track: Choosing vector stores, Postgres + pgvector.

Walkthrough: 50-chunk policy lab

1. Chunk policies with overlap; attach tenant_id, doc_type, source_path.
2. Embed with a pinned model; upsert stable ids path#chunk_i.
3. Query with where={"tenant_id":"acme"}.
4. Delete one doc’s ids; re-upsert; confirm count unchanged.
5. Measure recall@5 on a golden set before changing prompts.

sequenceDiagram
  participant Dev
  participant Ch as Chroma
  participant Emb as Embed model
  Dev->>Emb: embed chunks
  Dev->>Ch: upsert ids + meta
  Dev->>Emb: embed question
  Dev->>Ch: query + where
  Ch-->>Dev: top-k docs

Alternatives

Production gotchas

• Embedding model swap without re-embed → garbage neighbors
• Missing where filters → cross-tenant leaks
• Huge documents in payload → context bloat; store pointers + tight chunks
• Embedded mode in multi-worker web servers — use client/server, not N in-process DBs
• No recall measurement — always keep a golden query set
• Duplicate ids from bad keying — index bloat + confusing hits
• Metric mismatch — cosine vs L2 vs IP vs how embeddings were trained
• Treating Chroma as long-term memory without retention policy

flowchart TD
  Leak[Cross-tenant hit] --> F[where filter missing]
  Dup[Duplicate chunks] --> I[Non-stable ids]
  Junk[Random neighbors] --> E[Embed model changed]
  Slow[Multi-worker corruption] --> M[Embedded mode misuse]

Failure modes checklist

1. query without where in a multi-tenant app
2. New UUID every ingest
3. Documents = entire 50-page PDF
4. Eval only on generated answers, never retrieval ids
5. Prod still on laptop path ./chroma_data with no backups

Hands-on next steps

1. Ingest ≥50 chunks with metadata; query with a filter.
2. Delete/re-upsert one doc; confirm no duplicates.
3. Compare the same queries on pgvector later.
4. Guided Vector DB.

Micro-project

1. Build collection policies_v1_<embed>_<dim>_cosine.
2. Upsert 50 chunks; run 10 golden queries; log ids.
3. Change chunk size; rebuild; compare miss lists.
4. Write “graduate when…” criteria for your app (QPS, tenants, HA).

Interview whiteboard: the five fields

Any vector store answer should name:

1. Collection
2. Stable id
3. Embedding (+ model version)
4. Document / payload
5. Metadata + where filter

Then draw ingest and query as separate paths sharing the collection.

Distance metrics and embed models

Changing embed model without rebuild is a silent outage for RAG quality.

Hybrid search graduation path

Chroma is enough to learn dense retrieval. When keyword precision matters (IDs, error codes), plan hybrid:

1. Dense top-k from Chroma
2. BM25/keyword from your search stack
3. Rerank → pack LLM context

Data track: Hybrid search and rerankers.

Backup and local ops

Even “just a lab store” needs:

• Path on durable disk
• Periodic copy of persistence dir
• Recreate-from-source ingest job (source of truth is docs, not only vectors)

If you cannot rebuild the index from documents + embed model id, you do not have a recovery plan.

Tradeoffs summary

Checklist

• Stable ids on upsert
• Mandatory tenant where in app wrappers
• Collection name encodes embed model
• Golden recall@k suite
• Client/server if multi-worker
• Rebuild-from-source documented

Glossary

Debugging playbook (first hour)

Anti-patterns

1. New UUID every ingest — duplicates forever.
2. Entire PDF as one document — retrieval unit too coarse.
3. Optional ACL filters — security theater.
4. Embedded mode in forked web servers — corruption / split-brain.
5. Vectors as source of truth — you cannot rebuild without docs.
6. Eval only on chat answers — miss retrieval failure.
7. Cosine collection + L2-trained embed assumptions — silent quality loss.

Security and privacy threat note

• Metadata is searchable and often PII — minimize fields; encrypt at rest if required.
• Document payloads in the store are readable by anyone with DB access — treat like a datastore.
• Delete means delete ids and stop serving soft-deleted flags if you use them.
• Lab paths on laptops are not backups — and not HIPAA/SOC2 “controls.”

Interview prompts you should be able to answer

1. Name the five fields of a vector item (collection, id, embedding, document, metadata).
2. Why does changing the embed model require a new collection?
3. When do you graduate from Chroma to pgvector / a dedicated vector DB?
4. How do you prove upsert did not duplicate?
5. How does Chroma fit under LlamaIndex without becoming a black box?

End-to-end: lab → small product

flowchart TB
  Docs[Source docs] --> Chunk[Chunk + metadata]
  Chunk --> Emb[Pinned embed model]
  Emb --> Upsert[Upsert stable ids]
  Upsert --> Ch[(Chroma server)]
  API[RAG API] --> Wrap[where tenant mandatory]
  Wrap --> Ch
  Ch --> Pack[Top-k excerpts]
  Pack --> LLM[LLM + citations]
  CI[Golden recall@k] --> Ch

Graduation trigger examples: need HA replicas, heavy hybrid ops, multi-region, or the team already runs Postgres as home (Postgres + pgvector).

Production readiness checklist

• Client/server if multi-worker
• Collection name: embed model + dim + metric
• Stable ids + rebuild-from-source job
• Mandatory tenant where in wrappers
• Golden recall@k suite
• Persistence path on durable disk + backup copy
• Retention / soft-delete policy
• Documented graduate-when criteria

How it works end-to-end (request lifecycle)

1. Create/open a collection (name encodes embed model + dim).
2. Embed chunks; upsert with stable ids + metadata.
3. Query: embed question → similarity search + where filters.
4. Return documents/metadatas/distances to the packer.
5. LLM answers with citations bound to ids.

flowchart TD
  Upsert[upsert ids+emb+meta] --> Col[(Collection)]
  Q[Query text] --> QE[Query embed]
  QE --> Search[query + where]
  Col --> Search
  Search --> Pack[Pack top-k]
  Pack --> LLM

Collection naming and versioning

wiki_emb-bge-small-v1_dim384
tickets_emb-bge-small-v1_dim384

When the embedding model changes, new collection + backfill — do not mix dimensions or spaces. Record embed model revision in ops docs.

Metadata filters and multi-tenancy

Always filter tenant_id (and often acl / space). Similarity without filters is a data leak waiting to happen.

Upsert discipline

• Stable ids from hash(doc_id, chunker_version, span)
• Upsert > delete+add for most updates
• Tombstone or delete on source removal
• Prove no duplicates in the micro-project

Local vs client/server

Neither replaces HA multi-region product stores. Graduate criteria belong in the design doc.

Hybrid and graduation path

Chroma is primarily dense retrieval. When SKUs/error codes dominate failures, add BM25 (hybrid article) or graduate to Weaviate/Elastic/OpenSearch/pgvector+tsvector. Keep chunk ids stable across the migration.

Observability

Log collection, top_k, filter, hit_ids, distances, embed_model, query_ms. Empty hits should be a first-class metric — silent empty retrieval → hallucinated answers.

Worked walkthrough: 50-chunk policy lab

1. Chunk 10 Markdown policies with heading splitter.
2. Upsert to Chroma with doc_type=policy.
3. 15 golden questions; measure recall@5.
4. Ablate chunk size; keep winner.
5. Write graduate-when: “>X vectors or need HA/hybrid.”

FAQ (Chroma)

Is Chroma production-ready?
For many early products yes; for regulated multi-tenant HA maybe not. Decide with criteria, not vibes.

Chroma vs pgvector?
Chroma: fastest learning loop. pgvector: SQL/ACL joins with existing Postgres.

Do I need LlamaIndex?
Optional — see LlamaIndex.

Deep dive: distance metrics

Cosine vs L2 vs IP must match how embeddings were trained/normalized. Mismatch quietly tanks recall. Pin metric next to embed model in the collection name docs.

Putting what / why / how together

Architecture that survives early production

flowchart TB
  subgraph writers [Writers]
    Ingest[Ingest job]
  end
  subgraph readers [Readers]
    API[Chat API]
  end
  Ingest --> Ch[(Chroma server)]
  API --> Ch
  API --> Emb[Embed service]
  Ingest --> Emb

Use one embed service for write and read. Divergent embed models are the classic silent RAG break.

Capacity intuition

Benchmark your query filter selectivity — sparse ACL filters change ANN behavior.

Backup / restore drill

1. Snapshot persistence directory or server volume.
2. Restore to a scratch instance.
3. Run golden queries; compare hit ids.
4. Document RPO/RTO even for “just a lab DB.”

Migration playbook off Chroma

1. Freeze writes or dual-write.
2. Export ids, documents, metadatas, embeddings (or re-embed).
3. Import to target store with same ids.
4. Shadow query diff on golden set.
5. Flip read traffic; keep Chroma read-only briefly.

Stable ids make this boring — which is the goal.

Interview whiteboard

Five fields: id, embedding, document, metadata, collection. Draw filter before ANN. State graduate-when criteria aloud.

Failure story bank

1. Re-embed with new model into same collection → garbage distances.
2. Forgot where={tenant} in one code path.
3. Local ephemeral container lost volume → empty prod.
4. top_k=50 packed into 8k context → truncated citations.

End-to-end lab checklist (do this once)

• Persistent volume configured
• Upsert idempotency proven
• Filter unit tests
• Golden recall@5 recorded
• Graduate-when paragraph in README
• Backup restore once

Production readiness checklist (Chroma)

• Persistence volume durable across restarts
• Collection naming encodes embed model + dim
• Upsert ids stable; duplicate test in CI
• Tenant/ACL filters on every query path
• Backup/restore drill completed once
• Golden recall@5 baselined
• Empty-hit metric + alert
• Graduate-when criteria written
• Embed service single-sourced for read/write

What “good” looks like in a design doc

Collection schema, id scheme, filter fields, embed model pins, persistence/backup, expected vector count growth, and the next store you’ll migrate to (with why). “Chroma for now” needs an exit ramp.

Common interview traps

• Treating Chroma as interchangeable with keyword search
• No metadata → no multi-tenant story
• Re-embedding into the same collection blindly
• Claiming ANN is exact nearest neighbor

Distance, recall, and honesty

ANN is approximate. Measure recall@k against brute force on a sample periodically. If filters are ultra-selective, verify the engine isn’t returning nonsense under empty candidate sets.

Glossary addendum

Micro-project stretch

Export your 50-chunk lab to a JSONL of {id,document,metadata,embedding}; re-import into a second Chroma collection; diff query hit ids — rehearsal for store migration.

How to evaluate Chroma retrieval

1. Freeze a golden set of (query → relevant chunk ids).
2. Measure recall@k and MRR.
3. Slice by doc_type and tenant.
4. Re-run after chunker or embed changes.

Do not judge Chroma by LLM answer vibes alone — split retrieval metrics. See Vector databases for ANN context.

Metadata cardinality pitfalls

High-cardinality filters (per-user ids on millions of rows) can hurt. Prefer tenant-level collections or composite indexes as you scale. Document the filter fields you promise to product.

Embedding cache for ingest

When re-running ingest, cache embeddings by content_hash in Redis/disk to save $. Invalidation: chunker version bump clears cache keys. This matters once corpora leave toy size.

Local RAG with Ollama

Pair Chroma with Ollama for offline demos: retrieve → pack → local generate. Same ids and filters you will use in prod; only the LLM base_url changes.

Putting what / why / how together

Ops metrics worth graphing

Alert on empty-hit spikes after deploys — usually embed mismatch or filter bugs.

Tenant isolation test (automate)

Create two tenants’ chunks; query as tenant A with filter; assert zero B ids across 100 random queries. Run in CI against ephemeral Chroma.

Hands-on stretch: hybrid sidecar

Keep Chroma for dense; add Postgres tsvector or a tiny OpenSearch for BM25 on the same chunk ids; fuse with RRF in 40 lines of Python. This teaches graduation without abandoning Chroma prematurely.

FAQ addendum

Can Chroma replace Redis caching?
No — different job. See Redis for AI caching.

Should I store raw PDFs in Chroma?
No. Store chunk text + metadata; PDFs in object storage.

Sizing worksheet (fill before “prod”)

If you cannot fill this, you are guessing — fine for a lab, not for a launch review.

Citation packing reminder

Chroma returns chunks; your packer must: respect token budgets, prefer neighbors via chunk_index, and never drop source metadata. Bad packing looks like a vector-DB failure in demos.

Putting the lab on a resume (honest claims)

Good: “Built filtered RAG over N docs with Chroma; measured recall@5; documented graduate-when criteria.” Bad: “Implemented production vector database at global scale” for a laptop collection.

Related

Data track: Vector databases, Choosing vector stores, Chunking and metadata. Key tech: LlamaIndex. Core: RAG building blocks, Embeddings. Guided RAG.