Embeddings and similarity

What an embedding is

An embedding model maps text (or images, audio, code) to a fixed-size vector so that semantically similar inputs land close together under a chosen similarity (usually cosine or dot product after normalization).

LLMs contain embedding tables and hidden states internally. Product semantic search / RAG almost always uses a dedicated embedder — smaller, cheaper, specialized for retrieval — not the full chat model.

flowchart LR
  Doc[Chunk] --> E[Embedder]
  Q[Query] --> E2[Same embedder]
  E --> V[(Vector store)]
  E2 --> ANN[ANN search]
  V --> ANN
  ANN --> TopK[Top-k chunks]

Mental model

Think of a map where distance ≈ meaning distance:

• “refund policy” ≈ “money back rules”
• “refund policy” ≉ “Kubernetes rolling update”

Important: nearby ≠ correct. Embeddings retrieve candidates. You still need hybrid search, rerankers, filters, and evals for production quality (RAG building blocks).

Why LLMs need embeddings

Model weights cannot hold your private, changing corpus. Pattern:

1. Index time — chunk docs → embed → store vectors + metadata
2. Query time — embed query → ANN top-k → pack into prompt → generate

That is RAG. Embeddings are the retrieval geometry; the LLM is the reader/writer.

How similarity works

Metric	Typical use	Note
Cosine similarity	Text embeddings	Insensitive to vector length if normalized
Dot product	Often equivalent if L2-normalized	Match training/serving assumptions
L2 distance	Some vision / older pipelines	Do not mix casually with cosine indexes

Ship rule: the same embedder (and version) must embed queries and documents. Mixing models silently destroys recall.

flowchart TD
  Ingest[Documents] --> Chunk[Chunk + metadata]
  Chunk --> Emb[Embed with model vX]
  Emb --> Idx[(Vector index)]
  User[Query] --> QEmb[Embed with model vX]
  QEmb --> Search[ANN / hybrid]
  Idx --> Search
  Search --> Rerank[Optional rerank]
  Rerank --> Pack[Context packer]

Step-by-step: building a tiny semantic search

1. Collect 20–50 short docs or FAQ answers.
2. Chunk consistently (start simple: ~300–800 tokens with light overlap).
3. Embed with one model; store vectors in memory, Chroma, pgvector, etc.
4. Embed queries; retrieve top-k; manually label hits vs misses.
5. Find at least one false positive (high similarity, wrong meaning) — that teaches limits better than a blog post.

Engineer checklist

Choice	Why it matters
Same embedder for index + query	Mixing breaks geometry
Model domain (multi-lingual, code)	General English models fail on code/SKUs
Dimensionality	Storage, speed, quality tradeoff
Chunk size / overlap	See Data track → chunking
Metadata filters	Tenant, ACL, time — before ANN
Metric + normalization	Must match how vectors were trained
Re-embed on model upgrade	Indexes are not forever

Tools today (2025–2026)

Layer	Examples
APIs	OpenAI / Cohere / Voyage / Google embedding endpoints
Open-weight	MTEB-leading sentence models on Hugging Face; local via sentence-transformers
Stores	pgvector, Chroma, Qdrant, Weaviate, Pinecone, Redis vector
Hybrid	BM25 + vectors + rerankers (cross-encoders / vendor rerank APIs)
Eval	Golden queries with labeled chunks; recall@k, MRR

Browse: Vector databases, Chroma, Hybrid search.

Failure modes

Symptom	Cause	Direction
Misses exact IDs / SKUs	Dense-only retrieval	Add lexical / hybrid
Good demo, bad prod	No eval set; chunk junk	Golden queries + chunk audit
Sudden quality drop	Embedder version drift	Pin versions; reindex
Cross-tenant leakage	Missing ACL metadata filters	Filter before or in query
“Semantic” duplicates	Near-duplicate chunks	Dedupe; parent-child chunking

Tradeoffs

• Larger embedders — often better quality; more cost/latency at ingest and query.
• Short chunks — precise retrieval; may lack context → parent expansion.
• Long chunks — more context; noisier similarity.
• Cross-encoder rerank — quality boost; CPU/GPU cost on candidates only.

When to use embeddings (vs not)

Use when: search/FAQ/RAG, clustering tickets, deduping text, recommendation-by-similarity.

Don’t force when: exact match on IDs, strict keyword legal phrases (pair with lexical), or tiny corpora where grep + LLM is enough.

Glossary

Term	Meaning
Dense retrieval	Search with continuous vectors
ANN	Approximate nearest neighbor search
MTEB	Benchmark suite for embedding quality
Cross-encoder	Jointly scores (query, doc); strong reranker
Dimensionality	Length of the embedding vector

Micro-project

Embed 20 queries + docs; list nearest neighbors; plot or table one false positive and explain why cosine lied.

Related guided path

Guided Embeddings and similarity embeddings lab; Data track vector databases + chunking; Classical ML literacy for pre-LLM embedding intuition (ML/DL literacy); next concept often RAG building blocks.