Embeddings and similarity

Vectors before "RAG framework"

Retrieval-augmented generation (RAG) adds external knowledge to LLM prompts. Before chunking strategies, vector databases, and rerankers, there is a simpler primitive: embeddings — fixed-size vectors that represent text meaning in a geometric space where similar ideas sit close together.

If two sentences paraphrase each other, their embedding vectors should point in nearly the same direction. Unrelated sentences should be far apart. Semantic search is linear algebra on those vectors, not keyword matching.

You saw embedding intuition in earlier modules (static word vectors, learned representations). Here embeddings become a product building block you will index, query, and compose into full RAG pipelines.

Callout — embeddings are not magic: They encode statistical similarity from training data, not truth. Wrong-but-confident retrieval is normal without hybrid search and evals.

How embedding models work (operationally)

An embedding API or local model maps text → vector ∈ ℝⁿ (common dimensions: 384, 768, 1536). Models are trained with contrastive or predictive objectives so co-occurring or paraphrase pairs align in space.

Typical workflow:

1. Offline: embed each document (or chunk) once; store vectors + metadata.
2. Online: embed the user query.
3. Rank: score every stored vector against the query vector.
4. Generate: pass top-k text chunks to an LLM (later lessons).

Popular choices for learning:

• API: OpenAI text-embedding-3-small, Voyage, Cohere embed endpoints.
• Local: sentence-transformers/all-MiniLM-L6-v2 — fast, CPU-friendly, good for tutorials.

Pick one and stay consistent within your project; switching models invalidates your index.

Similarity metrics

Given query vector q and document vector d:

Cosine similarity — measures angle, ignores magnitude:

[ \text{cos}(\mathbf{q}, \mathbf{d}) = \frac{\mathbf{q} \cdot \mathbf{d}}{|\mathbf{q}| |\mathbf{d}|} ]

Range [-1, 1] for arbitrary vectors; often [0, 1] for normalized embedding APIs. Default choice when vectors are L2-normalized.

Dot product — equivalent to cosine when vectors are unit length; faster on some hardware.

Euclidean distance — small distance = similar; use when vectors are not normalized.

Most vector DBs expose cosine or dot product inner search. For your first index, normalize vectors and use cosine — results are interpretable and comparable across queries.

Semantic search behavior

What works well:

• Paraphrases: "refund policy" ↔ "how do I get my money back"
• Conceptual overlap: "Kubernetes pod crash" ↔ "container restart loop"
• FAQ-style questions with varied wording

What fails:

• Exact identifiers: SKUs, ticket IDs, error codes — lexical search wins (hybrid lesson).
• Negation: " flights without layovers" may still rank layover-heavy docs.
• Out-of-domain queries: embeddings return something nearby — often wrong.
• Temporal facts: "latest pricing" without metadata filters.

Document one paraphrase success and one failure in your project notes — required for the micro-project.

Building a minimal index

In-memory index for ≤ few thousand docs:

import numpy as np

class MemoryIndex:
    def __init__(self):
        self.ids: list[str] = []
        self.vectors: list[np.ndarray] = []
        self.texts: list[str] = []

    def add(self, id: str, vector: np.ndarray, text: str):
        self.ids.append(id)
        self.vectors.append(vector / np.linalg.norm(vector))
        self.texts.append(text)

    def search(self, query_vec: np.ndarray, k: int = 5):
        q = query_vec / np.linalg.norm(query_vec)
        mat = np.stack(self.vectors)
        scores = mat @ q
        top = np.argsort(scores)[::-1][:k]
        return [(self.ids[i], float(scores[i]), self.texts[i]) for i in top]

Persist ids, vectors, and texts to disk (numpy .npz + jsonl) so rebuild is not required every query.

Choosing an embedding model

Decision factors for your first index:

Dimension vs speed: Higher dimensions (1536) can capture nuance but increase index size and query cost. MiniLM-class 384-d models often suffice for FAQ-scale corpora under 10k chunks.

Language coverage: Multilingual models (e.g. paraphrase-multilingual) when queries mix English and Hindi; English-only models misalign cross-lingual pairs.

Domain fit: General models work for mixed prose; code and legal corpora benefit from domain-tuned embedders if available.

API vs local: APIs offload GPU ops and track model updates; local models eliminate per-token embed fees and keep data on-prem.

Re-embedding entire corpus on model change is expensive — treat model choice as a versioned decision (embed_model=voyage-2) logged alongside index builds.

Normalization and numerical stability

If your API returns unnormalized vectors, always L2-normalize before cosine search. Near-zero vectors (empty string embed attempts) produce NaNs — filter empty documents at ingest.

For debugging, print pairwise cosine between a query and the top hit vs a random document — sanity check that scores spread sensibly (e.g. top 0.82, random 0.12, not everything 0.4).

End-to-end retrieval sketch

docs.jsonl  ──embed──▶  index.bin
                              ▲
user query ──embed── query_vec─┘
                              │
                         top-5 chunks
                              │
                         (later) LLM answer

This lesson stops at ranked titles + scores. Generation with citations comes after chunking and grounding lessons.

Callout — measure before optimizing: Print top-5 for ten hand-written queries before tuning chunk size or buying a bigger embedding model. Qualitative scan saves weeks.

Engineering problem (staff framing)

RAG quality starts with embedding space geometry and similarity choice.

Diagram — Query → neighbors

flowchart LR
  Q[Query] --> E[Embed] --> S[Similarity] --> TopK[Top-k docs]

Precise definitions & mental model

Bi-encoders, cosine vs dot, asymmetric query/doc models.

Tradeoffs — when to use what

Bi-encoder recall vs cross-encoder precision (rerank).

Failure modes (interview + on-call)

Embedding stale docs; query language ≠ index language.

Production & OSS practices

Embedding version pinned to index; dual-write on migrate.

Micro-project: Semantic search

In m5/semantic_search/:

1. Index ≥50 short documents (FAQ entries, README sections, synthetic paragraphs).
2. Use a small embedding API or local sentence-transformer.
3. CLI: query "..." → print top-5 titles + cosine scores.
4. In NOTES.md, document one paraphrase query that works and one failure case (negation, OOD, identifier).

Include build_index.py and query.py (or unified CLI with subcommands).

Checklist

• Index build script + query script
• Vectors normalized if using cosine
• Failure case documented with explanation
• Dependencies pinned in project README