Hybrid search

Pure vector search fails on identifiers

Semantic retrieval excels at paraphrase — until a user searches ERR_CONNECTION_RESET, order id ORD-8842, or function name parseReceiptJSON. Embedding models smooth language; they do not reliably treat exact tokens as first-class citizens. Hybrid search combines lexical ranking (BM25, Elasticsearch term statistics) with vector similarity, then fuses scores into one ranked list.

Hybrid is the default architecture for production RAG over mixed corpora — not an optional optimization for large scale only.

Callout — measure lift on YOUR queries: Hybrid adds complexity. Keep it only if your eval set shows recall gains on identifier-heavy or keyword-heavy questions.

BM25 in one paragraph

BM25 scores documents by term frequency and inverse document frequency with length normalization. If a rare token appears in few docs, matches boost sharply. That is why SKU and error-code queries love lexical search.

Libraries: rank_bm25 in Python for teaching; OpenSearch/Elasticsearch/Typesense in production. Your micro-project can index the same chunk texts in memory for BM25 alongside your vector DB.

Dual retrieval pipeline

query ──┬──▶ embed ──▶ vector top-N
        │
        └──▶ tokenize ──▶ BM25 top-N
                    │
                    ▼
              fusion → top-k for LLM

Retrieve more than k from each leg (e.g. N=20, fuse to k=5) — reciprocal rank fusion (RRF) works on ranks, not raw scores, avoiding scale mismatch between cosine and BM25.

RRF formula for document d:

[ \text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)} ]

Common constant k=60. Implement in ~10 lines; no score normalization required.

Alternative weighted linear fusion after min-max normalizing each score — tune weights on dev queries.

When lexical wins

Build a query taxonomy in eval notes:

Tag eval queries with class; report recall@k per class before and after hybrid.

Building BM25 index in Python

For course-scale corpora, rank_bm25 is sufficient:

from rank_bm25 import BM25Okapi

tokenized = [doc.lower().split() for doc in corpus_texts]
bm25 = BM25Okapi(tokenized)

def bm25_search(query: str, k: int = 20) -> list[tuple[int, float]]:
    tokens = query.lower().split()
    scores = bm25.get_scores(tokens)
    top = sorted(enumerate(scores), key=lambda x: -x[1])[:k]
    return top

Use the same tokenizer policy for index and query. For code identifiers with underscores, consider preserving _ splits or using character n-grams for error codes — whitespace tokenization breaks ERR_CONNECTION_RESET into useless tokens unless you also index the full string as one token via metadata field.

Store parallel arrays: chunk_ids[i] aligns with corpus_texts[i] and BM25 row i.

Production hybrid stacks

At scale, teams often use OpenSearch/Elasticsearch with dense_vector + BM25 in one cluster rather than fusing two separate services. Concepts remain: two scoring signals, fusion layer, unified chunk id. Your micro-project teaches fusion logic that transfers directly when you migrate off in-memory BM25.

Score normalization pitfalls

Weighted linear fusion requires normalizing BM25 and cosine scores to comparable ranges — miscalibrated weights favor one leg always. RRF avoids normalization but hides magnitude signal. If identifier queries still miss after RRF, boost BM25 weight manually on queries matching regex [A-Z]{2,}-[0-9]+ — query-class routing is production-common.

Log which leg contributed the winning chunk id for each query in eval — builds intuition for tuning.

Eval queries for hybrid

Include deliberate lexical-heavy cases in eval set:

• Error codes and UUID fragments
• Product SKUs and API method names
• Mixed alphanumeric identifiers (ORD-8842-A)

Without these, hybrid looks equal to vector-only on average while failing the queries production users actually type.

Makefile targets

Add make hybrid-eval running baseline vs hybrid on eval/queries.jsonl and printing delta table — one command for milestone reviewers reproduces your LIFT.md numbers.

Tuning fusion

Ablation checklist:

1. Vector-only baseline
2. BM25-only baseline
3. RRF hybrid
4. Optional weighted fusion with grid search on dev set

Watch latency: two retrievals plus fusion still cheaper than one huge cross-encoder rerank (next lesson) — but profile anyway.

If hybrid barely beats vector-only on your set, ship simpler stack until corpus diversifies.

Implementation sketch

def rrf_fuse(rankings: list[list[str]], k_rrf: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k_rrf + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

Use stable chunk ids across both indexes — duplicate text with different ids breaks fusion.

Callout — same chunk granularity: BM25 and vector indexes must chunk identically; otherwise fusion compares different text units.

Engineering problem (staff framing)

Dense misses rare tokens; lexical misses paraphrase. Hybrid is default serious search.

Diagram — Hybrid fusion

flowchart TD
  Q --> D[Dense ANN]
  Q --> L[Lexical BM25]
  D --> F[Fusion / RRF]
  L --> F --> R[Rerank optional]

Precise definitions & mental model

BM25, RRF/linear fusion, sparse+dense.

Tradeoffs — when to use what

Complexity ↑; recall ↑ on mixed query types.

Failure modes (interview + on-call)

Unweighted fusion; not normalizing scores; ignoring filters.

Production & OSS practices

A/B fusion weights; log which channel won.

Micro-project: BM25 + vector

In m5/hybrid_search/:

1. Index corpus with BM25 (in-memory OK) and your vector store from lesson 5.3.
2. Implement RRF (or weighted) fusion query path.
3. On ≥15 eval queries (include identifier-style), report recall@5 vector-only vs hybrid.
4. Document query classes where lexical leg carried the win in LIFT.md.

Checklist

• Same chunk ids in both indexes
• Baseline vector-only scores recorded
• Measurable lift on at least one query class
• Fusion approach documented in README