Hybrid search and rerankers

Why vectors alone miss

Semantic search fails on exact IDs, error codes, SKUs, rare proper nouns. Keyword / BM25 fails on paraphrase and cross-lingual near-matches. Hybrid retrieves with both; a reranker (usually a cross-encoder) rescoring the shortlist often wins precision for the LLM context window.

Chunking still matters more than fancy fusion — fix chunking and metadata first. Ground yourself in vector databases so ANN vs BM25 is not vendor fog.

flowchart LR
  Q[Query] --> V[Vector ANN]
  Q --> K[BM25 / keyword]
  V --> Merge[Merge / RRF / weighted]
  K --> Merge
  Merge --> RR[Reranker]
  RR --> LLM[LLM context]

One-sentence definition

Hybrid search fuses lexical and dense retrieval so exact tokens and paraphrase both contribute candidates; a reranker re-scores a shortlist with a stronger model for precision before packing the LLM prompt.

What this article is for

By the end you should be able to:

1. Explain when BM25 beats vectors and when vectors beat BM25 — with product examples
2. Implement RRF (or defend a weighted mix) without score-scale bugs
3. Place a cross-encoder correctly: retrieve wide, rerank narrow
4. Budget latency and dollars so hybrid+rerank does not blow TTFT
5. Eval with recall@k / nDCG ablations and ship degrade modes that stay ACL-safe

If you only remember one ship rule: retrieve wide, fuse ranks, rerank to a token budget, never skip filters on either leg.

The failure modes that force hybrid

Interview cue: hybrid is not “more AI.” It is covering two different error distributions so the shortlist fed to the LLM is less wrong.

Where hybrid sits in the RAG stack

flowchart TB
  Docs[Docs / tickets / policies] --> Chunk[Chunk + metadata]
  Chunk --> Emb[Embed → ANN]
  Chunk --> Lex[Index text → BM25]
  User[User query] --> QEmb[Embed query]
  User --> QTok[Analyze tokens]
  QEmb --> ANN[Vector top-N]
  QTok --> BM[BM25 top-N]
  ANN --> Fuse[RRF / fusion]
  BM --> Fuse
  Fuse --> CE[Optional rerank]
  CE --> Pack[Pack + cite]
  Pack --> LLM[Generate]

Upstream mistakes (bad chunks, missing ACL metadata, wrong embedding model) dominate fusion tweaks. Treat hybrid as a middle layer, not a substitute for chunking or a sane store choice.

BM25 / keyword path

Inverted index: term frequency, document length normalization, IDF. Strengths: exact tokens, debuggable matches, cheap. Weaknesses: vocabulary mismatch (“cancellation” vs “cancel my plan”).

Keep a searchable text field (or parallel search engine) even if your “vector DB” is specialized — many engines (Weaviate, Elastic, OpenSearch) do both. Postgres can use tsvector alongside pgvector.

Mental model (enough for product work)

BM25 scores a document higher when:

• Query terms appear often in the doc (TF, saturated — more hits help less after a point)
• Those terms are rare across the corpus (IDF)
• The doc is not absurdly long relative to average (length normalization)

You do not need to derive the formula in an interview. You need to know: token presence and rarity drive the score, so SKUs and error codes shine when the analyzer does not destroy them.

What BM25 is good at in AI products

Analyzer gotchas (the silent BM25 killer)

Ship rule: when you enable keyword search, freeze a 20-query ID slice (ERR-\\d+, SKU-…) and gate launch on that slice — not only on paraphrase FAQs.

BM25 field design

Do not dump ACL roles into free-text. Filters are a security boundary; boosts are a ranking nicety.

Dense / vector path (quick refresh)

Bi-encoder embedding models map query and doc into the same space. ANN returns approximate nearest neighbors. Strengths: paraphrase, soft topical match. Weaknesses: rare literals, embedding drift across model versions, filter-sparse tenancy.

Hybrid does not replace knowing HNSW/IVF and filtered search — see the flagship vector databases article. For this page, remember:

• Same embedding model for query and corpus
• Stable chunk ids shared with the BM25 index
• Filters applied on the vector leg too

Fusion methods

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranks in rank_lists:
        for r, doc_id in enumerate(ranks, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + r)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

flowchart TD
  Q[Query] --> VTop[Vector top 40]
  Q --> BTop[BM25 top 40]
  VTop --> RRF[RRF merge]
  BTop --> RRF
  RRF --> CE[Cross-encoder top 8]
  CE --> Pack[Pack for LLM]

Why RRF is a good default

Vector scores and BM25 scores live on different scales. RRF ignores raw scores and uses ranks, so you can fuse Pinecone hits with Elasticsearch hits without inventing fragile normalization. Tune k (often 60) lightly; spend more time on candidate depths (e.g. 40+40).

Properties you can defend in an interview:

1. Scale-free — no z-score or min-max required across systems
2. Diminishing returns — rank 1 beats rank 2 by more than rank 40 beats rank 41
3. Multi-list friendly — add a third list (e.g. sparse neural) with the same formula
4. Debuggable — print each leg’s ranks beside the fused score

Weighted mix pitfalls

If you mix raw scores:

• BM25 can dominate or vanish depending on analyzer length
• Cosine vs inner-product engines differ
• Always min-max or z-score on a validation set — never vibes
• Recompute normalization when the corpus or analyzer changes

def minmax(xs: list[float]) -> list[float]:
    lo, hi = min(xs), max(xs)
    if hi - lo < 1e-12:
        return [0.5 for _ in xs]
    return [(x - lo) / (hi - lo) for x in xs]

def weighted_fuse(vector_hits, bm25_hits, alpha: float = 0.5):
    # hits: list[(id, score)] — normalize within each list, then mix by id
    ...

Prefer RRF until you have labeled data proving weighted mix wins and an owner for recalibration.

Cascade patterns

Cascades are product decisions. Document them; do not bury them in glue code.

Union-then-rerank (the workhorse)

Most production RAG hybrids look like:

1. Vector top 40 + BM25 top 40
2. Dedupe by chunk id
3. Cross-encoder score all unique pairs (often 50–80)
4. Keep top 5–10 for the prompt

RRF before the cross-encoder is optional when you will rerank everything anyway — but RRF still helps if you truncate before rerank for cost.

Rerankers

Bi-encoder (embedding model): encode query and doc separately — fast retrieval.
Cross-encoder: jointly encode (query, doc) — slower, sharper relevance for a shortlist (e.g. 20–50 candidates → top 5–10).

Use async or small dedicated GPU/CPU pool for rerank if p95 TTFT is tight. Skip rerank for obvious FAQ exact matches (cache hit). See Redis for AI caching.

# Shape — batch cross-encoder scores
pairs = [(query, hit.text) for hit in candidates]
scores = rerank_model.predict(pairs)
ranked = [h for _, h in sorted(zip(scores, candidates), reverse=True)]

Why cross-encoders win precision

Bi-encoders compress each side independently — great for ANN, lossy for fine pairwise relevance. Cross-encoders see the interaction between query tokens and doc tokens (attention over the pair). That is why they are too slow for millions of docs but excellent for dozens.

Reranker selection notes

Ship rule: retrieve wide (20–50), rerank to narrow (5–10). Reranking only 5 candidates wastes the model.

Batching, truncation, and text length

Cross-encoders have max sequence lengths. Truncate docs deliberately:

• Prefer head + heading over random middle
• Keep query intact; truncate doc first
• Batch pairs (8–32) for GPU efficiency
• Cap candidates hard — 200 pairs at p95 will melt budgets

Domain mismatch

A web-trained cross-encoder on legalese or medical notes can invert good RRF order. Always compare:

Conditional rerank

flowchart LR
  Hits[Hybrid candidates] --> Agree{Top sets agree?}
  Agree -->|yes| Pack[Skip rerank / light pack]
  Agree -->|no| CE[Cross-encoder]
  CE --> Pack

Heuristics that work in practice:

• Jaccard of vector top-10 vs BM25 top-10 below threshold → rerank
• Exact-ID regex match already in BM25 top-3 → skip CE
• Cache hit on normalized query → skip retrieval entirely (Redis)

Query rewriting (optional sibling)

Multi-query / HyDE / rewrite-for-BM25 can lift recall before fusion. Treat as another lever with evals — each rewrite costs an LLM call. Guided Rerank / rewrite covers rewrite + rerank together.

flowchart TD
  U[User query] --> RW{Rewrite?}
  RW -->|multi-query| Qs[q1,q2,q3]
  RW -->|HyDE| Hyp[Hypothetical answer embed]
  RW -->|no| Q0[Original]
  Qs --> Retr[Retrieve each + fuse]
  Hyp --> Retr
  Q0 --> Retr
  Retr --> RR[Rerank]

Only keep rewrites that move recall@k on your eval set more than they burn latency budget.

Rewrite + hybrid interaction

Rewrites multiply retrieval cost. A sane pattern:

1. Rewrite once (or skip)
2. Run hybrid on the rewritten query (and optionally original)
3. Fuse all lists with RRF
4. Rerank once

Do not rerank three independent pipelines and then argue about which “won.”

Latency budget (example chat turn)

If rerank blows the budget, shrink candidate set or rerank only when vector/BM25 disagree (e.g. top ids Jaccard below threshold).

End-to-end budget example

Assume product SLO: p95 time-to-first-token ≤ 1.5 s. Rough allocation:

If retrieval alone is 400 ms, you will never “optimize the prompt” into happiness. Measure retrieval as its own span.

Cost budget (pairs × price)

Hosted rerank billed per query–doc pair:

• 40 candidates × $X / 1k pairs × QPS = monthly bill
• Multi-query ×3 without shrinking candidates triples spend

Ship rule: put a hard max_rerank_pairs in config. Version it. Alert when average pairs approach the cap.

Evals that matter

On a fixed query set with relevance labels:

• recall@k — did any gold chunk appear?
• nDCG@k — ranking quality
• Answer faithfulness — did the LLM use the right span?
• p95 added latency — rerank cost

Compare: vector-only vs hybrid vs hybrid+rerank. Expect hybrid to win on ID-heavy corpora; rerank to win when top-20 is noisy.

How to build the ablation table

1. Freeze 30–100 queries with graded relevance (0/1/2) or binary gold ids
2. Slice: paraphrase / exact-ID / long-tail / multilingual (as applicable)
3. Run three systems with identical filters and chunk corpus
4. Report recall@5, recall@10, nDCG@10, p50/p95 retrieval ms
5. Only then decide whether CE is worth the ms

system,slice,recall@5,ndcg@10,p95_ms
vector,paraphrase,0.82,0.70,45
hybrid,paraphrase,0.86,0.74,60
hybrid+ce,paraphrase,0.88,0.81,150
vector,exact_id,0.41,0.30,42
hybrid,exact_id,0.93,0.88,58
hybrid+ce,exact_id,0.94,0.90,145

If hybrid already nails exact-ID and CE only helps paraphrase by 1%, enable CE only on the disagree path.

Faithfulness vs retrieval

A perfect retriever can still yield bad answers if packing truncates the gold span or the LLM ignores citations. Keep retrieval metrics and generation metrics separate — see Core evals fundamentals mindset: gate launch on both.

Architecture sketch

flowchart TB
  Q[Query] --> Emb[Embed]
  Q --> Lex[Analyze tokens]
  Emb --> VDB[(Vector ANN)]
  Lex --> SE[(BM25 engine)]
  VDB --> Fuse[RRF]
  SE --> Fuse
  Fuse --> CE[Reranker]
  CE --> Pack[Pack + cite]
  Pack --> LLM[LLM]

Engines that do hybrid inside one system (Weaviate, Elastic, OpenSearch) reduce glue code — still verify filter + ACL behavior. App-side RRF is portable across store choices.

In-engine hybrid vs app-side fusion

Ship rule: pick the fusion location for ops reasons, then prove quality with the same ablation table either way.

Postgres hybrid sketch

With pgvector:

1. ts_rank / websearch_to_tsquery for BM25-ish ranking
2. Cosine / IP distance for vectors
3. RRF in SQL or application code on chunk_id
4. Optional CE in a sidecar

Useful when you already live in Postgres and want one backup story — not when you need massive ANN QPS.

Interface contract (staff bar)

Three measurable metrics

1. recall@5 / nDCG@10: vector vs hybrid vs hybrid+rerank
2. p95 latency contribution of rerank
3. Exact-ID query success rate (dedicated slice)

Two degrade modes

1. Reranker timeout → serve RRF order; log degrade
2. BM25 cluster down → vector-only with user-visible “exact match may be weaker”

Threat note

Fusing without re-applying ACL on one leg, or remapping ids incorrectly after rerank, can surface unauthorized chunks. Catch with filter-both-legs tests and citation id equality asserts.

Suggested response schema

POST /retrieve
{
  "query": "...",
  "tenant_id": "...",
  "roles": ["..."],
  "k_vector": 40,
  "k_bm25": 40,
  "rerank": true,
  "fusion": "rrf_k60_v1"
}
→
{
  "hits": [
    {
      "id": "chunk_…",
      "text": "…",
      "score": 0.91,
      "sources": ["vector", "bm25"],
      "cite": {"doc_id": "…", "span": "…"},
      "debug": {"vector_rank": 3, "bm25_rank": 12, "rrf": 0.03, "ce": 0.91}
    }
  ],
  "degraded": null,
  "fusion": "rrf_k60_v1"
}

Strip debug in production responses if needed; keep it in logs.

Failure modes

• Normalizing scores incorrectly → BM25 always wins
• Reranker trained on web docs, applied to legalese without check
• Retrieving k=5 then reranking 5 — too little headroom
• Ignoring citations after fusion (id mapping bugs)
• Adding HyDE + multi-query + rerank with no latency SLO
• Hybrid without analyzers for your language / tokenization
• Letting rerank see pre-filter leaks
• Dual indexes out of sync (vector upsert succeeded, BM25 failed)
• Stale BM25 after chunk edit while vectors refreshed (or reverse)
• Packing CE top-1 that is a near-duplicate of top-2 — wasted tokens

Index sync (underrated)

Hybrid assumes the same chunk id exists in both stores with the same text. Operationally:

Alert on count skew: |count_vector - count_bm25| per tenant.

Debugging playbook

1. Split metrics: vector-only vs BM25-only on the failing query
2. Print RRF top-20 ids — is the gold present before rerank?
3. If gold present pre-rerank but absent after → reranker domain mismatch
4. If gold absent entirely → chunking / embedding / filters, not fusion
5. Check analyzer: are SKUs being tokenized away?
6. Diff filters: did one leg drop tenant_id?
7. Confirm text equality: BM25 body == vector payload text for that id

Score debugging template

For a single failing query, log:

vector_top: [(id, score), ...]
bm25_top:   [(id, score), ...]
rrf_top:    [(id, rrf), ...]
rerank_top: [(id, ce_score), ...]
gold_ids:   [...]
filters:    {tenant, roles, doc_type}
analyzer:   {name, version}

If gold is in rrf_top but not rerank_top, the cross-encoder is the bug. If gold is nowhere, go upstream.

Production readiness checklist

• BM25 and vector paths share stable chunk ids
• Filters on both retrieval legs
• RRF (or documented fusion) with versioned config
• Rerank candidate depth ≥ 20 when enabled
• Ablation table in design doc
• Latency SLO includes rerank
• Degrade flags for rerank / BM25 outage
• Index count skew alerts
• Exact-ID eval slice gated
• Citation ids asserted equal through CE
• Analyzer tests for SKU / error-code tokens
• On-call runbook: “BM25 down” and “CE timeout”

Interview prompts

1. Why is RRF often better than naive score mixing?
2. Bi-encoder vs cross-encoder roles in a RAG stack?
3. When would you skip hybrid entirely?
4. How do you keep citations correct through fusion + rerank?
5. How do you prevent ACL leaks when fusing two engines?
6. Design a conditional rerank policy under a 200 ms retrieval budget.
7. Walk through debugging a query where hybrid helps offline but hurts online p95.

Strong whiteboard answer (sketch)

Prompt: “Add hybrid + rerank to our support RAG.”

1. Shared chunk ids; SoT text in Postgres; derived ANN + OpenSearch/Elastic or tsvector
2. Filters mandatory both legs; integration test for cross-tenant
3. RRF k=60; depths 40/40; CE to top 8
4. Ablation on 50 labeled tickets including ERR-/SKU- slice
5. Degrade: CE timeout → RRF; BM25 down → vector + banner
6. Metrics: recall@5, nDCG@10, p95 retrieval, exact-ID success
7. Config version in logs; feature flag for CE

Weak answers only name Cohere Rerank or “Elastic hybrid search” without filters or evals.

Candidate depth playbook

If gold rarely appears in top-50, fix chunking/embeddings before buying a fancier reranker.

Depth vs token budget

Reranking to 10 then packing 8 is fine. Reranking to 10 then packing 20 (because “more context”) reintroduces noise the CE just removed. Align CE cutoff with pack budget.

Sparse + dense in one engine

Some systems support sparse neural vectors (e.g. SPLADE-like) alongside dense. Treat sparse neural as “learned BM25.” Still evaluate against classic BM25 — classic often wins on SKUs with less ops pain.

Three-way RRF (dense + BM25 + sparse) is possible — measure whether the third list earns its keep.

Online quality signals

• Thumbs-down with retrieved ids attached
• “Citation clicked” rate
• Exact-ID query success (regex slice of queries matching ERR-\\d+ / SKU patterns)
• Empty retrieval rate after filters
• Degrade-mode rate (CE timeout / BM25 down)
• Duplicate-citation rate (packing bugs)

Wire these to weekly retrieval review, not only generation eval dashboards.

Feedback → labels loop

Online thumbs are noisy. Promote a sample to the offline gold set monthly:

1. Cluster thumbs-down by missing gold pattern (ID miss vs paraphrase miss)
2. Add 5–10 queries per cluster with human labels
3. Re-run ablation; adjust α / depths / CE on/off
4. Do not hot-patch fusion weights from a single angry ticket

When hybrid is enough (skip rerank)

• Latency SLO < ~100 ms retrieval total
• Corpus small and clean
• Exact cache already handles hot FAQs
• Cross-encoder shows <2% nDCG lift on evals
• Mostly exact-ID traffic already served by BM25-heavy hybrid

When to skip hybrid (vector-only)

• Pure conversational memory / soft topical browse
• No ID-like queries in production traffic analysis
• You lack a maintainable text index and will not staff one yet (temporary — document the debt)

When to skip vectors (BM25-only)

Rare for LLM products, but valid for pure catalog SKU lookup with template answers and no paraphrase need. Most “AI search” still wants dense for natural language.

RRF with filters (correctness)

Pseudo-flow:

1. Bind tenant_id / ACL on server
2. BM25 search with filters
3. Vector search with filters
4. RRF on id lists
5. Rerank texts for surviving ids
6. Assert every id still matches tenant before pack

Skipping filters on either leg is a security bug, not a quality footgun.

sequenceDiagram
  participant App
  participant V as Vector ANN
  participant B as BM25
  participant CE as Reranker
  App->>App: bind tenant + roles
  App->>V: top40 + filters
  App->>B: top40 + filters
  V-->>App: ids_v
  B-->>App: ids_b
  App->>App: RRF merge
  App->>CE: pairs for merged ids
  CE-->>App: scores
  App->>App: assert ACL on ids
  App->>App: pack + cite

Test cases you should automate

A/B in production (careful)

If you A/B hybrid vs vector-only:

• Use retrieval metrics + answer faithfulness, not clickbait engagement alone
• Keep citation UI identical
• Cap traffic until eval parity proven offline
• Stratify by query slice (exact-ID vs paraphrase)

Online-only optimization often rewards fluent wrong answers.

Feature flags

Shadow mode: run hybrid in parallel, log Jaccard vs vector-only, do not change user-visible hits until confidence is high.

Worked comparison table (example)

Ship hybrid; enable rerank if the +ms fits SLO and nDCG lift is real on your labels.

Reading the table like a staff engineer

• Hybrid recovered exact-token failures — big recall jump, modest ms
• CE bought ranking quality (nDCG) more than raw recall — typical
• If your SLO forbids +100 ms, try conditional CE or smaller candidate sets before abandoning CE forever

Minimal hybrid service API

POST /retrieve
{ query, tenant_id, roles, k_vector, k_bm25, rerank: bool }
→ { hits: [{id, text, score, sources: [vector|bm25], cite}] }

Version fusion config in the response for debugging (fusion: rrf_k60_v1).

Idempotency and caching

• Cache key: hash(normalized_query, tenant, roles, fusion_version, corpus_version)
• Exact FAQ hits short-circuit before hybrid (Redis)
• Do not cache across tenants
• Invalidate on corpus_version bump (re-index)

Observability for hybrid retrieval

Log every request:

query_id, tenant_id, fusion, k_v, k_b, rerank,
vector_ms, bm25_ms, rrf_ms, ce_ms, total_ms,
hit_ids[], sources_per_hit[], degraded, empty

Dashboards:

• p95 ce_ms and timeout rate
• Exact-ID success (regex cohort)
• Empty rate by tenant
• Fraction of hits that came from BM25-only / vector-only / both

Tracing: one parent span retrieve, children embed, ann, bm25, rrf, rerank, pack.

Anti-patterns

Worked walkthrough: support desk RAG

1. Corpus: 80k ticket resolutions + 2k policy chunks; shared chunk_id; metadata tenant_id, product, acl_roles.
2. Indexes: OpenSearch BM25 on body + id_tokens; Qdrant/pgvector dense ANN.
3. Query: “payment failed with ERR-1842 after retry” → BM25 nails ERR chunk; vector brings related retry-policy paraphrase.
4. RRF: gold ERR chunk rank 1 BM25 / rank 15 vector → strong fused rank.
5. CE: promotes the specific ERR runbook over generic payment FAQ.
6. Eval: 40 queries; hybrid lifts exact-ID slice from 0.45 → 0.92 recall@5; CE lifts nDCG@10 +0.08 on noisy policy asks.
7. Prod: CE on when Jaccard(vector,bm25)<0.4; Redis exact cache for top FAQs; degrade banner if OpenSearch unhealthy.

Design doc: what “good” looks like

A staff-level hybrid/rerank design usually includes:

• Traffic slices (exact-ID %, paraphrase %, multilingual %)
• Engine choices + why (in-engine vs app-side)
• Analyzer plan for IDs
• Fusion method + version + depths
• Rerank model, hardware, max pairs, conditional policy
• Ablation results on a frozen set
• Latency / cost model at target QPS
• ACL test plan
• Degrade modes + user-visible copy
• Rollout flags and shadow plan
• Owner for analyzer / fusion config changes

Missing any of the security or eval bullets → not ready.

Putting what / why / how together

FAQ (hybrid + rerank)

Is RRF outdated?
No — it remains a strong, simple default. Learned fusion comes after you have labels and ops capacity.

Must rerankers be cross-encoders?
Most common yes for shortlists. LLM-as-judge rerank is possible but usually slower/costlier.

Can hybrid run entirely in the vector DB?
Sometimes (Weaviate/Elastic). App-side RRF is more portable across stores.

Does hybrid replace good chunking?
Never. Bad chunks poison both legs.

Should every query rerank?
No. Use caches, agreement heuristics, and eval-driven flags.

How big should k be before RRF?
Start 30–50 per leg. If gold is missing pre-fusion, go upstream — not to k=200 forever.

What if BM25 and vector disagree totally?
That is often when CE helps most — or when filters/analyzers are broken. Debug before celebrating disagreement.

Can I fuse more than two lists?
Yes (multi-query, HyDE, sparse neural). Each list must earn its recall lift vs latency.

Glossary

Micro-project

On 30 queries: vector-only vs hybrid (RRF) vs hybrid+rerank. Report recall@5 and nDCG@10; note p95 added latency for rerank.

Stretch goals:

1. Add an exact-ID slice of 10 queries; show hybrid’s lift
2. Implement conditional rerank; compare p95 vs always-on CE
3. Break the analyzer on purpose (stem SKUs); watch the ID slice collapse
4. Write the degrade path: kill CE mid-test; confirm RRF still returns

Hands-on next steps

1. Guided Hybrid search — implement RRF in the lab
2. Guided Rerank / rewrite — CE + rewrite levers
3. Chunking and metadata — fix upstream first
4. Postgres + pgvector — tsvector + vectors in one DB
5. Choosing vector stores — in-engine hybrid options
6. Core RAG building blocks — end-to-end mental model

Related

Guided Hybrid search and Rerank / rewrite; vector databases; example Marketplace ranking meets LLMs; Core RAG building blocks.