Rerank / rewrite

Second-stage ranking buys quality cheaply

First-stage retrieval (vector + BM25 hybrid) optimizes for recall under latency budgets — return 20–50 candidates fast. Many are loosely related. A reranker cross-encoder scores each (query, chunk) pair jointly and reorders the shortlist. Often this lifts answer quality more than swapping to a bigger embedding model, because cross-encoders see full interaction between query and passage.

Alternative lever: query rewriting — an LLM expands acronyms, disambiguates intent, or generates hypothetical answer text (HyDE) before retrieval. Rewrite and rerank can stack; measure each separately.

Callout — profile the P95: Reranking 50 pairs adds 100–400 ms depending on model. Ship only if offline eval gain justifies online latency.

Reranker architecture

query → first-stage top-20 chunks
              ↓
     cross-encoder scores each pair
              ↓
         top-5 → LLM context

Models:

• API: Cohere Rerank, Voyage rerank, Jina rerank endpoints.
• Local: cross-encoder/ms-marco-MiniLM-L-6-v2 via sentence-transformers — fine for course projects.

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c.text) for c in candidates]
scores = model.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])

Rerank after hybrid fusion so you pay cross-encoder cost on a bounded set.

Query rewriting patterns

Example rewrite prompt:

Given a user question, produce two search queries that would retrieve
relevant documentation. Output JSON: {"queries": ["...", "..."]}

Retrieve for each query; union + dedupe candidates before rerank.

Always log original and rewritten queries for debugging.

Measuring lift

On frozen eval set from prior lessons, report:

• Recall@k after first stage
• Recall@k after rerank (gold chunk in top-5 post-rerank)
• MRR (mean reciprocal rank) if single gold chunk
• p50/p95 latency end-to-end
• Cost per query (embed + rerank API + optional rewrite LLM)

Table template:

If +Rewrite barely moves recall, drop it for production simplicity.

Callout — same eval, same chunks: Reranker experiments must use identical first-stage candidate pools logged to disk — otherwise you confound retrieval randomness.

Failure modes

• Overfitting reranker to dev set — keep held-out queries.
• Near-duplicate chunks — reranker tops five copies of same paragraph; dedupe by source_id first.
• Language mismatch — English reranker on multilingual corpus without language filter.

Cost-aware pipeline design

Stack components multiply cost:

rewrite (LLM) → embed × N → hybrid retrieve → rerank 50 pairs → generate (LLM)

Profile each stage on 100 queries. Often rerank on 20 candidates captures 90% of lift vs reranking 100 — diminishing returns matter at scale.

Cache embeddings of frequent queries (head terms in support portals). Cache rerank results keyed by (query_hash, candidate_ids_hash) with short TTL when corpus is mostly static.

For rewrite, use a small model or distilled prompt — HyDE with frontier model on every query rarely pays on identifier-heavy workloads.

Debugging rerank regressions

When rerank hurts recall:

• Candidates never contained gold chunk — rerank cannot help; fix first stage.
• Gold chunk ranked low by cross-encoder — training domain mismatch; try multilingual reranker or fine-tune reranker on domain pairs (advanced).
• Duplicate chunks crowd out diversity — dedupe before rerank.

Log pre- and post-rerank rankings side by side in eval output for manual inspection on miss cases.

Query rewrite eval isolation

When testing HyDE or multi-query rewrite, ablate:

1. Baseline hybrid only
2. Hybrid + rewrite (no rerank)
3. Hybrid + rerank (no rewrite)
4. Full stack

Otherwise you cannot attribute lift. Rewrite hurts identifier queries if hypothetical doc drifts off-domain — disable rewrite when query matches identifier regex.

Latency budget worksheet

If product SLA is 800 ms to first token, full stack above fails — drop rewrite or rerank fewer candidates.

When to skip rerank

• Tiny corpus (<200 chunks) where top-5 is exhaustive scan anyway.
• Strict sub-100ms SLA with acceptable recall already.
• Edge deployment without GPU and tight CPU budget.

Document skip decision with numbers, not preference.

Engineering problem (staff framing)

First-stage recall + second-stage precision. Query rewrite expands recall carefully.

Diagram — Rewrite → retrieve → rerank

flowchart LR
  Q --> RW[Rewrite/expand] --> Ret[Retrieve] --> RR[Cross-encoder rerank] --> Ctx

Precise definitions & mental model

Cross-encoders, HyDE/multi-query, risk of drift in rewrite.

Tradeoffs — when to use what

Latency/cost of rerank vs quality; rewrite helps or hallucinates queries.

Failure modes (interview + on-call)

Rewrite changes intent; rerank model domain mismatch.

Production & OSS practices

Cap candidates; timeout budgets; eval rewrite separately.

Micro-project: Measure lift

In m5/rerank/:

1. Log first-stage candidates for eval queries to candidates/.
2. Add reranker step; optionally one rewrite variant.
3. Produce lift_report.json with recall and latency vs baseline.
4. README recommendation: ship rerank yes/no with evidence.

Checklist

• Baseline and rerank pipelines on identical candidates
• Latency measured (not estimated)
• Clear ship/no-ship recommendation
• Rewrite logged if used