Marketplace ranking meets LLMs: Uber/Airbnb-style re-rank patterns

Two different jobs: candidates vs polish

Public search and marketplace talks (Uber, Airbnb, and peers) usually describe a multi-stage funnel: cheap retrieval → feature-rich learning-to-rank → optional neural re-rank → UI.

LLMs enter late. Replacing the whole funnel with “just embed and prompt” collapses under latency, cost, and inventory scale.

Pattern: keep classical stages; add LLM re-rank / reason / explain on a shortlist.

Funnel architecture

flowchart TD
  Q[Query + context] --> Ret[Retrieval / geo / inventory filters]
  Ret --> LTR[Learning-to-rank / GBDT]
  LTR --> Short[Top N 20-100]
  Short --> LLM[LLM re-rank or reason]
  LLM --> UI[Results + optional explanations]
  LTR --> UI

When to skip the LLM stage: high QPS, tight P99, low-stakes browse, or when LTR already saturates metrics.

Where LLMs help

1. Semantic re-rank of ambiguous natural-language queries (“quiet place near metro for remote work”)
2. Cross-encoding style judgment when features miss soft preferences
3. Explanations (“why this listing”) — careful with hallucination
4. Query understanding → structured filters (dates, party size, constraints)

sequenceDiagram
  participant U as User
  participant S as Search API
  participant R as Ranker
  participant L as LLM service
  U->>S: NL query
  S->>R: candidates
  R-->>S: top 50
  S->>L: re-rank brief + features
  L-->>S: ordered ids + scores
  S-->>U: page 1

Feature hygiene still matters

LLM re-rank prompts should include machine features, not only titles:

• Price, distance, rating, availability
• Business rules (compliance, fraud flags)
• Personalization signals allowed by policy

Otherwise the model “likes” photogenic copy and ignores hard constraints.

Latency & cost envelopes

Stage	Typical budget mindset
Retrieval	ms
LTR	low ms–tens of ms
LLM re-rank	tens–hundreds of ms; batch shortlist
Explanation	async or on expand

Techniques: distill to a small cross-encoder; cache re-ranks for identical (query, candidate set hash); run LLM only when query complexity score is high.

Failure modes

• Constraint violation: model promotes sold-out / geo-invalid items → always re-apply hard filters after LLM order
• Popularity bias: LLM echoes training priors; monitor fairness slices
• Prompt injection via listing text: treat inventory text as untrusted
• Metric theater: LLM improves “relevance” labels but tanks booking conversion

Eval design

Offline:

• Pairwise preference labels on shortlists
• Constraint violation rate must be ~0 after filters
• Latency histograms for the LLM stage alone

Online:

• Interleaving / switchback experiments
• Guardrail metrics: conversion, refunds, support tickets

What to ship

1. Feature-rich shortlist from existing ranker
2. LLM re-rank behind a flag + complexity gate
3. Post-LLM hard filter pass
4. Cached results for repeated queries
5. Eval set of ambiguous NL queries with expected constraints

Design takeaway

Marketplace AI is still mostly IR + ranking science. LLMs are a precision instrument on the last mile — powerful when bounded, expensive when naïve.