Cost and latency routing

One model is rarely optimal

Frontier models are expensive and slow for “classify this ticket.” Tiny/local models fail on hard reasoning. Routing picks a model (or cache hit) per request — the highest-ROI inference lever after “don’t call the model.”

flowchart TD
  Req[Request] --> Cache{Prompt / semantic cache?}
  Cache -->|hit| Done[Return cached]
  Cache -->|miss| Triage{Complexity / risk?}
  Triage -->|low| Cheap[SLM / cheap API]
  Triage -->|high| Strong[Frontier model]
  Triage -->|tool-heavy| Agent[Agent stack]
  Cheap --> Escalate{Low confidence?}
  Escalate -->|yes| Strong

Interview cue: Routing is a product policy with evals — not a single classifier accuracy number. Include cost, latency, and failure cost (wrong refund advice ≠ wrong emoji caption).

What “routing” owns vs what engines own

Confusing these layers produces dashboards that look green while the bill explodes.

Levers (in order of typical ROI)

1. Don’t call the model — exact cache, rules, retrieval-only answers
2. Shrink context — better RAG, less dump (chunking)
3. Smaller / cheaper model for easy intents
4. Self-host quantized + batched serving for steady QPS (vLLM, quantization)
5. Speculative decoding when acceptance is high (speculative decoding)
6. Frontier only on hard / high-risk turns

flowchart LR
  Free[Cache / rules] --> Small[SLM / cheap API]
  Small --> Mid[Mid-tier / self-host]
  Mid --> Fron[Frontier]
  Fron --> Human[Human review if needed]

Rough cost intuition (illustrative)

Numbers change weekly — log your $/successful task.

Routing signals

Start simple: keyword / classifier intent → model map, plus mandatory escalate list for high-risk intents. Fancy learned routers come after you log outcomes.

# Shape — boring router first
def route(req):
    if exact_cache_hit(req):
        return "cache"
    if req.intent in HIGH_RISK:
        return "frontier"
    if req.plan == "free" or req.intent in EASY:
        return "slm"
    if req.prompt_tokens > 12000:
        return "batch_or_frontier"
    return "mid"

Architecture

Keep routing in the gateway, not inside each engine:

flowchart TB
  Client --> GW[API gateway]
  GW --> Cache[Redis exact / semantic]
  Cache -->|miss| Router[Router policy]
  Router --> SLM[vLLM SLM pool]
  Router --> API[Frontier HTTP API]
  Router --> Batch[Batch queue]
  SLM --> Log[Telemetry: model, $, TTFT]
  API --> Log

Pair with Redis for AI caching and engine-level KV / batching so you do not confuse app cache hits with engine prefix cache hits.

Latency classes as first-class routes

SLOs must be class-specific. One global p95 lies.

Cost model you should actually log

Per request:

# Shape — attach to every completion span
span = {
    "model_id": chosen,
    "route_reason": "intent=faq;plan=free",
    "prompt_tokens": u.prompt,
    "completion_tokens": u.completion,
    "usd_estimate": price(chosen, u),
    "ttft_ms": ttft,
    "cache_hit": False,
    "escalated": False,
}

Without $/successful task, teams optimize vanity tokens/sec while margins die.

Unit economics sketch

[ \text{cost per successful task} \approx \frac{\sum ${\text{model}} + ${\text{infra}}}{#\text{ successful tasks}} ]

Track separately: cache hit rate, escalate rate, and retry rate — each moves the denominator and numerator differently.

Escalation patterns

1. Always escalate list (refunds, account deletion, medical)
2. Confidence escalate — judge / self-report / schema validation fail
3. User escalate — “answer didn’t help” → stronger model retry
4. Shadow — run cheap + frontier in parallel offline to label disagreement

flowchart TD
  Cheap[Cheap model answer] --> Val{Schema / judge OK?}
  Val -->|no| Fron[Frontier retry]
  Val -->|yes| Risk{High-risk intent?}
  Risk -->|yes| Fron
  Risk -->|no| Return[Return cheap]
  Fron --> Return2[Return strong]

Cap escalate rate — a broken router that escalates 90% deletes the savings.

Shadow evaluation loop

Budgets and soft fail

Ship rule: budget enforcement belongs in the gateway with audited overrides — not “hope FinOps notices next month.”

Anti-patterns

• One frontier model for all traffic “for simplicity”
• Routing only on cost, ignoring failure cost
• Semantic cache in front of personalized / entitlement-gated answers
• No model_id in logs → cannot do FinOps
• Changing routes without an eval harness on hard cases
• Optimizing engine tok/s while $/task rises from longer prompts
• Escalating everything that looks “uncertain” without a rate cap

Metrics to review weekly

Weekly review ritual (lightweight)

1. Top intents by $ spend
2. Escalate outliers
3. Cache miss clusters (candidates for rules / FAQ)
4. Quality regressions on cheap routes
5. Capacity: interactive vs batch pool saturation

Connecting back to engines

Routing decides which model and pool. Engines decide how efficiently that pool runs:

• Continuous batching → more QPS per GPU
• Quantization → more concurrency
• KV / prefix cache → better TTFT on shared prefixes
• Speculative decoding → faster decode when acceptance is high

Do not expect routing alone to fix a badly sized max-model-len or a single overloaded pool.

Micro-project

On 100 prompts: log per-request model, tokens, $ estimate, and TTFT. Add a cheap-vs-strong router (even rules-based). Report total $ and failure rate vs all-frontier baseline. Include at least 10 high-risk prompts that must always escalate.

Related

• Guided Cost accounting and Caching and latency
• Redis for AI caching
• vLLM · Quantization
• Example library: cost control for LLM apps (Real-world examples)