ShipAI

Real-world examples

Cost control for LLM apps: cache, shrink, and route

Unit economics break demos. Practical levers — semantic cache, smaller models, cascades, and prompt budgets — with the failure modes of each.

12 minPattern inspired by high-scale LLM product teams
  • cost
  • caching
  • routing
  • latency

Framed from public engineering talks, blogs, and OSS patterns. Not confidential internals or invented quotes.

The bill arrives after the launch thread

Teams celebrate a working agent; finance sees token spend linear in users × turns × tools. Public guidance from high-scale products converges on a simple doctrine: measure unit cost early, then apply a stack of levers — cache, shrink, route — before asking for a bigger model budget.

Cost equation (make it explicit)

cost ≈ Σ turns × (prompt_tokens × p$ + completion_tokens × c$ + tool_overhead)

Instrument cost per successful task, not only tokens/day. A “cheap” model that needs five retries can lose.

Lever 1 — Caching

flowchart TD
  Req[Request] --> Exact{Exact cache hit?}
  Exact -->|yes| Ret[Return cached]
  Exact -->|no| Sem{Semantic cache?}
  Sem -->|hit + safe| Ret
  Sem -->|miss| Model[Call model]
  Model --> Store[Store under policy]
Cache type Good for Danger
Exact (hash of messages) Idempotent APIs, FAQ Privacy if shared cross-tenant
Semantic (embed ≈) Paraphrased FAQ Wrong answer on near-miss
Prompt prefix / KV Shared system prompts Isolation bugs
Tool result Expensive APIs Staleness

Ship rule: tenant-scoped keys; TTLs; never cache personalized regulated answers without review.

Lever 2 — Smaller models

Distill, quantize, or fine-tune a small model for narrow tasks (classification, extraction, routing).

flowchart LR
  Task[Task] --> Gate{Hard?}
  Gate -->|no| SLM[Small / cheap model]
  Gate -->|yes| Big[Frontier model]
  SLM -->|low conf| Big

Confidence gating beats “always frontier.” Calibrate on a labeled set; don’t trust raw logprobs alone for open generation.

Lever 3 — Routing and cascades

  • Intent router (SLM or embeddings) → specialist prompts / models
  • Speculative cascades: draft with small, verify with large only on disagreement
  • Batch offline workloads; reserve interactive SKUs for UX paths

Lever 4 — Prompt and context diets

  • Drop dead few-shots; version prompts
  • Summarize history; retrieve only what evals need
  • Cap tool JSON verbosity
  • Structured outputs reduce “chatty” completions

Latency is a cost cousin

Retries, long tools, and giant contexts burn both money and user patience. Track cost and TTFT on the same dashboard.

Failure modes

  • Semantic cache serves outdated policy text
  • Router silently sends hard tasks to weak models → silent quality cliff
  • Over-aggressive max_tokens truncation mid-JSON
  • “Savings” that push work into human support (false economy)

What to ship

  1. Per-tenant and per-feature cost metrics
  2. Exact cache for deterministic endpoints
  3. Complexity router with an override to big model
  4. Monthly prompt budget review in eng ritual
  5. Kill switch / spend cap per API key

Checklist for a design review

  • What is cost per successful task today?
  • Which 20% of traffic drives 80% of spend?
  • Which caches are tenant-safe?
  • What quality gate blocks a cheaper default?

Cost control is product engineering — not a finance spreadsheet afterthought.