Netflix-style LLM gateway: batching, KV cache, and one API

The product problem

Product teams want one chat/completions contract. Platform teams need to swap GPUs, swap open-weight vs vendor APIs, A/B post-trained checkpoints, and enforce budgets — without forking every client.

Public engineering discussions from large streaming / consumer platforms (Netflix-class serving) converge on the same shape: an LLM gateway that looks like a stable HTTP API while inference engines underneath handle continuous batching and KV-cache reuse.

This article frames that pattern from public knowledge — not confidential internals.

Architecture at a glance

flowchart LR
  Clients[Clients / services] --> Gateway[LLM Gateway]
  Gateway --> Router[Router + policy]
  Router --> EngA[vLLM / TensorRT-LLM]
  Router --> EngB[Vendor API]
  Router --> EngC[Canary checkpoint]
  Gateway --> Obs[Traces + cost meters]
  EngA --> KV[KV cache / prefix store]

Gateway responsibilities (ship these first):

1. Authn/authz and per-tenant quotas
2. Model alias → concrete deployment mapping
3. Request validation (max tokens, tools schema, content filters)
4. Streaming SSE/WebSocket fan-out with backpressure
5. Idempotency keys for non-stream completions where retries matter
6. Unified tracing: request_id, model id, token counts, TTFT, TPOT

Inference engine responsibilities:

• Continuous batching of decode steps
• Paged / block KV cache
• Prefix caching for shared system prompts
• Speculative decoding (optional later)

Continuous batching (why throughput jumps)

Naive “one request = one GPU reservation until done” wastes compute during decode. Continuous batching interleaves tokens from many sequences on each step:

sequenceDiagram
  participant G as Gateway
  participant E as Engine
  participant GPU as GPU
  G->>E: req A (prefill)
  G->>E: req B (prefill)
  E->>GPU: batch prefill A+B
  loop Decode steps
    E->>GPU: step tokens for active seqs
    GPU-->>E: next tokens
    E-->>G: stream deltas
  end

Tradeoffs

Failure modes: head-of-line blocking when a few huge-context jobs dominate; OOMs when max concurrent sequences × max context exceeds KV budget.

What to ship: expose max_num_seqs, max_model_len, and a queue depth metric. Reject or shed load before the GPU OOMs.

KV cache and prefix reuse

Autoregressive decoding stores key/value tensors per layer per token. Memory is the scarce resource, not FLOPs alone.

Prefix caching (common in vLLM-class engines): if many requests share the same system prompt / RAG preamble, cache the KV for that prefix and skip recompute.

flowchart TD
  Prompt[System + tools schema] --> Hash[Prefix hash]
  Hash --> Hit{KV hit?}
  Hit -->|yes| Decode[Decode only]
  Hit -->|no| Prefill[Prefill + store blocks]
  Prefill --> Decode

Tradeoffs

• Aggressive prefix sharing → big TTFT wins for “same template, different user turn”
• Too-coarse hashing → cache pollution; too-fine → low hit rate
• Multi-tenant prefixes need tenant-scoped cache keys or you leak attention state across customers (isolation bug class)

What to ship: measure prefix hit rate; document which prompt segments are stable enough to pin.

Gateway routing patterns

alias: "assistant-default"
  → primary: llama-3.1-70b-awq @ cluster-a
  → fallback: vendor-gpt-class @ api
  → canary: 5% → internal-sft-v12

Routing should be data-driven, not hardcoded in clients:

• Feature flags / percentage canaries
• Cost ceilings (route cheap model unless complexity score high)
• Latency class (interactive vs batch)
• Capability tags (tools, vision, json_schema)

Observability that matches serving

Track at least:

• TTFT (time to first token) — user-perceived snappiness
• TPOT (time per output token) — stream smoothness
• Queue wait — gateway vs engine
• KV memory pressure — leading indicator of OOM
• $/1k tokens by alias and tenant

Without these, “the model feels slow” is unactionable.

Checklist: minimal gateway you can defend in review

• OpenAI-compatible /v1/chat/completions with streaming
• Model aliases + canary map in config (not code)
• Per-key RPM/TPM limits
• Structured logs with token + latency fields
• Load shed on queue depth / KV pressure
• One load test: shared system prompt → measure prefix-cache benefit

Interview / design cues

Be ready to explain why batching improves throughput but can hurt P99, how KV memory scales with concurrent context, and why a gateway exists even when you “only call one vendor API” today (policy, metering, swap-out).

Further reading in ShipAI

Curriculum modules on serving SLMs, quantization tradeoffs, and industry case labs rebuild a miniature gateway — use this article as the architecture brief before you code.