Netflix — in-house LLM serving

Why Netflix cares about in-house serving

Public engineering posts from Netflix describe owning the inference path for generative models used in product features — not because they reject cloud APIs entirely, but because latency, cost at scale, customization, and operational control matter once LLM traffic becomes always-on product surface area. In-house serving stacks unify routing, observability, safety filters, and model lifecycle (post-training artifacts → production endpoints).

You will not rebuild Netflix's platform. You extract patterns and build a toy OpenAI-compatible gateway that routes to local and remote models — the same abstraction shape many serving platforms expose.

Architecture notes to extract from reading

While reading Netflix (or comparable streaming-platform) public posts on LLM serving and post-training, capture:

1. Separation of training vs serving — different SLOs, different teams, artifact promotion pipeline.
2. Model gateway — single client-facing API; backends swap without app rewrites.
3. Observability — request tracing, GPU utilization, error taxonomy — parallels your production module spans.
4. Post-training integration — fine-tuned adapters or domain models registered as named endpoints.
5. Safety and policy — filters at gateway, not scattered in each app.

Write industry/netflix/notes.md — bullet architecture facts with links, not prose summary of entire blog.

Callout — Gateway pattern is the lesson: Apps speak OpenAI-style /v1/chat/completions; gateway picks local-llama, gpt-4o, or domain-refund-slm by route rules.

Mini model gateway design

Components:

Client → Gateway (OpenAI-compatible) → { Backend A: local vLLM/Ollama
                                      → Backend B: OpenAI API
                                      → Backend C: mock for tests }

Features for portfolio minimum:

• POST /v1/chat/completions JSON + optional streaming
• Route header or model name prefix selects backend (gpt-4o → remote, local/* → Ollama)
• Log backend, latency_ms, tokens per request
• Health per backend

Use FastAPI or lightweight proxy — reuse production module patterns.

Routing rules as product lever

Example policy table:

Gateway encodes cost/latency/quality tradeoffs PMs can reason about without reading Python.

Failure handling

Backend down → structured 503 with retry_after; optional fallback model (document quality risk). Circuit breaker per backend after N failures.

Comparison to calling APIs directly

Your agent from prior modules should call gateway URL, not provider URL — one config change for environments.

Model artifact promotion pipeline

Even at toy scale, simulate promotion: dev model alias → eval gate → staging alias → canary → prod alias. Gateway resolves alias to backend URL — rollback is pointer revert, not redeploy every consumer app.

Observability at the gateway

Gateway is ideal choke point for unified logs: every model call logged once with routed backend, latency, tokens — apps stop duplicating instrumentation. Your mini gateway should log even if downstream apps also log — proves centralization value in notes.

Multi-tenant routing (conceptual)

Production gateways route by tenant tier: enterprise gets dedicated GPU pool; free tier shares queue. Stub header X-Tenant-Tier routing to mock slow vs fast backend demonstrates pattern in portfolio demo.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Study public engineering patterns around Netflix: LLM serving at scale. Rebuild the idea, not scraped proprietary text — focus on transferable architecture.

Diagram — Netflix pattern (conceptual)

flowchart LR
  Client-->Gateway-->Batch[Batching/KV cache]-->GPU

Precise definitions & mental model

Extract 3 transferable patterns from Netflix's public tech narrative on LLM serving at scale; map each to a ShipAI module artifact.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Cargo-culting brand names without metrics; inventing fake citations; cloning UI not architecture.

Production & OSS practices

Write a lab README: hypothesis, architecture diagram, eval, cost model, what you'd ask their eng in an interview.

Micro-project: Mini model gateway

Ship in industry/netflix/ or portfolio gateway folder:

1. notes.md from public reading (≥8 architecture bullets with citations).
2. OpenAI-compatible gateway with ≥2 backends (one local or mock, one remote or mock).
3. Route table documented; demo curl for each backend.
4. Request log with backend and latency.
5. One paragraph: what Netflix-scale adds that your toy omits (GPU fleet, multi-region, etc.).

Acceptance: agent or curl switches models by changing model field only.

Checklist

• Architecture notes with source links
• Gateway implements /v1/chat/completions
• ≥2 backends routed by model name
• Per-request backend logging
• Honest scale gap paragraph in README