Core Concepts

Serving and streaming

From generate() to production APIs — TTFT, tokens/sec, SSE streaming, batching, and OpenAI-compatible façades.

45 min

Also in guided path: API + streaming

From notebook generate() to a product API

In a notebook, model.generate() blocks until the full string exists. Users experience something different: a blank spinner, then a burst of text — or, if you stream well, tokens appearing as they are produced.

Serving is the stack that turns model weights (or a vendor API) into a reliable, multi-tenant, metered HTTP API. Streaming is how you expose partial output so perceived latency feels acceptable.

Two clocks users feel

  1. TTFT — time to first token (snappiness)
  2. Tokens/sec — how fast the rest arrives

Budget and alert on both. A system with great tokens/sec but 8s TTFT feels broken for chat.

sequenceDiagram
  participant Client
  participant API
  participant Engine
  Client->>API: POST /chat/completions stream=true
  API->>Engine: schedule
  Engine-->>API: token
  API-->>Client: SSE data
  Engine-->>API: token
  API-->>Client: SSE data

Mental model: prefill vs decode

Phase What happens Dominates
Prefill Process the whole prompt; build KV cache TTFT for long prompts
Decode Emit tokens one-by-one (or speculative chunks) Tokens/sec, streaming UX

Inference track deepens this (KV-cache, continuous batching, vLLM). Core concept: chat UX is an inference scheduling problem, not just “call the model.”

flowchart LR
  Prompt[Prompt tokens] --> Prefill[Prefill]
  Prefill --> KV[(KV cache)]
  KV --> Decode[Decode loop]
  Decode --> Stream[SSE / WS chunks]
  Stream --> UI[User sees tokens]

Step-by-step: what a streaming chat request does

  1. Client POSTs messages with stream: true and Accept: text/event-stream (or vendor SDK).
  2. Gateway auth, tenancy, rate limits, idempotency key.
  3. Scheduler places the job (batch with others on GPU engines).
  4. Prefill completes → first token events leave the API.
  5. Decode continues until stop / max tokens / client abort.
  6. Usage counters finalize; traces record TTFT and tokens/sec.

OpenAI-compatible façades (why they matter)

A widely useful pattern: expose OpenAI-shaped /v1/chat/completions whether the backend is OpenAI, Anthropic (translated), vLLM, Ollama, or a gateway router.

Benefits:

  • Swap models without rewriting every client
  • Local → staging → vendor with one SDK
  • Easier eval harnesses and proxy middleware

Engineer defaults

  • Prefer streaming for interactive chat; non-stream for batch jobs
  • Log prompt vs completion tokens and TTFT per request
  • Separate interactive vs batch queues (see Networking)
  • Propagate cancellation when users hit stop — don’t burn decode
  • Disable proxy buffering on streaming paths

How to build a minimal streaming demo

  1. Call any model API with streaming enabled.
  2. Forward chunks over SSE from your backend to a browser.
  3. Measure TTFT (client timestamp: request start → first chunk).
  4. Compare short vs long system prompts — feel prefill.

Tools today (2025–2026)

Layer Examples
Hosted APIs OpenAI, Anthropic, Google, Azure OpenAI
Local / open Ollama, llama.cpp server, vLLM, SGLang, TensorRT-LLM
Gateways LiteLLM, custom BFF, cloud AI gateways
Load / batching vLLM continuous batching, Triton
Observability OTel metrics for TTFT, tokens/sec, queue time

Failure modes

Symptom Cause Direction
Tokens arrive in one dump Proxy buffering proxy_buffering off / correct SSE
Great bench, bad UX Measuring only throughput Track TTFT under load
Spiky latency No batching / bad scheduling Inference engine + queues
Stream hangs forever No idle timeout Layered timeouts
Cost surprise Retries without idempotency See networking article
Client can’t swap models Vendor-locked payloads OpenAI-compatible façade

Tradeoffs

  • Streaming — better UX; harder clients, retries, and partial failure handling.
  • Self-host — control and data plane; ops burden (Open-weight vs APIs).
  • Larger batches — higher throughput; can hurt TTFT if mis-tuned.

When this concept is enough vs go deeper

Enough for app engineers wiring chat UIs and gateways. Go to Inference track when you own GPUs, KV cache tuning, quantization, or speculative decoding.

Glossary

Term Meaning
TTFT Time to first token
SSE Server-Sent Events — common token stream transport
Prefill / decode Prompt processing vs generation
Continuous batching Scheduler interleaves many sequences for GPU efficiency

Micro-project

Stream tokens over SSE from a local or API model; measure TTFT for two prompt lengths; write the numbers in one table.

Guided API + streaming; Inference vLLM; Networking for AI apps; industry Netflix-style gateway examples.

Project checklist0/3 done