vLLM

What vLLM solves

Naive model.generate() in a Python process works for demos. Production chat needs many concurrent requests, KV-cache memory discipline, and fair scheduling. vLLM is an open-source serving engine built for that: PagedAttention + continuous batching + OpenAI-compatible APIs.

Sibling articles go deeper on the primitives: KV-cache / prefill / decode, continuous batching, quantization, speculative decoding, cost & latency routing.

flowchart TB
  Clients[Clients / gateway] --> API[OpenAI-compatible API]
  API --> Sched[Scheduler]
  Sched --> Batch[Continuous batch]
  Batch --> Eng[GPU worker]
  Eng --> KV[Paged KV cache]
  Eng --> Out[Token streams]
  Out --> Clients

Interview cue: vLLM is not “a faster Hugging Face.” It is a scheduler + memory manager for autoregressive decode under concurrency. Quoting tokens/sec without concurrency and max context is incomplete.

Mental model: the GPU is a scarce scheduler

Each request needs:

1. Weights (shared across requests)
2. Activations (transient per forward)
3. KV cache (grows with prompt + generated tokens)

Throughput collapses when KV fragments memory or when the engine waits for static batches. vLLM’s bet: treat KV like a paged virtual memory and schedule at iteration granularity.

What “serving” actually means

Ship rule: keep product policy in the gateway. Keep the engine boring and measurable.

PagedAttention (intuition → ops)

Autoregressive decoding stores key/value tensors per token (the KV cache). Contiguous allocation wastes GPU memory when sequences finish at different lengths — classic malloc fragmentation.

PagedAttention borrows OS paging:

• Store KV in fixed-size blocks
• Map logical token positions → physical blocks
• Free blocks when a sequence finishes
• Optionally share blocks for identical prefixes (system prompts)

Benefits:

• Higher batch size for the same VRAM
• Less waste when requests complete early
• Prefix caching wins for agent fleets with shared instructions

You do not reimplement this — but you must size max model length, GPU memory utilization, and max num seqs knowing KV dominates interactive memory.

flowchart LR
  subgraph Logical
    S1[Seq A tokens]
    S2[Seq B tokens]
  end
  subgraph Physical blocks
    B1[Block]
    B2[Block]
    B3[Block]
    B4[Free]
  end
  S1 --> B1
  S1 --> B2
  S2 --> B3

Rough KV sizing (order of magnitude)

[ \text{KV bytes} \approx 2 \times L \times H_{kv} \times d \times T \times b ]

where (L) = layers, (H_{kv}) = KV heads (GQA/MQA matters), (d) = head dim, (T) = tokens, (b) = bytes per element (2 for FP16, 1 for FP8). Multiply by concurrent sequences.

This is why max-model-len × concurrency is the real capacity plan — not “does the 8B fit?”

Continuous batching (engine view)

Static batching waits for a full batch of prompts — terrible TTFT under load. Continuous (iteration-level) batching:

1. Adds new requests as GPU slots / KV blocks free
2. Runs decode steps for mixed-length sequences together
3. Evicts finished sequences immediately

Result: much higher aggregate tokens/sec and better utilization. Details: Continuous batching.

gantt
  title Static vs continuous intuition
  dateFormat X
  axisFormat %s
  section Static
  Wait for batch :a0, 0, 3
  Run all together :a1, 3, 8
  section Continuous
  ReqA :b1, 0, 6
  ReqB joins :b2, 2, 7
  ReqC joins :b3, 4, 9

Prefill vs decode (why knobs fight each other)

Long prompts steal compute from everyone’s decode. Isolate batch/offline jobs from interactive traffic when you can. See KV-cache, prefill, and decode.

Ship rule: one pool for “chat with 2k context” and “summarize 100k PDF overnight” will make p95 TTFT look random. Split queues or priority classes.

Chunked prefill (why it exists)

Engines often chunk huge prefills so interactive decode steps still get GPU time. Without chunking, one agent tool dump can freeze TTFT for every other tenant. Treat “max batched tokens per step” as a fairness knob, not a mystery flag.

When to use vLLM

Operational knobs that matter

# Shape only — pin versions in your portfolio README
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 64

# Client against OpenAI-compatible server
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in 3 bullets."}],
    stream=True,
)
for event in stream:
    print(event.choices[0].delta.content or "", end="")

Tensor parallel vs “buy a bigger GPU”

TP reduces per-GPU weight footprint but adds cross-GPU communication. Measure your latency under concurrency — TP is not free.

Capacity planning sketch

1. Pick p95 prompt tokens and p95 output tokens from production logs (or a guess with a safety factor).
2. Estimate KV per sequence at that length (include GQA head count).
3. max_num_seqs ≈ (usable_vram − weights − fragmentation) / kv_per_seq.
4. Load-test at that concurrency; watch preemptions and queue time.
5. Revisit after enabling quantization or prefix cache.

Worked ballpark (illustrative)

Suppose usable VRAM after weights ≈ 40 GB, KV ≈ 0.5 GB per 8k sequence:

Replace with your engine’s reported KV usage — spreadsheets lie; metrics do not.

Architecture around the engine

Keep auth, quotas, routing, and logging in a gateway — not inside every model process.

flowchart TD
  User --> GW[API gateway: auth, quotas, routes]
  GW --> Cache[Exact / semantic cache]
  Cache -->|miss| Router[Model router]
  Router --> VLLM[vLLM pool A]
  Router --> API2[Frontier API]
  VLLM --> Obs[Metrics: TTFT, TPOT, KV used]

Pair with Redis caching and cost/latency routing.

Multi-pool layout (interactive vs batch)

flowchart LR
  GW[Gateway] --> IQ[Interactive queue]
  GW --> BQ[Batch queue]
  IQ --> P1[vLLM pool chat]
  BQ --> P2[vLLM pool batch]

Same model weights, different max-model-len, priority, and SLOs. This single split fixes more “random TTFT” tickets than swapping engines.

Observability checklist

• TTFT p50/p95, TPOT p50/p95
• Achieved tokens/sec vs GPU util
• KV cache utilization / preemptions
• Queue time vs generate time
• OOM / retry rates
• Per-tenant QPS and token spend
• Chat-template / tokenizer version pinned in deploy metadata
• Prefix-cache hit rate (if enabled)

Ship rule: if you only graph GPU utilization, you will miss queueing and prefill storms.

Failure modes

• OOM on long contexts — lower max len or concurrency; quantize KV/weights
• Starvation — mixed short/long jobs; isolate batch vs interactive
• Tokenizer / chat template mismatch — garbage or refusal loops
• GPU memory utilization too high — intermittent OOM under load
• Ignoring prefill storms — agent tool dumps kill interactive TTFT
• Unbounded max-model-len — “just in case” 128k that never ships but kills batch size
• Silent replica drift — two pods, different quant / template / max-len
• Benchmarking only concurrency=1 — marketing tokens/sec, useless for capacity

Alternatives (same problem class)

Bake-offs must use your prompt length distribution and concurrency — synthetic short prompts flatter every engine. See also TensorRT-LLM and SGLang.

Bake-off checklist

1. Fixed prompt set from production (short / RAG / long).
2. Concurrency sweep: 1, 4, 16, 64.
3. Report TTFT p95 and decode tok/s — never one number.
4. Pin model revision, template, quant, max-len.
5. Repeat after 30 minutes warm (prefix cache / kernels).

Micro-project

Serve a small instruct model with vLLM. Measure tokens/sec at concurrency 1 / 4 / 16. Record max-model-len and when OOMs start. Plot the cliff. Optional: enable prefix caching with a shared system prompt and report TTFT delta.

Related

• Serve your SLM — product path around the engine
• Deploy, cost, latency, observability — Netflix-style serving labs
• Continuous batching · KV-cache · Quantization