Continuous batching

Static batching wastes the GPU

If you wait to fill a batch of size (B), short requests stall behind the slowest filler. If you run batch=1, the GPU underutilizes during decode while other users wait in a queue.

Continuous batching (iteration-level scheduling) inserts new sequences into the running batch as others complete — core to vLLM-class engines and peers (TGI, TensorRT-LLM in-flight batching, SGLang).

gantt
  title Continuous batching intuition
  dateFormat X
  axisFormat %s
  section GPU
  ReqA decode     :a1, 0, 8
  ReqB decode     :a2, 2, 6
  ReqC decode     :a3, 5, 9

Interview cue: Continuous batching is iteration-level scheduling, not “bigger static batches.” Sequences enter and leave every decode step.

The problem in product terms

Serving is a queueing + packing problem. The GPU is happiest when every step has useful work; users are happiest when short jobs finish early.

Static vs continuous — side by side

flowchart LR
  subgraph Static
    W[Wait for B] --> R[Run until all done]
    R --> N[Next batch]
  end
  subgraph Continuous
    Q[Queue] --> S[Scheduler each step]
    S --> Mix[Mixed decode batch]
    Mix --> Free[Free finished KV]
    Free --> S
  end

Dynamic batching vs continuous batching (naming trap)

Vendors overload “dynamic batching.” Sometimes it still means “wait up to N ms to fill a static batch.” Continuous / in-flight / iteration-level means admit mid-flight. Ask: can a new request join while others are still decoding?

Why it couples to KV management

You cannot freely add sequences unless you can allocate and free KV without fragmenting the GPU heap. That is why PagedAttention-style block allocators and continuous batching ship together:

1. Request arrives → reserve KV blocks for max planned length (or grow)
2. Each decode step → advance tokens; append KV blocks as needed
3. Request finishes / cancels → free blocks immediately for newcomers

Without paging, “continuous” admission still dies on fragmentation. See KV-cache, prefill, and decode.

flowchart TD
  Arrive[Request arrives] --> Reserve[Reserve KV blocks]
  Reserve --> Step[Decode / prefill step]
  Step --> Grow{Need more blocks?}
  Grow -->|yes| Alloc[Allocate blocks]
  Grow -->|no| Step
  Alloc --> Step
  Step --> Done{Finished?}
  Done -->|yes| Free[Free blocks]
  Done -->|no| Step
  Free --> Next[Admit waiting request]

Prefill vs decode inside the scheduler

Real engines mix prefill chunks and decode steps. Policies differ:

Ship rule: continuous batching raises aggregate tokens/sec. It does not automatically fix p95 TTFT if prefills are unbounded — still isolate heavy jobs.

Token budget per step

max_num_batched_tokens (name varies) caps how many tokens the engine processes in one scheduler iteration — across prefill chunks and decode tokens. Too low → underutilization. Too high → one fat prefill monopolizes the step.

Tune with a realistic mix, not identical 50-token prompts.

Knobs product engineers actually set

Load-test with a realistic mix: 80% short chat, 15% RAG, 5% long dump — synthetic identical prompts hide scheduler pathologies.

# Shape — synthetic load mix for a bake-off
mix = [
    ("short", 0.80, 64, 128),    # name, weight, prompt_tok, out_tok
    ("rag", 0.15, 4096, 256),
    ("dump", 0.05, 32000, 512),
]

Fairness and multi-tenancy

Continuous batching without limits → noisy-neighbor hell:

• One agent loop with huge tool dumps fills KV
• Free-tier scrapers raise everyone’s queue time
• Offline eval jobs sneak onto the interactive pool

Patterns:

1. Separate pools (interactive GPU vs batch GPU)
2. Admission control (max concurrent per tenant)
3. Token budgets at the gateway (cost/latency routing)
4. Preemption (pause low-priority when KV tight) — engine-dependent

flowchart TD
  GW[Gateway quotas] --> IQ[Interactive queue]
  GW --> BQ[Batch queue]
  IQ --> EngA[Engine pool A]
  BQ --> EngB[Engine pool B]
  EngA --> KV[Paged KV]
  EngB --> KV2[Paged KV]

Per-tenant caps (minimal policy)

Engines schedule tokens; gateways schedule business rules.

Timeline sketch (use this for the micro-project)

Five requests, rough wall clock:

Draw your own Gantt with different arrival times — the continuous column should show early finishers freeing capacity.

gantt
  title Five-request contrast intuition
  dateFormat X
  axisFormat %s
  section Static B4
  Wait ABD for C :s0, 0, 3
  Run ABCD :s1, 3, 9
  section Continuous
  A :c1, 0, 5
  B :c2, 1, 6
  C :c3, 2, 4
  D :c4, 4, 8

What “good” looks like in metrics

Failure modes

• Calling “dynamic batching” when you still wait for a full static batch
• Raising max_num_seqs until OOM / thrash without measuring p95
• Ignoring prefill chunking → TTFT spikes under load
• No per-tenant caps → one customer owns the batch
• Benchmarking only throughput on identical short prompts
• One pool for interactive + overnight summarization
• Assuming continuous batching fixes a too-large max-model-len

Micro-project

Contrast static vs continuous batching with a 5-request timeline sketch (Gantt or table). Annotate when each request’s first and last token occur. Optional: add a 6th long-prefill request and show how chunking / separate queues change the sketch.

Related

• KV-cache, prefill, and decode
• vLLM
• Serving and streaming
• API and streaming