Advanced Key Tech

TensorRT-LLM and SGLang

High-performance inference stacks beyond vanilla vLLM — NVIDIA TensorRT-LLM and SGLang’s structured generation runtime.

70 min

Also in guided path: NVIDIA — agent toolkits and eval/obs

What these stacks are

vLLM is the default open high-throughput engine for many teams. Two other names show up in serious serving stacks:

Stack What it is Gravity
TensorRT-LLM (TRT-LLM) NVIDIA’s LLM inference toolkit: build optimized engines, run with high GPU efficiency, often beside Triton Peak NVIDIA performance, enterprise paths
SGLang Fast serving runtime with strong structured / constrained generation and “programs over LLMs” JSON schemas, multi-call graphs, radix-cache reuse

Neither replaces your product gateway, auth, or evals. They replace (or sit under) the decode engine when vanilla Hugging Face generate or an untuned server cannot meet latency/$ SLOs.

flowchart TD
  App[App / gateway] --> Eng{Engine}
  Eng --> vLLM[vLLM]
  Eng --> TRT[TensorRT-LLM]
  Eng --> SGL[SGLang]
  Eng --> Triton[Triton ensemble]

Interview cue: Separate engine (how tokens burn on GPU) from platform (Ray/K8s/Triton ops). Picking TRT-LLM vs SGLang is an engine decision.

The engineering problem

Pain Typical fix
GPU under-utilized under multi-tenant chat Continuous batching + paged KV (vLLM or TRT-LLM)
Need last % of NVIDIA throughput / FP8 paths TensorRT-LLM engines
Strict JSON / grammar / multi-step LLM programs SGLang constrained decoding + runtime
Heterogeneous zoo (embed + classify + LLM) Triton front door; TRT-LLM for the LLM leaf
Shared system + tool prefixes dominate cost Prefix / radix caching (SGLang, vLLM prefix cache)

Sibling inference articles: KV-cache / prefill / decode, continuous batching, quantization, speculative decoding.

Architecture: TensorRT-LLM

Mental model:

  1. Take HF (or similar) weights
  2. Build an optimized engine for a target GPU + precision (FP16/FP8/INT4…)
  3. Serve that engine (standalone runtime, Triton backend, or cloud NIM-style packaging)
  4. Clients speak HTTP/gRPC — often OpenAI-shaped via a gateway
flowchart LR
  HF[HF checkpoint] --> Build[trtllm-build / compile]
  Build --> Eng[Engine artifact]
  Eng --> Runtime[TRT-LLM runtime / Triton]
  Runtime --> Client[Gateway / app]
Concept Why it matters
Engine build Compile-time specialization → fast serve, slower iteration
Precision / quant Memory and throughput vs quality (quantization)
In-flight batching Same family of ideas as continuous batching
KV cache management Prefill vs decode economics (KV cache)
GPU SKU pin Engine built for A100 ≠ free lunch on H100 without rebuild

Ship rule: treat engine builds like container images — version, pin GPU SKU, promote through staging with golden prompts.

Build vs serve tradeoff

Phase Cost Benefit
Build / compile Minutes–hours; CI artifact Peak kernels for that GPU + precision
Serve Fast path at runtime Predictable latency/$
Rebuild on every tiny weight change Painful Only when quality/SLO demand it

For rapid LoRA / prompt iteration, many teams prototype on vLLM and promote hot models to TRT-LLM once traffic stabilizes.

Architecture: SGLang

SGLang optimizes runtime programs over LLMs: multi-call flows, constrained decoding, and aggressive prefix / radix caching when prompts share structure (system prompts, tool schemas, RAG templates).

flowchart TB
  Prog[SGLang program / API] --> Sched[Scheduler + cache]
  Sched --> Decode[Decode workers]
  Constr[JSON / grammar constraints] --> Decode
  Cache[(Radix / prefix cache)] <--> Sched
Strength Use when…
Constrained outputs Structured agents, tool args, form fill
Shared prefixes High cache hit on stable system + schema
Multi-step LLM graphs Fewer round-trips through your app server

Pair with structured outputs literacy — the runtime enforces; you still design the schema.

Radix / prefix cache intuition

flowchart TD
  Sys[System prompt] --> Shared[Cached prefix]
  Tools[Tool schema] --> Shared
  Shared --> U1[User A continuation]
  Shared --> U2[User B continuation]

When 80% of every request shares the same system + tools block, caching that prefix dominates TTFT and GPU save. Schema churn → cold cache → latency cliff — version tool schemas deliberately.

How they fit an LLM product

flowchart TB
  GW[API gateway] --> Router{Route}
  Router -->|chat open weights| Eng[vLLM / TRT-LLM / SGLang]
  Router -->|embed / rank| Triton[Triton or small servers]
  Router -->|closed API| Cloud[OpenAI / Anthropic]
  Eng --> OTel[OTel spans + tokens]
Layer Owns
Gateway Auth, quotas, streaming, model routing
Engine Prefill/decode, batching, constraints
Sidecars Embeddings, rerankers, safety classifiers
Obs TTFT, tokens/sec, cache hit, engine version

Industry NVIDIA labs (NVIDIA — agent toolkits and eval/obs) often show specialized engines + shared metrics — same pattern.

Decision + ops shape

Pick engine (one-pager exercise):

Signal Lean toward
Fastest path to OpenAI-compatible self-host vLLM
Already on Triton + NVIDIA optimization culture TensorRT-LLM
Constrained decoding / LLM programs are the product SGLang (evaluate)
Mixed classic ML + LLM Triton ensemble + LLM leaf engine
Local / CPU demos Ollama / llama.cpp

Ops checklist for any of them:

1. Pin model id + revision + engine/build hash
2. Separate interactive vs batch queues
3. Export TTFT, TPOT, queue time, GPU util, cache hit
4. Golden-set gate before promote
5. Rollback = previous engine artifact, not “git revert hope”
# Shape — client stays OpenAI-compatible where possible
# client = OpenAI(base_url="http://engine:8000/v1", api_key="unused")
# resp = client.chat.completions.create(model="meta-llama/...", messages=[...])
# Log: model, engine_name, engine_version, ttft_ms, output_tokens

Bake-off protocol (minimum)

  1. Capture your prompt length distribution and concurrency.
  2. Fix model revision + chat template.
  3. Measure TTFT p50/p95, TPOT, tokens/sec, error rate, quality on a golden set.
  4. Repeat with structured-output load if that is the product.
  5. Pick on task metrics + p95, not a blog’s tokens/sec screenshot.

Failure modes

  • Build/serve skew — engine built for A100 served on H100 without rebuild
  • Quant surprise — FP8/INT4 wins throughput, loses edge-case reasoning; measure task metrics
  • Constraint ≠ correctness — valid JSON can still be wrong business logic
  • Cache invalidation — prompt/tool schema change → cold radix cache → latency cliff
  • Tokenizer mismatch with training / chat template (Hugging Face)
  • Over-focus on tokens/sec — ignore TTFT and p99 under concurrency
  • Missing engine version in traces — cannot bisect regressions (OpenTelemetry for LLMs)
  • Speculative decoding hype — acceptance rate too low → net loss (speculative decoding)

Production checklist

  1. Engine/build hash in deploy metadata and OTel attributes.
  2. Interactive vs batch isolation.
  3. Golden prompts + task evals gate every promote.
  4. Quantization quality measured on hard cases, not only format tasks.
  5. Prefix/tool schema versioning to protect cache hit rate.
  6. Rollback artifact retained for N releases.
  7. Capacity plan from KV-cache math, not vibes.

Alternatives

Stack When
vLLM Default open self-host; strong continuous batching
TGI / other servers Team already standardized
llama.cpp / Ollama Local / CPU-friendly demos (Ollama)
Hosted APIs No GPU ops; pay per token (OpenAI / Anthropic)
Ray Serve front Multi-model product graph in front of engines (Ray)

Micro-project

  1. Write a one-pager: vLLM vs TRT-LLM vs SGLang for your traffic (QPS, TTFT, structured needs).
  2. List three shared prefixes you’d want cached (system, tools, RAG header).
  3. Sketch where Triton stops and the LLM engine starts.
  4. Inference track: continuous batching, speculative decoding.

Triton Inference Server · vLLM · Inference track · NVIDIA — agent toolkits and eval/obs

Project checklist0/3 done