Advanced Key Tech

Triton Inference Server

NVIDIA Triton — multi-framework model serving, ensembles, dynamic batching, and where it sits beside vLLM / TensorRT-LLM.

65 min

Also in guided path: Netflix — in-house LLM serving

What Triton is

NVIDIA Triton Inference Server is a production inference server: load models from many backends (TensorRT, ONNX Runtime, PyTorch, Python, …), expose HTTP/gRPC, apply dynamic batching, and compose ensembles (DAGs of models as one endpoint).

Common in shops already deep on NVIDIA stacks and heterogeneous model zoos — not only generative LLMs.

flowchart LR
  Client --> Triton
  Triton --> B1[TensorRT / ONNX]
  Triton --> B2[Python preprocess]
  Triton --> B3[Embedding model]
  B2 --> B1
  Triton --> Metrics[Prometheus metrics]

Interview cue: Triton is a multi-model serving plane. For large chat decode, specialized engines (vLLM, TensorRT-LLM) often own the LLM leaf; Triton still shines for the embedding / classifier / ensemble zoo beside them.

The engineering problem

Need Triton
One operational plane for many models Model repository + unified metrics
Mix preprocess + infer + postprocess Ensembles / BLS
Throughput under load Dynamic batching
GPU sharing across small models Concurrent model execution
Versioned promote of ONNX/TRT artifacts Repository versions

Without something like Triton you run N ad-hoc Flask/FastAPI wrappers — each with its own batching story, metrics, and readiness probes. That works until the model count and on-call pain explode.

Architecture

Concept Meaning
Model repository Versioned directory tree Triton loads
Backend Runtime plugin (tensorrt, onnxruntime, python, …)
Dynamic batching Prefer throughput by batching in a time window
Ensemble DAG of models presented as one model name
BLS / Python Control flow and light logic in-server
Instance groups How many copies on which GPU/CPU
Rate limiter / queue Protect GPUs under overload 
flowchart TB
  subgraph repo [Model repository]
    EmbV[embed / 1]
    ClfV[classify / 3]
    Ens[rag_preprocess ensemble]
  end
  HTTP[HTTP/gRPC] --> Triton[Triton]
  Triton --> repo

Ensemble example (RAG-ish)

flowchart LR
  Tok[tokenize] --> Emb[embed]
  Emb --> Out[vectors out]

Client calls one ensemble name; Triton runs the pipeline. ANN search usually stays a sidecar (vector DB) rather than inside Triton — keep responsibilities clear (vector databases).

Dynamic batching vs latency

Knob Effect
Preferred batch size Higher GPU util
Max queue delay Caps wait before firing a partial batch
Separate model instances Isolate interactive vs batch SLOs

Ship rule: one dynamic-batch window for “chat autocomplete embed” and “nightly 10M chunk re-embed” will make interactive p95 look random. Split models or instance groups.

How it fits LLM apps

Component Typical hosting
Embedding models Triton (batching wins)
Classifiers / rerankers / safety Triton
LLM decode vLLM / TRT-LLM (± Triton or gateway in front)
Classic CV / ASR Triton
Light preprocess Python / BLS backend 
flowchart TB
  GW[API gateway] --> Router{Route}
  Router -->|embed / rank / classify| Triton
  Router -->|chat decode| Engine[vLLM / TRT-LLM / SGLang]
  Router -->|closed API| Cloud[OpenAI / Anthropic]
  Triton --> VDB[(Vector DB sidecar)]
  Engine --> OTel[OTel + Prometheus]
  Triton --> OTel

Industry labs (e.g. Netflix-style serving stories in How real companies use AI) often show gateways + specialized engines + shared metrics — Triton is one mature option for the non-chat zoo.

How to use (ops shape)

models/
  embedder/
    config.pbtxt
    1/
      model.onnx
  rag_preprocess/
    config.pbtxt          # ensemble / BLS

config.pbtxt declares backend, input/output tensors, dynamic_batching { }, and instance groups (GPU/CPU). Clients use HTTP /v2/models/{name}/infer or gRPC — pin the protocol version your SDK expects.

# Pseudocode — client infer
# inputs = serialize tensors
# POST /v2/models/embedder/versions/1/infer
# parse outputs; always log model name + version on the trace
#
# span.set_attribute("triton.model", "embedder")
# span.set_attribute("triton.version", "1")

Capacity planning sketch

  1. Measure p95 input size (tokens / image dims) and QPS by model.
  2. Load-test with and without dynamic batching; record latency histograms.
  3. Size instance groups so GPU memory holds concurrent models without fragmentation thrash.
  4. Define readiness: warm-up complete before traffic.
  5. Promote repository versions like container images — staging golden prompts first.

Decision tree: where does the model live?

Signal Lean toward
Generative chat primary, OpenAI-compatible vLLM / SGLang
Peak NVIDIA engine + existing Triton culture TRT-LLM backend / Triton leaf
Many small ONNX/TRT models + ensembles Triton
Python product graph, multi-step routing Ray Serve in front
Managed, no GPU ops Cloud endpoints

Failure modes

  • Warm-up / cold start for large models — readiness probes lie until first infer
  • Batching vs latency — max queue delay hurts TTFT; split interactive vs batch
  • Version pins — clients must request the intended model version
  • Python backend GIL / perf — keep heavy math in TensorRT/ONNX
  • GPU memory fragmentation across many concurrent models
  • Ensemble opacity — one endpoint hides which stage failed
  • Missing metrics correlation — scrape Triton; join request ids via OpenTelemetry

Production checklist

  1. Repository layout + versioning documented; never “overwrite /1 in place.”
  2. Dynamic batching configs reviewed per SLO class.
  3. Prometheus metrics: queue time, infer time, GPU util, fail count.
  4. Readiness includes warm-up infer for heavy models.
  5. Client SDKs pin model name + version.
  6. Ensembles stay thin — ANN / business logic outside when possible.
  7. Rollback = previous repository version + previous config.pbtxt.

Alternatives

Stack When
vLLM / TGI / SGLang Generative LLM primary workload
TorchServe / BentoML Python-centric serving, less NVIDIA-specific
Ray Serve Python deployment graphs, multi-model product logic
Cloud endpoints Managed; less server ops

Micro-project

  1. Write an ensemble diagram: tokenize → embed → (external) ANN.
  2. Decide what stays in Triton vs vLLM vs vector DB — one paragraph each.
  3. List three Prometheus signals you’d alert on for an embedding model.
  4. Industry case labs + Inference track for the decode side.

TensorRT-LLM and SGLang · vLLM · Ray · Continuous batching

Project checklist0/3 done