Quantization for inference

Why quantize at serve time

Weights and KV caches dominate VRAM. Lower precision → more concurrent sequences or larger context — if quality holds for your evals. Training-time QLoRA is related but different: here we care about serving latency, concurrency, and task metrics.

flowchart TD
  FP16[FP16 / BF16 weights] --> Q[Quantize]
  Q --> W4[4/8-bit weights]
  W4 --> Serve[Higher batch / longer context]
  Serve --> Eval{Task evals OK?}
  Eval -->|no| Back[Less aggressive quant / calibrate]
  Eval -->|yes| Ship[Ship with canary]

Interview cue: Quantization is a quality ↔ capacity trade. “We fit 4-bit” is not a win until tool-calling / RAG faithfulness still pass.

The product question (not the blog question)

Ask in this order:

1. What fails if quality drops 2%? (refunds vs emoji captions)
2. How much concurrency / context do we unlock?
3. Can we measure it on a pinned eval suite before and after?
4. Can we canary under the same model id policy (or a new id)?

If you cannot answer (1) and (3), do not ship aggressive quant.

What you can quantize

Weight-only INT4 is common on GPUs (AWQ/GPTQ). FP8 weights/activations/KV show up on modern NVIDIA stacks. GGUF paths dominate local CPU/Apple via llama.cpp / Ollama.

flowchart TB
  subgraph Memory
    W[Weights]
    KV[KV cache]
    A[Activations]
  end
  W --> Cap[Concurrency / context headroom]
  KV --> Cap
  A --> Speed[Step latency on some HW]

Common families

flowchart LR
  Labs[Lab / laptop] --> GGUF[GGUF Q4/Q5]
  GPU[Multi-tenant GPU API] --> AWQ[AWQ / GPTQ / FP8]
  Edge[Edge CPU] --> GGUF
  Train[Train adapters] --> QLoRA[QLoRA — different stage]

Weight-only vs weight+activation

How much VRAM do you actually save?

Order-of-magnitude for weights only (ignore KV):

Real checkpoints add overhead (scales, packing, unused embed shards). KV still grows with context × concurrency — quantizing weights alone does not fix a 128k-context concurrency cliff. Combine with KV-cache planning and continuous batching.

Worked capacity intuition

Calibration and data

GPTQ/AWQ-style methods need a calibration corpus:

• Prefer samples that look like production prompts (tools, JSON, RAG packs)
• Random Wikipedia-only calibration underestimates tool-calling damage
• Pin calib set version next to the quant artifact hash
• Include multilingual / code slices if those are product paths

artifact: llama3-8b-instruct-awq-w4a16
calib: prod-prompts-v3-n512
eval: toolcall-suite-v2 + rag-faithfulness-v1
hash: sha256:...

Ship rule: a quantized checkpoint without calib+eval metadata is an untraceable regression waiting to happen.

Quality evaluation (non-negotiable)

Do not ship on perplexity alone.

Compare same prompts, same decoding params (temperature, stop, max tokens). Log retry rates — noisier logits often show up as more “fix + regenerate” loops.

# Shape — paired eval FP16 vs INT4
rows = []
for prompt in suite:
    a = complete(model_fp16, prompt, **decode_params)
    b = complete(model_w4, prompt, **decode_params)
    rows.append({
        "id": prompt.id,
        "ok_fp16": grade(a),
        "ok_w4": grade(b),
        "tool_valid_w4": schema_ok(b),
    })

Serving integration

Ship rule: treat quantized weights as a new model id in routing, caches, and eval dashboards — never silently swap FP16 → INT4 behind the same name.

flowchart LR
  Router[Gateway router] --> ID1[model: llama3-8b-fp16]
  Router --> ID2[model: llama3-8b-awq]
  ID1 --> Eval[Separate eval + $ dashboards]
  ID2 --> Eval

Throughput expectations (measure, don’t assume)

Quant can help because:

• More sequences fit → better continuous batching packing
• Less memory traffic for weights on some kernels
• Or… kernel path is slower and you lose tok/s at low concurrency

Always report tok/s at the concurrency you actually run.

When not to quantize aggressively

• Regulated answers where small factual drift is unacceptable without human review
• Brand-new model with no calibration data yet
• Already-memory-comfortable SLM where quality > another +20% concurrency
• Speculative decoding draft/target mismatch made worse by heavy quant (measure acceptance)
• You cannot run a canary or paired eval suite

Failure modes

• Silent quality regression on tool-calling / JSON
• Tokenizer / template unchanged but logits noisier → higher retry rate
• Comparing perplexity only — always use task metrics
• Mixing quant variants in one replica set without pinning
• Ignoring KV memory — “4-bit fits” then OOM at concurrency 8 with long prompts
• Calibrating on the wrong domain
• Reusing the same model string after a quant swap → poisoned caches / wrong FinOps

Micro-project

Serve one model at 16-bit and 4-bit. Compare tokens/sec and a 50-prompt quality set (include ≥10 tool/JSON prompts). Record which failures appeared only at 4-bit. Report concurrency headroom gained on your GPU.

Related

• Guided Quantization tradeoffs
• vLLM · Continuous batching
• Fine-tuning LoRA / QLoRA — training-time cousin, different lifecycle