Speculative decoding

Idea

Autoregressive decode is serial on the big model: one forward pass → one (or few) tokens. Speculative decoding lets a cheap draft model propose several tokens; the target model verifies them in parallel (one forward over the draft span). Accepted tokens are “free” speed; rejected tokens fall back and you continue.

sequenceDiagram
  participant Draft
  participant Target
  Draft->>Draft: propose k tokens
  Draft->>Target: verify block
  Target-->>Draft: accept prefix / reject

Mathematically, verification is designed so the output distribution matches the target (lossless when implemented correctly) — you are not “approximating” the big model’s distribution if acceptance sampling is done right. You are buying wall-clock speed when the draft is often right.

Interview cue: Speculative decoding speeds decode when draft acceptance is high. It is not a substitute for continuous batching or better routing.

Why decode is the pain point

If your traffic is RAG-heavy with huge prompts and short answers, fix KV / prefill and routing first.

Algorithm sketch

1. Draft model generates (k) candidate tokens given the current prefix.
2. Target model runs one forward pass that scores those positions (plus maybe a bonus token).
3. Accept the longest prefix consistent with target probabilities (scheme-dependent).
4. On rejection, keep the corrected token from the target and repeat.

flowchart TD
  Prefix[Current prefix] --> Draft[Draft proposes t1..tk]
  Draft --> Verify[Target verifies in parallel]
  Verify --> Acc{Accept length?}
  Acc -->|m of k| Emit[Emit m tokens]
  Acc -->|0| Fix[Emit target correction]
  Emit --> Prefix
  Fix --> Prefix

Variants you will hear about

Engines (vLLM, TensorRT-LLM, etc.) expose different knobs — read their docs for the exact acceptance rule and whether lossless mode is on.

Acceptance rate is the gate

Measure acceptance rate = tokens accepted / tokens drafted (plus mean accepted stretch).

# Shape — log per request
span = {
    "draft_tokens": k_total,
    "accepted_tokens": m_total,
    "acceptance_rate": m_total / max(k_total, 1),
    "mean_accept_stretch": mean_stretch,
    "tok_s_vs_baseline": ratio,
}

Ship rule: gate enablement on acceptance + net tok/s, not on a blog’s “2× faster” claim.

When it wins

When it loses

flowchart LR
  Traffic[Traffic mix] --> Pred{Predictable?}
  Pred -->|yes| Try[Try speculative]
  Pred -->|no| Skip[Prefer batching + quant + route]
  Try --> Acc{Acceptance high?}
  Acc -->|yes| Keep[Keep + monitor]
  Acc -->|no| Disable[Disable for that route]

Cost model (intuition)

Per “successful” span of (m) accepted tokens you paid roughly:

• Draft compute for (k) tokens
• One target forward over the span (not (m) serial target steps)

If (m) is usually close to (k), wall-clock decode drops. If (m\approx 0), you paid draft + target for almost nothing.

Choosing (k)

Sweep (k) offline; keep per-route configs.

Ship rule: enable speculative decoding per route / model pair, not globally. Creative chat and JSON tool calls may disagree.

Pairing with the rest of the stack

See quantization for inference and cost/latency routing.

flowchart TB
  GW[Gateway route] --> Spec{Spec enabled for route?}
  Spec -->|yes| Pair[Draft + target on engine]
  Spec -->|no| TargetOnly[Target only]
  Pair --> Metrics[Acceptance + tok/s]
  TargetOnly --> Metrics

VRAM reality check

Loading two models (draft + target) costs memory you could have spent on concurrency. Sometimes a smaller draft on CPU/other device is offered — measure end-to-end, including PCIe. Sometimes Medusa-style heads avoid a second model — different failure modes.

Observability

Log:

• Acceptance rate (overall + per route)
• Mean accepted stretch length
• Tokens/sec vs baseline canary
• TTFT (should be ~unchanged; if not, look at scheduler interactions)
• Quality canary — confirm lossless config is actually enabled
• VRAM / max_num_seqs before vs after

Canary plan

1. Shadow: run speculative on a copy of traffic offline; compare outputs if claiming lossless.
2. Online: 5–10% of a predictable route; watch acceptance + user thumbs / task success.
3. Kill switch: feature flag per route.

Failure modes

• Draft/target tokenizer or chat-template mismatch → garbage acceptance
• Claiming speedup without holding quality/distribution constant
• Enabling on RAG-heavy prefills and declaring “speculative doesn’t work”
• Oversized (k) with low acceptance → pure overhead
• Ignoring VRAM of loading two models on the same GPU
• Global enable across creative + tool routes
• No acceptance metric → flying blind

Micro-project

Draw a diagram for your traffic mix (e.g. 50% FAQ, 30% RAG, 20% creative). Mark where speculative decoding helps vs hurts; list the metric you would gate enablement on (acceptance rate threshold) and a kill-switch plan.

Related

• vLLM · Continuous batching
• Serve your SLM
• TensorRT-LLM and SGLang for engine-specific features