Mixture of Experts (MoE)

Dense vs sparse — the core idea

In a dense Transformer, every token runs through the same feed-forward network (FFN) weights. Capacity scales with FFN size; so does FLOPs per token.

A Mixture of Experts (MoE) replaces (or augments) that FFN with a pool of expert FFNs plus a router. Each token activates only a small subset (e.g. top-(k) of (N) experts). Result:

• Total parameters ↑ (many experts)
• FLOPs per token grow much more slowly (only (k) experts run)
• Serving and training get harder (routing, load balance, expert parallelism)

flowchart TD
  Tok[Token hidden state] --> Gate[Router / gate]
  Gate -->|top-k scores| E1[Expert 1]
  Gate --> E2[Expert 2]
  Gate --> Ek[Expert k of N]
  E1 --> Comb[Weighted combine]
  E2 --> Comb
  Ek --> Comb
  Comb --> Out[Next layer input]

Interview cue: MoE is “sparse compute, dense-looking parameter count.” Quoting “trillion parameters” without active params / tokens is incomplete.

How routing works (intuition)

For token representation (x):

1. Router computes scores over (N) experts (often a linear layer + softmax or sigmoid variants)
2. Select top-(k) experts (commonly (k=1) or (2))
3. Run those expert FFNs
4. Combine outputs with router weights

Training adds load-balancing losses or constraints so a few experts do not monopolize all tokens (collapse).

Term	Meaning
Experts	Specialized FFN parameter sets
Active params	Params touched for one token
Capacity factor	How many tokens an expert may accept per step
Expert parallelism	Shard experts across GPUs/hosts

Why labs ship MoE

flowchart LR
  Goal[More capacity] --> Dense[Dense: all FLOPs scale]
  Goal --> MoE[MoE: params scale faster than FLOPs]
  MoE --> Train[Train: more knowledge per FLOP]
  MoE --> Serve[Serve: careful scheduling]

Relative to dense models at similar train FLOPs, MoE can absorb more total parameters — useful for multilingual, code+natural mix, or long-tail skills — if routing learns useful specialization.

Trade: systems complexity moves into the critical path.

Production reality: serving MoE is not free

Topic	Why it hurts in prod
Memory	Many params often resident even if compute is sparse
Load imbalance	Hot experts → stragglers, wasted capacity
Networking	Expert-parallel all-to-all style traffic
Batching	Tokens in a batch take different experts → irregular kernels
Debugging	Behavior depends on “which expert fired”
Quantization	Expert-wise sensitivity; validate quality carefully

Inference engines (vLLM, TensorRT-LLM, SGLang, and lab-internal stacks) add MoE-specific kernels and scheduling. See vLLM, continuous batching, TensorRT-LLM and SGLang.

sequenceDiagram
  participant S as Scheduler
  participant R as Router
  participant G as GPU group
  S->>R: Batch of tokens
  R->>R: Assign experts
  R->>G: Dispatch uneven expert loads
  Note over G: Hot expert = latency tail
  G-->>S: Combined hidden states

MoE vs dense — decision table

Prefer dense when…	Prefer MoE when…
Simple serving, predictable latency	You need max quality per train FLOP
Small/med models on one GPU	Frontier-scale training budgets
Easy quant + distill story	Provider already ships MoE API/weights
Debugging must stay simple	You have expert-parallel infra

Product teams often consume MoE via APIs without operating routers — still know the failure modes when latency tails spike or “70B-active-class” marketing appears.

Failure modes

1. Expert collapse — router always picks the same experts; capacity wasted
2. Train/serve skew — different batching changes effective routing stats
3. Tokenizer/domain shift — new traffic overloads experts tuned on old mix
4. False equivalence — comparing dense 70B to MoE “141B” on params only

How to explain MoE in 8 sentences (project)

Use this as a teammate briefing outline:

1. Dense FFNs use the same weights for every token.
2. MoE has many expert FFNs and a router.
3. Each token runs only top-(k) experts.
4. Total parameters grow; FLOPs per token grow slower.
5. Training needs load balancing so experts stay utilized.
6. Serving needs expert parallelism and careful batching.
7. Memory can stay high even when compute is sparse.
8. Evaluate on active compute + quality, not headline parameter count.

Cross-links

• Scaling laws and compute — params vs FLOPs
• Open-weight vs APIs — how MoE shows up in model cards
• Industry Meta/Google literacy labs (How real companies use AI)
• Inference batching articles — routing adds scheduling complexity

Checklist

1. When reading a model card, can you find #experts, top-(k), active params?
2. For self-host: do you have memory headroom for all experts you load?
3. Are latency SLOs defined on p95/p99, not only mean tokens/sec?
4. Do evals cover domains that might stress rare experts?

Related

Scaling laws; Meta/Google literacy labs; Inference batching and engine pages.