Sampling (temperature, top-k/p)

Generation is a policy, not a checkpoint

Training gives you a conditional probability distribution over the next token: logits → softmax → P(token | context). Decoding turns that distribution into a discrete choice. Same checkpoint, different decoding policies, visibly different text.

Product teams tune decoding as much as they tune prompts. A demo that feels " robotic" might need higher temperature; a support bot that hallucinates needs lower temperature and tighter top-p. This lesson makes decoding explicit so you control the knob instead of copying defaults blindly.

Callout — argmax is almost never the right default for creative text: Greedy decoding repeats ("the the the") because the highest-probability token often repeats locally. Stochastic sampling with tuning looks more human.

From logits to a token: the pipeline

At each step:

logits = model(context)[-1]          # (vocab_size,)
scaled = logits / temperature
probs = softmax(scaled)
probs = apply_top_k(probs, k)
probs = apply_top_p(probs, p)
probs = renorm(probs)
token = sample_multinomial(probs)

Order matters in implementations: typically temperature first, then filter, then renormalize, then sample.

Temperature

Temperature T divides logits before softmax:

P_i = exp(logit_i / T) / Σ exp(logit_j / T)

Intuition: logits are log-scores. Dividing by T < 1 amplifies differences (confident); T > 1 shrinks them (uncertain).

Use low T (0.2–0.5) for factual extraction, JSON-like outputs, code completion where syntax matters. Use T ≈ 0.7–1.0 for conversational demos. Use T > 1 only when you want wild variation and can tolerate nonsense.

Top-k sampling

Top-k keeps only the k highest-probability tokens; zero the rest; renormalize.

probs_filtered[token] = probs[token] if rank(token) <= k else 0

Example with k=5: even if token #892 has tiny probability, it is excluded if it ranks below 5th.

Top-k prevents sampling from the long tail of garbage tokens — especially important early in training when tail mass is noisy. Typical values: k=40–50 for open-ended text; k=1 is greedy; k=0 or disabled means no filter.

Downside: fixed k ignores context — when the model is confident (sharp distribution), k=50 still allows 49 unlikely tokens; when uncertain (flat distribution), k=50 might cut off valid options.

Nucleus (top-p) sampling

Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability ≥ p.

Sort tokens by prob descending
Include tokens until cumulative sum >= p
Zero the rest; renormalize

When the model is confident, nucleus might include only 3 tokens. When uncertain, it might include 200. This adapts to context better than fixed k.

Production defaults often combine temperature + top-p (e.g. T=0.8, p=0.95). OpenAI API exposes both; open-source servers (vLLM, llama.cpp) mirror them.

Greedy vs beam search (brief)

Greedy: argmax each step — fast, repetitive.

Beam search: keep top-B partial sequences — common in machine translation; rare in open-ended LLM chat because diversity matters.

For your mini-LLM project, focus on stochastic sampling. Mention beam search only if you compare MT-style tasks.

Ablation methodology

Hold prompt and checkpoint fixed; vary one decoding parameter at a time. Generate N tokens (≥200) per setting; judge readability, repetition, diversity, and factual drift (on tiny models, "factual" means stays in corpus style).

Example grid:

Log type-token ratio (unique tokens / total) as a cheap diversity metric. Higher often means more varied; too high means gibberish.

Callout — document decoding in every sample file: "Checkpoint X, T=0.8, top_p=0.9, prompt=Y" — otherwise samples are not reproducible or comparable.

Defaults for product scenarios

Your tiny LM will look bad at almost all settings compared to frontier models — the ablation teaches relative effects, not absolute quality.

Engineering problem (staff framing)

Decoding turns logits into text. Temperature/top-k/top-p change product behavior as much as prompts.

Diagram — Decoding controls

flowchart TD
  Logits --> Temp[Divide by T]
  Temp --> Mask[Top-k / top-p mask]
  Mask --> Softmax --> Sample

Precise definitions & mental model

• Temperature — T→0 greedy; T↑ flatter.
• Top-k — keep k largest logits.
• Top-p (nucleus) — keep mass ≥ p.
• Stop sequences — product-critical.

Tradeoffs — when to use what

Failure modes (interview + on-call)

High T + tools → invalid JSON; forgetting stop tokens; comparing models at different decoding settings.

Production & OSS practices

Pin decoding in eval harness; expose carefully in UX; log params with traces.

Deep dive (FAANG / OSS bar)

Interaction effects

Top-p with temperature is not commutative in spirit: temperature reshapes mass; nucleus truncates. Always document the pair (T, p) in evals. Comparing two models with different decoding is an invalid bakeoff.

Structured output decoding

For JSON, prefer constrained decoding / grammar guidance when available; otherwise validate-and-retry with T≈0. High temperature is the enemy of schemas.

Micro-project: Ablation table

In m3/sampling/:

1. Load your mini-LLM checkpoint from lesson 3.5.
2. Pick one fixed prompt (20–50 tokens of context from your corpus).
3. Run at least 6 decoding configurations; generate 200 new tokens each.
4. Save outputs in ablation/ and summarize in ABLATION.md table: settings, diversity metric, qualitative rating (1–5 coherence), one example snippet.
5. State your recommended defaults for (a) a portfolio demo and (b) a creative writing toy.

Include at least: greedy/T→0, baseline T=1, top_k only, top_p only, combined production-like settings.

Checklist

• Ablation covers temperature, top-k, and top-p (not all at once only)
• Same prompt and checkpoint across runs
• ABLATION.md with table and recommended defaults
• Decoding params recorded alongside every sample file