Reasoning and test-time compute

Train-time vs test-time compute

Scaling laws mostly describe train FLOPs. A parallel lever: spend more inference compute on hard queries — longer chain-of-thought, multiple samples, search, verifiers, or tool-backed checks.

Test-time compute is the product name for that lever. Reasoning-oriented models and agent loops make it first-class: you trade latency and $ for reliability on tasks that greedy short answers fail.

flowchart TD
  Q[Hard query] --> Router{Easy or hard?}
  Router -->|Easy| Fast[Small / greedy decode]
  Router -->|Hard| Sample[Sample n reasoning paths]
  Sample --> Vote[Vote / verify / tool check]
  Vote --> Ans[Final answer]
  Fast --> Ans

Patterns (building blocks)

Pattern	Idea	Cost shape
Chain-of-thought / scratchpad	Model writes intermediate steps	Longer decode
Self-consistency	Sample (n) paths; majority / judge	(n\times) generate
Best-of-(n) + verifier	Score candidates with RM or rules	Generate + score
Tool-augmented reasoning	Code exec, calc, search	Extra RPCs
Search / MCTS-style	Expand trees of thoughts	Highly variable
Process vs outcome rewards	Supervise steps, not only final	Frontier training

sequenceDiagram
  participant U as User
  participant G as Gateway
  participant M as Model
  participant V as Verifier / tools
  U->>G: Hard question
  G->>M: Sample path 1..n
  M-->>G: Candidates
  G->>V: Check / execute
  V-->>G: Pass/fail signals
  G-->>U: Selected answer

Why it works (intuition)

Many errors are decoding noise, not missing weights. Multiple samples explore different reasoning traces; aggregation reduces variance. Tools move exact arithmetic and fresh facts out of the weight matrix. Verifiers catch “fluent but wrong” finals.

Limits: correlated failures (same bias in all samples), verifier gaming, and tasks where the model simply lacks knowledge — then retrieve or refuse.

Production reality

Latency and cost explode

Self-consistency with (n=8) is roughly (8\times) generate cost (plus TTFT quirks if not batched). Reasoning models that emit long hidden/visible chains burn tokens even for “simple” asks if you do not gate them.

Gate expensive paths

Use a router (heuristics, small classifier, or cheap model):

• FAQ / extractive → fast path
• Math, coding, multi-hop → slow path

See cost/latency routing.

Product UX

• Stream partial progress (“working…”) so long thinks feel alive
• Cap max reasoning tokens and max tool loops
• Cache repeated hard queries when answers are stable

Evals

Measure accuracy and $/query, p95 latency, and abstain rate. A 2-point accuracy win that 10×s cost may be wrong for your SLA.

Reasoning models vs agent loops

Approach	Where “compute” lives	Strength
Long CoT model	Tokens inside one generate	Single-call simplicity
Agent + tools	Multiple model/tool steps	External truth, APIs
Hybrid	Short think + tools + verify	Best of both when engineered

Agents are test-time compute with a control plane. See agents and ReAct and multi-agent orchestration. Context packing for long thinks: context engineering.

Failure modes

Failure	Symptom	Mitigation
Unbounded think	Timeouts / bill shock	Hard token + $ budgets
Verbose ≠ correct	Long wrong answers	Verifiers / unit tests
Router miss	Easy Q on slow path	Calibrate on prod traces
Overtrust CoT	Users see flawed rationale	Hide or sanitize traces
Tool loops	Infinite repair	Max steps + circuit breaker

Experiment (project)

On 30 hard questions:

1. Greedy short decode — accuracy, $, latency
2. CoT single sample
3. Self-consistency (n=5) and (n=10)
4. Plot accuracy vs median $

Ship the plot in your design doc; it ends religious debates about “always use the reasoning model.”

Tools and stack literacy

• Provider reasoning / “thinking” modes with token budgets
• Open stacks: sampling + majority vote in your gateway
• Code interpreters / sandboxes as verifiers
• Batch-friendly engines so (n) samples share GPU efficiently (vLLM, continuous batching)

Checklist

1. Is expensive reasoning gated?
2. Hard caps on tokens, samples, tool steps?
3. Eval includes cost and latency, not only accuracy?
4. Clear policy on exposing raw chains to end users?

Related guided path

What’s coming + staying current frontier topics; Build real AI agents agents when tools are the compute; Inference routing articles.