Reasoning models / test-time compute

Frontier literacy: spending compute at inference

Reasoning models and test-time compute techniques (extended chain-of-thought, internal deliberation tokens, best-of-N sampling, tree search) improve performance on hard tasks by spending more inference FLOPs and wall-clock time — not by bigger training runs alone. Product implications: latency SLOs break, cost per request spikes, and UX must show "thinking" states — but plan quality on math, code, and multi-step planning may jump.

You will not train o-series models. You benchmark them against standard chat models on fixed hard tasks and document when extra compute is worth it.

What test-time compute means operationally

Technique	Effect	Product cost
Long CoT / reasoning mode	More tokens before answer	↑ latency, ↑ $
Best-of-N + verifier	Sample N, pick winner	N× generation cost
Tool-augmented reasoning	External steps	tool + model cost
Adaptive compute	Stop early on easy inputs	needs router

Measure quality per dollar and quality at p95 latency, not leaderboard bragging only.

Callout — Reasoning mode is not free quality: Easy tasks may get slower with no gain — route hard tasks only.

Hard task benchmark design

Pick 5–10 tasks across types:

• Multi-step arithmetic word problem (no tool)
• Buggy code snippet fix (with tests)
• Ambiguous policy question needing retrieval
• Logic puzzle with distractors
• Planning task (dependency ordering)

Fixed prompt template, temperature 0 where applicable, record:

• Final answer correctness
• Total tokens (input+output)
• Wall time ms
• Tool calls count (if allowed)

Store frontier/benchmark/results.json.

Side-by-side comparison

Run at least two configurations:

• Standard chat model (fast)
• Reasoning / extended model or high reasoning effort setting

Table in frontier/reasoning_report.md:

Task	Fast model	Reasoning model	Δ quality	Δ cost	Δ latency

Include one task where reasoning hurts (overthinking wrong path) — honesty builds credibility.

Product implications

Document decisions for hypothetical product:

• When to auto-enable reasoning (user tier, task classifier)
• UX: progress indicator, cancel button, budget cap
• Fallback if reasoning timeout — degrade to fast model with disclaimer
• Eval strategy — reasoning outputs vary; judges need stability checks

Link to cost accounting and streaming lessons.

When not to use reasoning models

• High-QPS chat where ms matters
• Tasks with deterministic tools (calculator beats CoT)
• Regulated answers requiring cite-only from retrieval

Router for reasoning spend

Implement stub router: if classifier confidence low OR user enables "deep analysis" toggle, use reasoning model; else fast model. Benchmark shows reasoning helps only on hard bucket — justifies router in product memo.

UX for long thinking

Show elapsed seconds and allow cancel — users abort runaway reasoning loops. Log cancelled runs separately — high cancel rate signals router misfire or UX gap.

Eval stability with reasoning

Reasoning outputs vary run-to-run — evals need multiple samples or majority vote for grading; document variance in reasoning_report.md when comparing to temperature-0 fast model.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Reasoning models trade tokens/latency for harder problem performance. Budget them.

Diagram — Reasoning vs flash path

flowchart TD
  Q --> Route{Hard?}
  Route -->|yes| Reason[Reasoning model]
  Route -->|no| Fast[Fast model]

Precise definitions & mental model

Test-time compute, hidden chain vs summary, routing.

Tradeoffs — when to use what

Accuracy vs cost/latency.

Failure modes (interview + on-call)

Always-on reasoning for FAQ; no timeout.

Production & OSS practices

Router + spend caps; eval on hard slice.

Micro-project: Side-by-side hard tasks

Ship:

1. Benchmark task definitions + correctness checks.
2. results.json from ≥2 model/settings.
3. reasoning_report.md with cost/latency/quality tradeoffs.
4. Product implications section (≥5 bullets).
5. Router sketch: when your capstone would enable reasoning mode.

Acceptance: report includes at least one task where reasoning is not worth cost.

Checklist

• ≥5 hard tasks with objective grading
• Side-by-side results for two configurations
• Cost and latency recorded per run
• Product implications documented
• Case where reasoning fails or overcosts analyzed