Scaling laws and compute

The engineering problem

Frontier models feel like magic. Product teams then ask: can we just train a smaller GPT on our laptop and get the same behavior? Scaling laws are the honest answer: within a regime, test loss (and often capability) falls as a power law of parameters (N), data (D), and train compute (C). Outside that regime — wrong data mix, under-trained small models, or capability cliffs — the curve bends.

This article is the systems literacy you need before arguing budget, choosing “7B vs API,” or explaining why Build an LLM from scratch’s tiny GPT is a teaching artifact, not a product base.

What “scaling laws” actually claim

Empirically, across many LLM pretraining runs:

[ L(N, D) \approx A N^{-\alpha} + B D^{-\beta} + E ]

(and similar forms with compute (C)). Intuition:

Axis	What it buys	What it does not buy alone
Parameters (N)	Capacity / representation power	Useful behavior without enough tokens
Data (D)	Coverage of language & skills	Quality if the corpus is garbage
Train compute (C)	Joint optimization of (N) and (D)	Guaranteed product metrics (latency, safety)

flowchart LR
  N[Parameters N] --> L[Lower test loss]
  D[Tokens / data D] --> L
  C[Train FLOPs C] --> L
  L --> Cap[Emergent-ish capabilities]
  Cap --> Prod[Product usefulness]
  Qual[Data quality / mix] --> Cap
  Align[SFT + preference] --> Prod

Ship rule: scaling laws explain pretrain loss trends. Product success also needs alignment, tools, retrieval, and evals — none of which show up in a clean (L(N)) plot.

Chinchilla-style intuition (without memorizing papers)

A classic failure mode: train a huge model on too few tokens (under-trained) or a tiny model on endless tokens (capacity-bound). Rough mental model used in industry debates:

1. Compute-optimal training balances (N) and (D) for a fixed FLOP budget.
2. Overtrained small models (more tokens than “optimal”) are often better for inference economics — you pay once at train time, serve cheaply forever.
3. Undertrained giants look impressive on parameter count and disappoint on quality per dollar.

For builders: when a vendor says “70B,” ask how many tokens, what mix, and what post-train. Parameter count without data/compute story is marketing.

Three compute budgets product engineers confuse

flowchart TD
  subgraph train [Train-time compute]
    PT[Pretrain]
    SFT[SFT / LoRA]
    Pref[Preference / RL]
  end
  subgraph serve [Serve-time compute]
    Prefill[Prefill]
    Decode[Decode]
    TTC[Test-time reasoning / search]
  end
  Budget[Org $ / GPU hours] --> train
  Budget --> serve
  train --> Model[Shipped weights]
  Model --> serve

Budget	Who owns it	Failure if ignored
Pretrain	Labs / hyperscalers	You cannot “catch ChatGPT” with a weekend job
Fine-tune / PEFT	Your ML platform	Overfit + catastrophic forgetting
Inference	Your product SRE	Unit economics blow up at scale

Interview cue: Separate train FLOPs from serve FLOPs. Reasoning models and agent loops move cost into inference — see reasoning and test-time compute and cost/latency routing.

Why your tiny GPT is not ChatGPT

Guided Build an LLM from scratch builds a mini-LLM so you feel tokenization, attention, and training loops. It deliberately does not match frontier capability because:

1. Orders of magnitude less (N), (D), and (C)
2. No large-scale data curation, filtering, or dedup pipeline
3. No SFT + preference stack that turns a next-token model into an assistant
4. No inference stack optimized for batching, KV-cache, and speculative decode

Treat the mini model as a microscope, not a competitor.

Data quality vs parameter count (product reality)

For narrow enterprise tasks, clean domain data + RAG + light PEFT often beats “just buy a bigger base model”:

Approach	When it wins	When it loses
Bigger API model	Broad reasoning, weak domain corpus	Cost, latency, data residency
Domain RAG	Facts change; citations required	Retrieval failures → confident nonsense
LoRA / QLoRA	Style, format, jargon	New world knowledge (memorizes poorly)
Full continued pretrain	Huge proprietary corpus + budget	Most product teams never reach this

Cross-link: fine-tuning LoRA/QLoRA, RAG building blocks, open-weight vs APIs.

Capability cliffs and eval humility

Loss curves are smooth; product metrics are not. Models can suddenly pass coding/math suites after a scale jump, then still fail your 50 golden tickets. Always:

1. Hold a fixed eval set while you change model size or provider
2. Measure task success, not only perplexity
3. Expect diminishing returns once retrieval + tools dominate the failure modes

Tools and literacy (2025–2026)

You rarely fit a scaling law yourself. You use the vocabulary when reading:

• Lab blogs / model cards (tokens trained, architecture dense vs MoE)
• Cloud TCO calculators (train vs serve)
• Open-weight cards on Hugging Face (size, license, intended use)

Pair with Mixture of Experts (sparse parameter growth) and Meta-scale literacy in guided industry labs.

Decision checklist

1. Is the gap algorithm understanding (build tiny) or product quality (buy/serve frontier)?
2. Have you split train vs serve spend in the budget doc?
3. Would better data / retrieval / evals move the needle more than +parameters?
4. If choosing open weights: is the model overtrained enough for your latency envelope?
5. If choosing APIs: are you paying for capability you actually use, or for comfort?

Related

Guided Why your tiny LLM ≠ ChatGPT; Meta scale literacy in How real companies use AI; Inference cost routing; this track’s MoE and reasoning articles.