Core Concepts

Mini-GPT from scratch

Next-token prediction at laptop scale — tokens, causal attention, train until loss drops, reload your checkpoint. The same loop frontier models use.

35 min

Also in guided path: Tiny GPT / mini-LLM end-to-end

Why this exists

Guided module Build an LLM from scratch walks the theory. This article is the read-up + lab pointer: a real mini-GPT you can train in minutes, then reload.

Repo: owned-llms (sibling of ShipAI). Lab log with measured numbers: Train a tiny GPT from scratch.

You are not reproducing ChatGPT. You are owning the loop:

flowchart TD
  A[Text] --> B[Token ids]
  B --> C[Embed + positions]
  C --> D[Causal transformer blocks]
  D --> E[Logits]
  E --> F[Loss / sample]
  F --> G[ckpt.pt]

Next-token prediction

Given previous characters, guess the next one. Training = higher probability on the true next char. Generation = sample, append, repeat.

A token here is a character ('H' → an id). Real LLMs use BPE (subwords). Same idea, larger alphabet. See Tokenization.

Train vs val: hold out ~10% so you can see memorization. Overfit = train loss much lower than val.

Loss

Cross-entropy is -log P(correct next token). Random model ≈ ln(vocab_size). Vocab 65 → ~4.17.

First measured run (Apple MPS, 2000 steps, ~0.8M params): 4.20 → 1.49 train / 1.70 val. That drop is the proof of learning. Pretty prose is not.

Architecture (decoder-only)

x = token_embed[idx] + position_embed[0..T]
for each block:
  x = x + Attention(LayerNorm(x))   # causal: no looking forward
  x = x + MLP(LayerNorm(x))         # per-token “thinking”
logits = lm_head(LayerNorm(x))

Attention: query / key / value. Score pairs, softmax, weighted sum of values. Heads = several of these in parallel. Details: Self-attention.

Generate: last-position logits / temperature → sample → append. The checkpoint is the weights. Kill the process, load out/ckpt.pt, sample again — that file is your model.

Run

cd owned-llms
source .venv/bin/activate   # after python3 -m venv .venv && pip install -r requirements.txt
python -m nanogpt.inspect
python -m nanogpt.train
python -m nanogpt.generate --prompt "ROMEO:" --tokens 400

Read CONCEPTS.md then nanogpt/model.py top to bottom. Guided depth (BPE, sampling, vs ChatGPT): lessons 3.1–3.8.

After this

  1. Dissect a small open instruct model (tokenizer, layer names, KV cache).
  2. LoRA a general owned LLM, then domain SLMs.
  3. Agents (tools + loop) — they sit on a model, they are not the first step.
  4. Inferix later: register the same *-vN as clients (route, traces, drift, promote).

Checklist

  • Explain next-token prediction without notes
  • Untrained loss near ln(V)
  • Trained val loss clearly lower
  • Generate after reload
  • One sentence: toy GPT vs Llama/Claude (algorithm vs scale)
Project checklist0/3 done