ShipAI

Real-world examples

Train a tiny GPT from scratch (real checkpoint, real loss drop)

Karpathy-style mini-GPT on Tiny Shakespeare — inspect, train, reload, sample. Same algorithm as frontier models at laptop scale, with measured numbers from a first run.

12 minPattern inspired by Karpathy nanoGPT
  • llm
  • training
  • transformers
  • nanoGPT

Framed from public engineering talks, blogs, and OSS patterns. Not confidential internals or invented quotes.

The product problem

You cannot operate an inference control plane (or ship an agent) if “the model” is a black box. You need to have trained weights, reloaded them, and generated text — once — so tokens, loss, and checkpoints are real objects.

This write-up is the lab log for that first loop. Implementation: sibling repo owned-llms. Course home: Build an LLM from scratch. Browse: Mini-GPT from scratch.

Not ChatGPT. Same software shape.

flowchart LR
  Text[Shakespeare text] --> Tok[Char tokens]
  Tok --> GPT[Tiny GPT]
  GPT --> Loss[Cross-entropy]
  Loss --> Ckpt[ckpt.pt]
  Ckpt --> Gen[Generate]

What we trained

Character-level GPT (a token is a letter, not BPE):

Knob Value
Corpus Tiny Shakespeare, 1,115,394 chars
Vocab 65
Architecture 4 layers, 4 heads, n_embd=128, block_size=64
Params 809,856
Optim AdamW, lr 3e-3, batch 64, 2000 steps
Device Apple MPS (~2 min)

Loss: untrained ~ln(65)≈4.17. This run: 4.20 → train 1.49 / val 1.70. If loss stays at 4.17, the model did not learn.

Samples look like broken Shakespeare (ROMEO: then fake scene headers). That is expected. Failure is a stub trainer or generate that ignores the checkpoint.

Commands

cd owned-llms
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python -m nanogpt.inspect     # shapes + untrained loss
python -m nanogpt.train       # writes out/ckpt.pt
python -m nanogpt.generate --prompt "ROMEO:" --tokens 400

Run generate twice after killing the process. Reload = your model.

Internals (one screen)

token ids → embed + positions
         → N × (LN → causal attention → residual → LN → MLP → residual)
         → LN → lm_head → logits

Causal mask: position t cannot see the future. Attention mixes across tokens; MLP thinks per token. Generate: last-position logits → temperature → sample → append.

Map: nanogpt/model.py, data.py, train.py, generate.py. Longer vocabulary: CONCEPTS.md in that repo.

Honest limits vs frontier

Tiny GPT Frontier LLM
Char vocab, 64 context BPE, 8k–1M context
~0.8M params Billions
Next-char Shakespeare Instruction + tools
You trained it You usually call an API

Same algorithm. Next in the course: dissect a 1B–3B open model, then LoRA an owned general LLM / domain SLM. Inferix comes after you have real *-vN checkpoints — register, route, drift, promote.

Checklist

  • inspect prints (B, T) and logits (B, T, V)
  • Train loss clearly below ln(V)
  • out/ckpt.pt exists
  • Generate works after a fresh process
  • README/METRICS record hardware and final val loss