Pretrain vs SFT vs preference (map only)

One objective, three training stages

You trained a mini-LLM on next-token prediction. That is pretraining in miniature — one objective, self-supervised labels from the corpus itself. Frontier models go through two more stages before they become helpful assistants: supervised fine-tuning (SFT) and preference optimization (RLHF, DPO, ORPO, and variants).

You will not run RLHF on a 70B model in this course. You will know where each stage sits in the lifecycle, what data each stage needs, and what product behaviors each stage unlocks. That map is how AI engineers talk to research, infra, and leadership without pretending every team trains from scratch.

Callout — base model ≠ chat model: The checkpoint after pretraining completes sentences; it does not reliably follow instructions or refuse harmful requests until post-training shapes behavior.

Stage 1: Pretraining

Goal: Learn general language, world knowledge, and reasoning patterns from raw text.

Data: Massive unstructured corpora — web crawl, books, code, papers (filtered and deduplicated).

Objective: Next-token prediction (same as your mini-LLM).

Compute: Dominates total training cost for frontier models — weeks to months on thousands of GPUs.

Output: A base model — strong at continuation, weak at instruction following. Prompt "Write a poem about security" and you may get more prompt text, not a poem.

Your mini-LLM lives entirely in this stage. That is honest scope: you implemented the core algorithm; scale and data are what differ.

Stage 2: Supervised fine-tuning (SFT)

Goal: Teach the model to follow instructions in a chat format.

Data: Curated (prompt, ideal response) pairs — human-written or distilled from stronger models. Smaller than pretrain corpus but higher quality per token.

Objective: Still next-token prediction, but only on the assistant's tokens in a structured template:

<|user|>Explain gradient descent briefly<|assistant|>Gradient descent...

Loss is masked on user tokens; model learns to continue as the assistant.

Compute: Much smaller than pretrain — often hours to days on a fraction of GPUs.

Output: An instruct model — follows formats, answers questions, still may be verbose, unsafe, or hallucinate confidently.

Product note: many startups never pretrain; they SFT or adapter-tune open weights (LLaMA, Mistral). SFT is the first stage most AI engineers touch in production.

Stage 3: Preference optimization (RLHF and friends)

Goal: Align outputs with human preferences — helpful, harmless, on-brand.

Data: Preference pairs (prompt, chosen response, rejected response) or ranked lists; sometimes AI-labeled at scale.

Methods:

Compute: Smaller than pretrain; can still be substantial for large models. Reward modeling + RL loops need careful infra.

Output: Chat/product model — what users see in ChatGPT, Claude, etc., modulo system prompts and tool layers.

You will not implement PPO in this module. Know that when users say "RLHF," they often mean the whole preference stage, not literally PPO only.

What happens after alignment

Production stacks add layers your diagram should include:

• System prompts — policy without weight updates.
• Tool use / function calling — fine-tuned or prompted; retrieves live data.
• RAG — retrieval augments context; not a training stage but paired with every chat model.
• Guardrails — classifiers, filters, moderation APIs on input and output.
• Continual updates — periodic SFT on new policies; rarely full re-pretrain.

A complete lifecycle diagram for product planning shows pretrain → SFT → preference → deployment wrapper (tools, RAG, moderation).

In scope vs out of scope for this course

This boundary is a feature. Interview credibility comes from knowing what you implemented and what you would need to reach production parity.

How stages show up in API products

When you call gpt-4 vs gpt-4-base (if exposed) vs fine-tuned variants, you are selecting different points on this pipeline. Open-weight ecosystems mirror the pattern:

• Base: Llama-3-70b
• Instruct: Llama-3-70b-Instruct
• Preference-tuned: vendor chat endpoints, DPO checkpoints on Hugging Face

Fine-tuning APIs (OpenAI, Together, etc.) usually operate after pretrain — SFT/LoRA on your data, not training from random init.

Callout — post-training does not fix bad base models: Preference tuning adjusts behavior at the margin. A tiny corpus pretrain cannot be "RLHF'd" into GPT-4 — the knowledge and capacity are not there.

Engineering problem (staff framing)

Product behavior comes from a pipeline: pretrain → SFT → preference. Mis-attributing stage wastes compute.

Diagram — Post-training lifecycle

flowchart LR
  PT[Pretrain CE] --> SFT[SFT instruct]
  SFT --> Pref[DPO/RLHF]
  Pref --> Dep[Deploy: prompts+RAG+tools+guards]

Precise definitions & mental model

Base vs instruct vs chat; SFT mask on assistant tokens; preference pairs; DPO vs PPO-RLHF.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Expecting SFT to add knowledge pretrain never saw; RLHF as magic safety; no evals between stages.

Production & OSS practices

Most startups start from instruct weights + LoRA/SFT; keep lifecycle diagram in design docs.

Deep dive (FAANG / OSS bar)

What each stage can and cannot fix

Product planning

If your customer wants a support bot on policy docs that change weekly, prefer RAG + SFT for style, not pretrain. If you need a specialized code dialect permanently, LoRA/SFT on open weights may beat prompt-only.

Micro-project: Lifecycle diagram

In m3/lifecycle/:

1. Draw a diagram (Mermaid, Excalidraw, or Figma) showing: Pretrain → SFT → Preference → Deploy with data type, objective, and approximate relative compute bar under each stage.
2. Add side boxes for: system prompt, RAG, tools, moderation.
3. Annotate where your mini-LLM sits and what would be required to move one stage right (data + compute order-of-magnitude estimates — rough is fine).
4. Write LIFECYCLE.md (1 page): which stage would you fine-tune for a customer-support bot on open weights? Which stage would a frontier lab spend 80% of GPU budget on?

Commit the diagram source and exported PNG/SVG.

Example Mermaid starter:

flowchart LR
  PT[Pretrain\nnext-token] --> SFT[SFT\ninstruction pairs]
  SFT --> PO[Preference\nDPO/RLHF]
  PO --> DEP[Deploy\nprompts + RAG + tools]

Checklist

• Diagram includes all three training stages plus deploy wrappers
• Your mini-LLM position marked explicitly
• LIFECYCLE.md answers the customer-support bot question
• Rough compute/data notes show pretrain dominates at frontier scale