Alignment basics — RLHF and DPO

Why pretrain alone is not ChatGPT

Pretraining maximizes likelihood of web (and code) text. That objective produces a strong next-token predictor, not a reliable assistant. It will:

• Continue toxic or misleading text when that is likely under the pretrain distribution
• Ramble instead of following instructions
• Lack a consistent notion of “helpful vs harmful” for your product policy

Alignment (in the industry sense) is the stack of stages that push the model toward human (or constitution) preferences: typically SFT → preference optimization (RLHF, DPO, and relatives). You rarely run PPO from scratch in app work — but you must know why base ≠ instruct ≠ “safe for my tools.”

The pipeline (systems view)

flowchart LR
  PT[Pretrain<br/>next-token] --> SFT[Supervised FT<br/>instruction demos]
  SFT --> Pref[Preference data<br/>A ≻ B]
  Pref --> RM[Reward model]
  RM --> RL[RLHF / PPO<br/>+ KL to SFT]
  Pref --> DPO[DPO / IPO / …<br/>direct prefer]
  RL --> Align[Aligned policy]
  DPO --> Align

Stage	Data	What changes
Pretrain	Trillions of tokens	World knowledge + fluency
SFT	(prompt, ideal response)	Follow instructions, format
Preference	(prompt, chosen, rejected)	Rank behaviors
RLHF	RM scores + on-policy samples	Optimize for reward under KL
DPO-style	Same pairs, no explicit RM	Classification-style preference loss

SFT: the underrated foundation

Before fancy preference math, supervised fine-tuning teaches:

• Chat templates and role structure
• Tool-call formatting
• “When unsure, ask / refuse / cite” behaviors you demo

Garbage SFT → preference training polishes garbage. For product teams collecting data: write gold responses, not just thumbs-up on random samples.

RLHF intuition (PPO-era mental model)

1. Collect comparisons: for prompt (x), humans (or stronger models) say (y_w \succ y_l)
2. Train a reward model (r_\theta(x, y)) to score completions
3. Optimize the policy with RL (often PPO) to raise reward while staying close to the SFT policy via a KL penalty

sequenceDiagram
  participant H as Preferrer
  participant RM as Reward model
  participant P as Policy LLM
  H->>RM: Train on A≻B pairs
  loop RL steps
    P->>P: Sample y ~ π
    P->>RM: Score r(x,y)
    RM-->>P: Reward + KL vs SFT ref
    P->>P: Policy update
  end

Why KL matters: unconstrained reward hacking produces gibberish that “looks” high-reward to a brittle RM.

DPO and relatives

Direct Preference Optimization (DPO) and cousins skip an explicit reward model by turning preference pairs into a closed-form objective on the policy (classification-style). Ops wins:

• Fewer moving parts (no RM training loop, less RL infra)
• Often easier to reproduce on mid-size open stacks

Tradeoffs:

• Still lives or dies on preference data quality
• On-policy RL methods can explore behaviors not in the pair set
• Hyperparameters and reference-model choice still matter

Builder takeaway: “We used DPO” is not a safety story. It is an optimization recipe.

Preference data is the product

Good preference pairs	Bad preference pairs
Clear win on your rubric	Ties / subjective noise
Cover refusal + helpfulness	Only “longer = better”
Include tool honesty	Reward confident hallucination
Multi-turn where product is multi-turn	Single-turn only

Sketch exercise: for a support bot, list 20 pairs that encode tone, escalation, PII refusal, and citation rules. That list is more valuable than naming the loss.

What product engineers actually do

Most app teams:

1. Buy already-aligned API models (OpenAI, Anthropic, Google, …)
2. Or download instruct/chat open weights and add LoRA for domain (not full RLHF)
3. Enforce policy in the product with guardrails, allowlists, and evals

You still need alignment literacy to:

• Choose base vs instruct checkpoints
• Debug “model got worse after FT” (you overwrote alignment)
• Design preference/feedback loops for future custom models
• Talk to vendors about refusal behavior and jailbreak posture

Cross-links: fine-tuning LoRA/QLoRA, guardrails and safety, evals fundamentals, guided Pretrain vs SFT vs preference (map only) lifecycle + Evals, guardrails, safety safety productization.

Failure modes

Failure	What you see	Mitigation
Reward hacking	Fluent nonsense / sycophancy	Better prefs + KL + diverse evals
Alignment tax	Weaker coding/math after prefs	Mix capability-preserving data
Over-refusal	Blocks legitimate asks	Policy-tuned pairs + false-positive review
Distribution shift	Prefs from chat, prod is tools	Collect tool-trace preferences
Fake safety	Marketing “aligned” only	Red-team + runtime guardrails

Tools (ecosystem literacy)

• TRL / Axolotl / Hugging Face — SFT and DPO-style trainers for open weights
• Vendor fine-tune APIs — often SFT-only; read whether preference stages are exposed
• Eval harnesses — refusal, toxicity, task success (not one vibe check)

Checklist

1. Can you draw pretrain → SFT → preference → ship for your bot?
2. Do you know if your base is base, instruct, or RL-aligned?
3. If fine-tuning, are you measuring alignment regressions (refusals, jailbreaks)?
4. Is product policy enforced in code (tools/authz), not only in prose system prompts?

Related

Pretrain vs SFT vs preference (map only) lifecycle map; Evals, guardrails, safety safety productization; this track’s LoRA and guardrails articles.