Advanced Concepts

Fine-tuning with LoRA and QLoRA

Parameter-efficient fine-tuning — adapters, rank, QLoRA memory math, and when PEFT beats prompts or full FT.

55 min

Also in guided path: LoRA / QLoRA fine-tune

The engineering problem

You have a strong base instruct model and a product that needs stable style, schema, or domain phrasing. Prompting works until it doesn’t — drift across versions, long system prompts, or users who ignore format. Full fine-tuning of a 7B–70B model updates every weight: slow, storage-heavy, and easy to destroy general instruction following.

Parameter-efficient fine-tuning (PEFT) — especially LoRA and QLoRA — freezes the base and learns small adapters so you can specialize without cloning the full checkpoint for every experiment.

Full fine-tune vs LoRA (mental model)

Forward pass with LoRA:

[ h = W x + \frac{\alpha}{r} B A x ]

where (W) is frozen, (A \in \mathbb{R}^{r \times d}), (B \in \mathbb{R}^{d_{out} \times r}), rank (r \ll d).

flowchart TD
  X[Input x] --> W[Frozen base W]
  X --> A[Adapter A]
  A --> B[Adapter B]
  B --> Delta["ΔW ≈ BA"]
  W --> Sum[Sum / scale]
  Delta --> Sum
  Sum --> H[Hidden h]
Approach Trainable params Checkpoint size Risk
Full FT All weights Full model copy Forgetting, cost
LoRA Adapters only MBs–GBs of adapters Underfit if (r) too small
QLoRA Adapters + 4-bit base Fits larger models on 1 GPU Quant noise on hard reasoning

Knobs that actually matter

Knob Effect Practical starting point
Rank (r) Capacity vs overfit 8–64; raise if underfit
(\alpha) / scaling Adapter strength Often (\alpha \approx 2r) or library default
Target modules Where knowledge/style lives q_proj,v_proj first; add MLP if needed
LR / epochs Easy to nuke alignment Low LR, early stop on eval
Data mix Catastrophic forgetting Mix general instruct + domain
Max length Truncation silently hurts Match real prompt packing

Ship rule: 1k clean, diverse, on-distribution examples beat 50k scraped noise. Label quality is the PEFT bottleneck.

QLoRA memory math (intuition)

QLoRA keeps the base weights quantized (commonly 4-bit) while training LoRA in higher precision (e.g. BF16/FP16 adapters + optimizer states for adapters only).

flowchart LR
  Base4[(4-bit frozen base)] --> Fwd[Forward]
  LoRA[BF16 LoRA adapters] --> Fwd
  Fwd --> Loss[Loss]
  Loss --> Opt[Optimizer on adapters only]

Why teams care:

  • Fit 7B–70B-class experiments on one consumer/pro GPU
  • Run more A/B adapter experiments per week
  • Still merge or serve adapters with a dequantized or quantized runtime later

Measure task quality, not only VRAM. Aggressive quant can blunt multi-step reasoning even when classification/format tasks look fine. Pair with quantization for inference — train-time QLoRA ≠ serve-time AWQ/GPTQ, but the precision story rhymes.

How to run it in practice (tooling)

Typical stack (2025–2026):

  1. Base: instruct checkpoint from Hugging Face / provider open weights
  2. Train: Hugging Face PEFT + TRL / Axolotl / Unsloth / Lightning
  3. Track: W&B or MLflow for loss, evals, adapter artifacts
  4. Serve: merge adapters into base or load LoRA dynamically (vLLM/Triton paths vary)
# Pseudocode shape — PEFT LoRA on a causal LM
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()

Cross-links: Hugging Face, Weights & Biases, MLflow LLMOps, guided Build & serve your SLM LoRA + serve SLM.

Decision tree: prompt vs RAG vs PEFT vs full FT

flowchart TD
  Start[Need better behavior?] --> P{Prompt + tools enough?}
  P -->|Yes| Stop[Ship prompts / skills]
  P -->|No| Know{Need new facts?}
  Know -->|Yes| RAG[Prefer RAG / tools]
  Know -->|No| Style{Need stable style/format/domain?}
  Style -->|Yes| LoRA[LoRA / QLoRA]
  Style -->|No| Eval[Fix evals first]
  LoRA --> Full{Clear data + big budget + must update all weights?}
  Full -->|Yes| FFT[Full / continued train]
  Full -->|No| Keep[Keep PEFT]
  1. Prompt + tools sufficient? Stop.
  2. Need facts that change? Prefer RAG / DB tools; FT alone memorizes poorly.
  3. Need style/format/domain phrases stably? PEFT.
  4. Full FT only with clear data flywheel + budget + regression evals.

Failure modes

Failure Symptom Mitigation
Overfit Great on train JSON, brittle in prod Holdout + paraphrased evals
Forgetting Soft refusals / chat quality die Mix general SFT data
Rank too low Never learns schema Raise (r) or target more modules
Data leakage Eval contaminated Strict split by user/doc id
Adapter hell 40 LoRAs, no owner One adapter per product capability + registry

Production checklist

  1. Freeze a prompt-only baseline before training
  2. Define offline eval (exact schema, rubric, or LLM-as-judge with care)
  3. Version base SHA + adapter + data snapshot together
  4. Canary serve; watch refusal rate and latency
  5. Document merge vs multi-adapter serving choice

Build & serve your SLM LoRA + serve SLM. Pair with alignment RLHF/DPO (preference ≠ LoRA style FT) and open-weight vs APIs.

Project checklist0/3 done