LoRA / QLoRA fine-tune

The owned-SLM skill: parameter-efficient fine-tuning

Full fine-tuning updates every weight in a multi-billion-parameter model — expensive and storage-heavy. LoRA (Low-Rank Adaptation) injects small trainable matrices into attention layers while freezing the base. QLoRA quantizes the frozen base to 4-bit and trains LoRA adapters in low precision, enabling fine-tunes on consumer GPUs (12–24 GB).

This lesson is hands-on: pick an open base (Llama 3.2 1B/3B, Mistral 7B, Phi-3 mini, Qwen2.5), train on your JSONL, save adapter weights to the portfolio.

Callout — base model license matters: Read the base model license (Meta Llama, Apache, etc.) before shipping derivatives.

What LoRA does

Instead of updating full weight matrix W, LoRA learns ΔW = BA where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k} with rank r ≪ d. Inference merges adapters into W or loads them sidecar. Typical targets: q_proj, v_proj, sometimes MLP layers.

Hyperparameters to log:

Stack choices

Hugging Face TRL + PEFT — standard path:

pip install transformers peft trl bitsandbytes accelerate datasets

Axolotl / LLaMA-Factory — YAML-driven configs for reproducibility.

Unsloth — optimized QLoRA training speeds on supported GPUs.

Course requirement: script or config you can re-run; black-box Colab-only without saved config does not count.

Minimal training sketch (conceptual):

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model = AutoModelForCausalLM.from_pretrained(
    base_id, load_in_4bit=True, device_map="auto"
)
peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, peft_config)
# SFTTrainer with train dataset, formatting_func applying chat template
trainer.train()
model.save_pretrained("m6/adapters/ticket_classifier_v1")

Apply the model's chat template in formatting_func — mismatched templates destroy fine-tune quality.

QLoRA requirements

• NVIDIA GPU with compute capability supporting bfloat16/fp16 preferred.
• bitsandbytes 4-bit NF4 weights + double quant.
• If no GPU: use cloud spot instance; log hardware in TRAIN_LOG.md.

CPU-only full fine-tune of 7B is not realistic; use 1B–3B or cloud.

Monitoring training

Log to m6/adapters/ticket_classifier_v1/TRAIN_LOG.md:

• Base model id and revision
• Dataset hashes (train/val file sha256)
• LoRA config JSON
• GPU model, VRAM, wall time
• Final train/val loss curves (screenshot or wandb link)

Stop early if val loss rises — classic overfit signal.

Artifacts to commit

Commit:

• Adapter weights (small, MB scale) OR LFS if larger
• Training config YAML / script
• TRAIN_LOG.md

Do not commit:

• Full base model weights
• Raw API keys
• Unredacted training data if policy forbids

Hardware planning

Rough VRAM guidance for QLoRA (varies by implementation):

Sequence length dominates memory — truncate training rows to p95 token length from stats.json rather than maxing 4096 everywhere.

Cloud spot instances (A10, L4) reduce cost; snapshot adapter to object storage before spot preemption.

Reproducibility checklist

Save in TRAIN_LOG.md:

• transformers, peft, torch, bitsandbytes versions
• Random seed and training_args JSON dump
• Git commit SHA of training script
• Exact CLI command or accelerate launch args

Reproducibility is approximate on GPU (non-deterministic ops) — val metrics should be within small noise band on rerun, not bit-identical.

Eval hooks during training

Optional eval_steps during SFTTrainer:

• Generate on 10 held-out prompts every N steps
• Log sample outputs to WandB or text file
• Early stop if outputs collapse to empty string or repeat token

Qualitative collapse often appears before val loss spikes — especially on small datasets.

Merging adapter for deployment

Some serve stacks prefer merged weights over runtime adapter load:

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(base, adapter_path)
merged = model.merge_and_unload()
merged.save_pretrained("m6/merged/ticket_classifier_v1")

Merged simplifies Ollama export; adapters simplify A/B between v1 and v2 on one base — pick per ops maturity.

Common failures

Callout — mask user tokens in loss: Train only on assistant tokens for instruction tuning — otherwise the model learns to predict user emails.

Engineering problem (staff framing)

LoRA adapts few params; QLoRA fits larger models on smaller GPUs. Still overfit and eval.

Diagram — LoRA adapter

flowchart LR
  X --> W[Frozen W]
  X --> A[A]
  A --> B[B]
  W --> Y
  B --> Y

Precise definitions & mental model

Rank r, alpha, target modules, QLoRA quantization during train.

Tradeoffs — when to use what

Full FT quality vs LoRA cost; rank too high ≈ full FT memory.

Failure modes (interview + on-call)

Train chat template mismatch; catastrophic forgetting; no smoke eval.

Production & OSS practices

Export merged vs adapter; version base+adapter digests.

Micro-project: Train adapter on open base

In m6/train/:

1. Fine-tune LoRA/QLoRA on your dataset from lesson 6.2.
2. Save adapter to m6/adapters/<task>_v1/.
3. Complete TRAIN_LOG.md with hyperparameters and hardware.
4. Smoke-generate 5 val inputs from checkpoint — paste outputs in log.

Eval properly in lesson 6.4.

Checklist

• Chat template matches base model
• Adapter loads without base re-download in README steps
• Train/val loss recorded
• Secrets and full weights not committed