Quantization tradeoffs

Bits are a product knob

Full-precision weights are research defaults; production picks quantization — representing weights in 8-bit, 4-bit, or mixed formats — to shrink memory, increase throughput, and run on cheaper hardware. The cost is quality drift: labels wrong more often, JSON schemas crack, or tone slips.

This lesson measures latency vs quality across quantization settings on your served SLM, publishes a comparison table, and closes Milestone 6: fine-tuned, served, beating prompt baseline, with explicit deployment recommendation.

Callout — measure YOUR adapter: Generic blog posts about Q4 vs Q8 do not substitute for re-running your report card metrics on each setting.

Quantization formats (practical)

QLoRA training already uses 4-bit base; serve quantization is a separate choice when exporting to Ollama GGUF or enabling vLLM AWQ.

Experiment design

Hold constant:

• Same test set from lesson 6.4
• Same prompt/template
• Same hardware class (document chip: Apple M2, RTX 4090, etc.)

Vary:

• Quantization level (e.g. F16, Q8, Q4_K_M, Q4_0)
• Optional: batch size 1 vs 8 for throughput

Metrics per setting:

• Quality: accuracy / macro-F1 (same script as report card)
• Latency: p50/p95 ms per request, batch=1
• Throughput: tokens/sec if batching relevant
• RAM/VRAM: peak usage
• Model size on disk

Run ≥3 timed requests warmup, then ≥50 timed requests for stable p95.

Reading results

Common patterns:

• Q8 ≈ baseline quality with ~2× memory savings — good default when unsure.
• Q4 often ≤1–2 point accuracy drop on classification; extraction tasks may break first.
• Aggressive Q2 saves RAM but kills rare classes — check confusion matrix, not just accuracy.

If Q4 matches Q8 on your narrow task, ship Q4 for laptop demos; keep Q8 for production if SLA allows RAM.

Publishing the table

m6/eval/quantization_table.md:

Recommendation paragraph: which setting to deploy and when to re-evaluate (after adapter v2, after base model upgrade).

Milestone 6 closure checklist

Your m6/README.md should prove:

1. Decision worksheet with chosen FT path
2. Licensed dataset + splits
3. LoRA adapter artifacts + train log
4. Report card: adapter beats prompt baseline on test
5. OpenAI-compatible serve + smoke client
6. Quantization table with recommendation

A reviewer clones, follows commands, reproduces within documented tolerance.

Callout — milestone is end-to-end: Missing serve or missing eval breaks the story — "I trained a model" without deployment path is incomplete engineering.

Apple Silicon and CPU inference

Mac users often serve via Ollama with Metal acceleration or llama.cpp GGUF. Quantization table still applies — Q4_K_M on Apple M2 may beat BF16 cloud GPU on cost for personal demos. Document chip and OLLAMA_NUM_PARALLEL settings; thermal throttling affects sustained p95 on laptops.

CPU-only servers favor smaller bases (1B–3B) and aggressive Q4; expect higher latency — verify against product SLA before claiming local deploy wins.

When to re-quantize

Re-run quantization table when:

• Adapter v2 trained with different rank or data distribution
• Base model version bumped (Llama 3.2 → 3.3)
• Task expands from classification to longer generation outputs

Treat quantization as part of release checklist, not one-time homework.

Quality beyond accuracy

Quantization may preserve classification accuracy while harming:

• Calibration of confidence
• JSON formatting reliability in generation tasks
• Tail latency variance (CPU dequant overhead)

If your task outputs structured JSON from SLM, include schema valid rate row in quantization table — accuracy can look flat while parse failures spike at Q4.

Deployment recommendation template

End quantization_table.md with:

## Recommendation
- **Dev laptop:** Q4_K_M — acceptable 2pt F1 drop, 40% faster p95
- **Production candidate:** Q8 — parity with BF16 on test set
- **Re-eval trigger:** adapter v2 or task expands to generation

Milestone reviewer reads this paragraph first — make it decisive.

When not to quantize yet

• Accuracy already barely beats baseline — quantization noise may erase win.
• Debugging template bugs — use BF16 until outputs stable.
• Regulatory requirement for exact reproducibility — document quantization seed/version.

Engineering problem (staff framing)

Quantization cuts memory/latency for quality. Measure task metrics, not only perplexity.

Diagram — Precision ladder

flowchart LR
  BF16 --> INT8 --> INT4 --> Edge[On-device]

Precise definitions & mental model

PTQ vs QAT; AWQ/GPTQ/GGUF families; calibration sets.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Quant without task eval; mixing kernels poorly.

Production & OSS practices

Per-task accept thresholds; keep BF16 teacher for diffs.

Micro-project: Latency/quality table

In m6/eval/ (extend serve setup):

1. Export or configure ≥3 quantization variants of your served model.
2. Re-run metrics + latency benchmark; fill quantization_table.md.
3. Update milestone README with final ship recommendation.
4. Note one quality regression example if Q4 fails specific rows.

Checklist

• Same test set across all quantization runs
• Latency measured with warmup
• Table committed with recommendation
• Milestone 6 README complete