When SLM/FT beats RAG or prompting

Ownership has a cost — and a payoff

You can ship most features with prompting plus optional RAG. Fine-tuning a small language model (SLM) — or serving a specialized adapter you own — enters when the economics of repetition, format control, latency, or data ownership flip the decision.

This module teaches the full owned-model path: curate data, LoRA/QLoRA fine-tune, evaluate against baselines, serve locally, quantize for production knobs. Before GPUs and JSONL, you need a decision framework so you do not fine-tune reflexively (expensive) or avoid ownership when it would clearly win (missed margin).

Callout — narrow tasks only: Fine-tuning shines on bounded input/output patterns — classification, extraction, formatting, domain phrasing — not open-ended reasoning chat.

Three approaches compared

None replaces the others permanently — production stacks combine them (RAG + small reranker FT, prompt guardrails + local classifier).

When fine-tuning / SLM wins

Strong signals:

Style and format must be tight. JSON schemas, brand voice, ticket triage labels — few-shots consume context and drift across model upgrades.

Volume and latency. Millions of inferences/month on a 3B model you serve may beat frontier API cost — if ops can run GPUs or CPU-quantized inference.

Stable domain language. Insurance codes, internal acronyms, SQL dialect fixes — patterns repeat; training memorizes distribution better than 10-shot prompting.

Offline / VPC constraints. Regulated environments block external APIs; owned weights inside the perimeter matter.

Small context suffices. Input fits in 2–4k tokens; no need for 128k frontier window.

When to stay on prompts or RAG

Strong counter-signals:

Requirements change weekly. Retrain lag kills you; prompts iterate in hours.

No clean labeled data. Garbage fine-tunes are worse than good prompts.

Open-ended reasoning. Planning, multi-hop research, novel problem solving — frontier models still lead on hard general tasks.

Ops cannot own serving yet. A fine-tune without deploy path is a notebook trophy.

Knowledge-heavy, format-light. Policy Q&A over changing docs — RAG first; maybe add small FT reranker later.

Decision worksheet structure

Complete m6/decision_worksheet.md for a real candidate task:

1. Task definition — input/output examples, success metric (accuracy, F1, latency p95).
2. Data — sources, volume, license, PII, label quality.
3. Prompt-only plan — model choice, estimated cost per 1k, failure modes.
4. RAG plan — corpus coverage, retrieval eval expectation.
5. LoRA plan — base model, train hardware, serve target, retrain cadence.
6. Recommendation — chosen path with kill criteria (e.g. "abort FT if val accuracy < prompt baseline after 500 steps").

Kill criteria prevent sunk-cost fallacy when fine-tune plateaus below baseline.

Preview: rest of this module

Lesson path you are committing to if FT wins:

6.2 Dataset curation + licenses
6.3 LoRA/QLoRA training
6.4 Eval vs base + teacher model
6.5 Serve OpenAI-compatible API
6.6 Quantization tradeoffs → milestone

Milestone bar: fine-tuned + served SLM beats prompt-only baseline on your narrow task with a published report card.

Callout — teacher baseline fairness: Compare against the same task with your best prompt pack on a capable API model — not a deliberately weak baseline.

Total cost of ownership worksheet

Extend your decision worksheet with a rough TCO sketch:

Prompt-only: monthly_calls × price_per_1k_tokens × avg_tokens.

RAG: above + embed ingest one-time + vector DB hosting + periodic reindex labor.

Fine-tune: labeling hours + GPU train time + engineer serve/on-call + retrain cadence when drift detected.

Fine-tuning wins when variable API cost over 12 months exceeds FT investment and task stability exceeds retrain frequency. Spreadsheet honesty prevents "we fine-tuned because it sounds cool."

Combining approaches

Production often stacks techniques:

• RAG for fresh policy docs + small FT classifier for intent routing.
• Prompt guardrails on output + local SLM for PII-sensitive sub-task.
• Teacher distillation: generate training rows from API model, train SLM for cheap inference (watch license).

Your worksheet should note hybrid architecture even when primary recommendation is single-path — interviewers ask about combos.

Example walkthrough: ticket triage

• Task: Map support email → {billing, shipping, account, bug, other}.
• Prompt-only: Works at 88% with GPT-4o-mini; cost $X/1k at 50k tickets/mo.
• RAG: Low value — labels not in docs.
• LoRA on 3B classifier-style FT: Target 92%+, local inference 40ms, one-time data label 800 examples.

Decision: LoRA if labeling budget exists and ops can run Ollama/vLLM. Else prompt-only until volume justifies investment.

Engineering problem (staff framing)

Choose prompt vs RAG vs FT/SLM by stability of knowledge, latency, privacy, style control.

Diagram — Technique choice

flowchart TD
  Need{Knowledge changes weekly?} -->|Yes| RAG
  Need -->|No style/format| FT[LoRA/SLM]
  Need -->|Simple| Prompt

Precise definitions & mental model

Parametric vs non-parametric memory; control vs freshness.

Tradeoffs — when to use what

Failure modes (interview + on-call)

FT for volatile facts; RAG for tone-only problems.

Production & OSS practices

Decision record in design doc; revisit quarterly.

Micro-project: Decision worksheet

Complete m6/decision_worksheet.md for a task you might actually build:

• Ticket triage, jargon rewriter, log line parser, SQL dialect fixer, etc.
• All sections filled with numbers (even estimates).
• Explicit recommendation and kill criteria.

Commit to course-portfolio; no code required this lesson.

Checklist

• Worksheet committed with clear recommendation
• Kill criteria explicit
• Baseline approach described fairly
• Data/license section not empty