ShipAI

Real-world examples

Fine-tune vs prompt vs RAG: a decision framework

Choose the cheapest lever that hits quality — with clear criteria, anti-patterns, and a path to escalate only when evidence demands it.

11 minPattern inspired by applied LLM platform guidance
  • finetune
  • prompting
  • rag
  • decision

Framed from public engineering talks, blogs, and OSS patterns. Not confidential internals or invented quotes.

Stop arguing in the abstract

Teams burn quarters fine-tuning when a prompt + tool would do — or ship RAG when the model needs behavior change, not knowledge. Use a framework grounded in public applied-LLM practice.

Decision tree

flowchart TD
  Need[What must improve?] --> Know{Missing knowledge?}
  Know -->|yes, docs change often| RAG[RAG / tools / search]
  Know -->|yes, stable niche style| FT[Consider SFT / adapters]
  Know -->|no| Beh{Behavior / format / tone?}
  Beh -->|yes| Prompt[Prompt + few-shot + schema]
  Beh -->|still failing| FT
  RAG --> Eval[Eval]
  Prompt --> Eval
  FT --> Eval
  Eval -->|fail| Escalate[Next lever]

Compare the levers

Lever Changes Needs Ops cost
Prompt / schema Instructions, format Eval harness Low
RAG / tools Access to facts & actions Retrieval quality Medium
Fine-tune (SFT/LoRA) Weights / style / domain talk Curated data + training High
Preference (DPO/RLHF-class) Ranked behavior Preference data Higher

When RAG wins

  • Knowledge updates weekly+
  • Citations required
  • Corpus larger than context
  • Multiple tenants with different corpora

Anti-pattern: RAG for “always answer in JSON with fields X,Y” — that’s prompting/constrained decoding.

When prompting wins

  • Output contracts (JSON Schema)
  • Tool selection policies
  • Tone and safety framing
  • Rapid iteration

Anti-pattern: 10k-token prompts full of manuals — move manuals to retrieval.

When fine-tuning wins

  • Stable domain jargon / format the base model mangled
  • Latency/cost: specialize a small model
  • Consistent style at scale where prompts are unwieldy

Anti-pattern: fine-tune to “know” yesterday’s pricing PDF — use RAG.

Escalation with evidence

  1. Define task metrics and a golden set
  2. Prompt hard for a week; log failures
  3. Add retrieval/tools for knowledge/action gaps
  4. Only then curate SFT data from cleaned traces
  5. Re-eval; watch for regression on general skills
flowchart LR
  P[Prompt baseline] --> M{Metric met?}
  M -->|yes| Ship[Ship]
  M -->|no| R[RAG/tools]
  R --> M2{Metric met?}
  M2 -->|yes| Ship
  M2 -->|no| F[Fine-tune SLM]
  F --> Ship

Failure modes

  • Fine-tune on unfiltered prod logs (PII, bad answers)
  • RAG without chunk ACLs
  • Prompt sprawl with no versioning
  • Changing three levers at once — causality dies

What to ship

A written ADR per feature: chosen lever, metrics, revisit date. Attach the eval numbers. That’s how staff engineers decide — not vibes.