Core Concepts

LLM project lifecycle

Scope → model choice → adapt (prompt/RAG/fine-tune) → evaluate → deploy → monitor — the GenAI-with-LLMs lifecycle for product teams.

40 min

The lifecycle (product view)

DeepLearning.AI’s Generative AI with LLMs organizes work as a lifecycle, not a single notebook. ShipAI uses the same spine:

flowchart LR
  Scope[Scope & success metrics] --> Choose[Choose model / modality]
  Choose --> Adapt[Adapt: prompt / RAG / FT]
  Adapt --> Eval[Offline + online eval]
  Eval --> Deploy[Deploy + gateway]
  Deploy --> Monitor[Monitor / feedback]
  Monitor --> Adapt

Skipping stages is how demos become outages: teams jump to a fancy agent before they can state a metric, or deploy without offline gates.

Scope before models

Write:

  • User job-to-be-done
  • Allowed failure modes
  • Latency and cost budgets
  • Data sensitivity class
  • Who is accountable when the model is wrong

If you cannot state a metric (exact match, grounded citation rate, task success), you are not ready to pick a model.

Scope artifact Example
Success metric ≥90% citation-valid on gold; p95 TTFT < 2s
Non-goals No autonomous refunds
Data class PII in tickets; redact before vendor
Fallback Escalate to human queue

Choose model / modality

Decisions at this stage:

Ship rule: pick the weakest model that passes the eval bar — not the strongest model in the press release.

Adapt: three levers

Lever Use when Cost to change
Prompt / tools Behavior is linguistic; tools supply facts Low
RAG Answers need your corpus Medium
Fine-tune Style/format/domain language stable Higher

Order of operations for most products: prompt → RAG/tools → fine-tune. See examples on fine-tune vs prompt vs RAG and Advanced Fine-tuning LoRA/QLoRA.

flowchart TD
  Metric[Metric failing] --> Q1{Missing private facts?}
  Q1 -->|Yes| RAG[Improve retrieval / citations]
  Q1 -->|No| Q2{Format / tool errors?}
  Q2 -->|Yes| Struct[Schemas + prompts]
  Q2 -->|No| Q3{Stable domain style?}
  Q3 -->|Yes| FT[Consider LoRA]
  Q3 -->|No| Data[Fix data / UX / scope]

Evaluate like a release gate

Offline golden sets → shadow traffic → canary. Never “demo vibe.”

Stage Purpose
Offline gold Catch regressions in CI
Shadow Compare new vs old on live traffic without user impact
Canary Small % of users; watch metrics + cost
Rollback plan One-click prompt/model revert

Details: Evals fundamentals.

Deploy and monitor

Failure modes

Failure Stage skipped Fix
Cool demo, no metric Scope Write exit criteria
Prompt spiral Adapt Time-box; change lever
Prod surprise Eval Offline gate + canary
Cost blow-up Deploy/monitor Budgets + routing
Silent quality drift Monitor Weekly gold refresh

Tradeoffs

  • Move fast on prompts — great early; hits a ceiling.
  • Invest in RAG early — if facts live in docs.
  • Fine-tune late — after prompts/RAG plateau and data is clean.

Glossary

Term Meaning
Adaptation Changing behavior via prompt, data, or weights
Canary Partial production rollout
Shadow Score new system on live inputs without serving it
Exit criteria Numeric bar to leave a lifecycle stage

Micro-project

Write a one-pager lifecycle for your capstone idea with exit criteria per stage (scope → choose → adapt → eval → deploy → monitor).

Onboarding (rubric), Evals, guardrails, safety, Deploy, cost, latency, observability.

Project checklist0/3 done