Prompt versioning

If you cannot name the version, you cannot debug it

Prompt changes are code changes. They alter behavior, break edge cases, and shift cost/latency profiles — often silently. Yet many teams still edit a string in a dashboard with no git history, no reviewer, and no link from a bad production response back to the prompt that generated it.

Prompt versioning treats prompts as release artifacts: named, immutable, attributable, and comparable. When a user reports "yesterday it classified refunds correctly; today it says other," you need to answer:

• Which prompt version was live at each time?
• Which model and temperature?
• What exact input was sent?

This lesson builds a minimal registry + A/B runner that scales from solo projects to team workflows.

Callout — immutability: Never edit v1.2.0/system.md in place. Cut v1.3.0/ or bump a semver folder. In-place edits destroy experimental reproducibility.

Version identifiers that work

Pick a scheme and stick with it:

Semver folders — prompts/ticket_classifier/v1.0.0/, v1.1.0/. Bump minor for instruction tweaks, major for schema or task changes.

Git SHA tags — embed git rev-parse --short HEAD at run time for ad-hoc experiments.

Registry manifest — a registry.yaml listing version id, path, author, date, changelog line.

Example manifest:

task: ticket_classifier
active: v1.1.0
versions:
  v1.0.0:
    path: prompts/ticket_classifier/v1.0.0
    notes: Initial five-label classifier
  v1.1.0:
    path: prompts/ticket_classifier/v1.1.0
    notes: Added billing vs shipping disambiguation few-shots

Production reads active from config or environment — not hardcoded in application logic.

What to store per version

Each version directory should be self-contained:

prompts/ticket_classifier/v1.1.0/
  system.md
  few_shots.jsonl
  schema.json          # if structured output
  CHANGELOG.md         # one paragraph
  metadata.yaml        # model hint, temperature, max_tokens

At runtime, log a bundle:

{
  "prompt_version": "v1.1.0",
  "model": "gpt-4o-mini",
  "temperature": 0,
  "messages_hash": "sha256:...",
  "latency_ms": 412,
  "input_tokens": 890,
  "output_tokens": 3
}

Store bundles in runs/ or a SQLite table. You will reuse this format for milestone evals.

A/B testing prompts

A/B here means offline comparison on a fixed case set, not necessarily live traffic splitting (that comes with online evals later). Process:

1. Freeze cases.jsonl — inputs with optional gold labels or rubric notes.
2. Run version A and version B with identical model settings except prompt.
3. Write results to results/v1.0.0.jsonl and results/v1.1.0.jsonl.
4. Score with automatic checks (exact label match) plus manual review spreadsheet for ambiguous rows.

Simple runner sketch:

for case in load_cases("cases.jsonl"):
    for version in ["v1.0.0", "v1.1.0"]:
        out = run_pack(version, case["input"])
        append_jsonl(f"results/{version}.jsonl", {
            "case_id": case["id"],
            "prompt_version": version,
            "output": out,
        })

Compare with a small report script: accuracy delta, regressions (v1.0 correct → v1.1 wrong), improvements, average token use.

Callout — change one variable: When A/B testing prompts, hold model, temperature, and case set constant. Otherwise you will attribute lift to the wrong change.

Promotion criteria

Define before running the experiment:

• Ship v1.1.0 if: accuracy ≥ v1.0.0 on golden set AND no new failures on billing/shipping edge cases AND token cost ≤ 110% of baseline.
• Reject if: any safety regression or structured output validation rate drops.

Write criteria in TASK.md so future you does not rationalize a pretty failure.

Building toward the milestone eval set

This module's milestone expects a versioned prompt pack plus eval artifacts. Use this lesson to:

1. Commit at least two prompt versions with changelogs.
2. Expand cases.jsonl to ≥20 rows with diverse difficulty.
3. Store scored A/B results under m4/evals/ with timestamp and git SHA.

The eval set becomes CI input in the evals module — invest in case quality now. Include IDs, input text, expected label or rubric, and tags (edge, negation, multilingual).

Integrating with application config

Production apps should not hardcode active: v1.1.0 in source. Common patterns:

Environment variable: PROMPT_VERSION=v1.1.0 read at startup — easy rollback by redeploying env only.

Feature flag service: LaunchDarkly-style flags map user cohorts to prompt versions for live A/B (pairs with online evals later).

Git submodule or package: Prompt packs published as internal pip package @company/prompts-ticket-classifier@1.1.0 — apps pin semver in requirements.

Regardless of mechanism, the runtime log must record the resolved version on every inference. Dashboards grouping error rate by prompt_version catch regressions within minutes instead of after a week of confused support tickets.

When multiple tasks exist (classifier vs summarizer), use separate registry files or namespaced manifests — do not share one active pointer across unrelated tasks.

Common anti-patterns

Engineering problem (staff framing)

Prompts are code. Unversioned prompts make regressions undebuggable.

Diagram — Prompt as artifact

flowchart LR
  Git[Prompt git SHA] --> Eval[Eval suite]
  Eval --> Gate[Ship / rollback]
  Gate --> Prod[Prod pin]

Precise definitions & mental model

Prompt templates, variables, pins, eval gates, ownership.

Tradeoffs — when to use what

Monorepo prompts vs CMS — control vs non-eng editability.

Failure modes (interview + on-call)

Hot-edit prod prompts; no golden set; silent provider default changes.

Production & OSS practices

Treat like feature flags + migrations; require eval delta in PR.

Micro-project: Registry + A/B

In m4/registry/:

1. Create registry.yaml with ≥2 prompt versions for the same task from lesson 4.1/4.2.
2. Implement ab_run.py that executes both versions against cases.jsonl and writes comparable result files.
3. Add compare.py or a notebook cell that prints accuracy and lists regressions/improvements.
4. Document promotion decision in DECISION.md — ship or hold, with evidence.

Checklist

• Prompt versions immutable in separate folders
• Every run logs prompt_version and model settings
• A/B compared on identical cases
• Eval results committed for milestone use