Weights & Biases

What Weights & Biases is (plain English)

Weights & Biases (W&B) is an experiment-tracking and collaboration platform widely used in ML and increasingly in LLMOps: log runs, compare metrics, attach artifacts (checkpoints, eval JSON, prompt files), and share dashboards.

Peers exist (MLflow, Aim, Neptune, Comet). ShipAI teaches W&B because courses and startups use it heavily — the habits transfer.

Analogy: git for experiments. Commits alone don’t tell you which prompt won — runs + configs + eval tables do.

flowchart LR
  Run[Train / prompt / eval run] --> Log[Params + metrics + artifacts]
  Log --> Dash[Compare dashboard]
  Dash --> Pick[Promote winner]
  Pick --> Reg[Registry / deploy]
  Prod[Production traces] --> Eval[Online eval sample]
  Eval --> Run

Interview cue: Experiment tracking is how you turn “vibes” into release evidence. Name what you version (prompt, retriever, model, suite).

Why LLM teams need a run tracker

Prompting, RAG configs, LoRA, and eval suites create combinatorial variants. Without tracking:

• “What beat baseline?” is Slack folklore
• You cannot reproduce a golden-set score
• Prompt edits ship without lineage

W&B (or equivalent) makes comparisons first-class.

How it fits LLM apps

Pair with MLflow for LLMOps when you need a registry-centric story (Databricks paths). Many teams use both concepts even if one tool wins.

flowchart TB
  subgraph research [Research / iterate]
    WB[W&B runs + tables]
  end
  subgraph release [Release]
    Reg[Model / prompt registry]
    Gate[Eval gate]
  end
  subgraph prod [Production]
    OTel[OTel traces]
    Sample[Online eval sample]
  end
  WB --> Gate
  Gate --> Reg
  Reg --> ProdServe[Serving]
  ProdServe --> OTel
  OTel --> Sample
  Sample --> WB

Architecture of a good run

flowchart TB
  subgraph run [wandb run]
    C[config: hparams]
    M[metrics over steps]
    A[artifacts: model / tables]
    S[summary: final scores]
  end
  Code[Git commit SHA] --> C
  Data[Dataset / golden set version] --> C

Minimum viable lineage: git SHA + data version + model id + prompt hash + eval suite id.

What to put in config vs metrics vs artifacts

How to use (shape)

# Shape — works offline with a markdown surrogate if you lack an account
import wandb

wandb.init(project="shipai-lora", config={
    "base_model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "lora_r": 16,
    "lr": 2e-4,
    "eval_suite": "report-card@v3",
    "git": "abc123",
})
# train loop...
wandb.log({"train/loss": loss, "eval/exact_match": em})
wandb.log({"eval_table": wandb.Table(dataframe=df_preds)})
wandb.finish()

For prompt-only work, still init a run: log inputs/outputs tables (redacted) and aggregate scores — not only training curves.

Local markdown surrogate (labs without an account)

# run_2026-08-12_prompt_v4
- git: abc123
- model: gpt-4.1-mini
- prompt_hash: sha256:...
- suite: rag-golden@v2
- recall@5: 0.72
- notes: chunk 512 → 768 helped billing questions

The habit matters more than the vendor. Promote the same fields when you adopt W&B or MLflow.

Prompt and RAG experiments (not just training)

LLM teams underuse trackers on non-training work. Log:

1. Prompt text / template version as an artifact
2. Retriever config (chunk, overlap, top-k, embed model)
3. Judge or golden metrics
4. Example failure rows in a table

flowchart LR
  P1[Prompt v3] --> E[Eval suite]
  P2[Prompt v4] --> E
  E --> T[W&B table]
  T --> Dec[Promote / reject]

Core companion: Evals fundamentals. Guided: Evals, guardrails, safety.

W&B vs MLflow (pick habits, not religion)

Many teams conceptually use both: W&B for research compare, MLflow registry for promote. See MLflow for LLMOps.

Walkthrough: promote a RAG change

1. Baseline run: chunk 512, top_k=5, recall@5=0.61.
2. Candidate: chunk 768, hybrid on, recall@5=0.74; latency +40ms.
3. Compare tables; spot regressions on two question types.
4. Fix metadata filter; re-run; still win.
5. Gate: attach suite artifact + run_id to release notes; stamp prompt_version / retriever_version on OpenTelemetry for LLMs spans.

Alternatives

Production gotchas

• Logging raw PII in tables — redact or hash
• Metric soup without a single decision metric
• Orphan runs — no git SHA → unreproducible
• Training-only culture — forget to log prompt/RAG configs
• Treating W&B as ground truth — still keep eval code in git
• No link to prod traces — store run_id / prompt_version on OTel spans
• Huge artifact uploads every step — sample failures instead
• Comparing different suites as if scores were commensurate

flowchart TD
  Bad[Cannot reproduce win] --> C{Missing?}
  C -->|git SHA| G[Log commit]
  C -->|data version| D[Log suite + dataset]
  C -->|prompt text| P[Artifact prompt]
  C -->|decision metric| M[Pick one primary metric]

Failure modes checklist

1. Ten metrics, no ship/no-ship rule
2. Prompt edited in UI only — never in git
3. Eval table contains emails / phone numbers
4. Prod model has no run_id lineage
5. “W&B says 0.9” but suite code changed silently

Hands-on next steps

1. Log a LoRA or prompt A/B run with config + eval table.
2. Compare two runs; write a 5-line promote/reject note.
3. Guided Eval vs base / teacher + Evals, guardrails, safety.
4. Read MLflow for LLMOps for registry gates.

Micro-project

1. Create two prompt variants for a 20-question golden set.
2. Log each as a W&B run (or markdown surrogate) with prompt hash + scores.
3. Pick a single decision metric; promote one.
4. Add prompt_version to a fake OTel span JSON.

Interview whiteboard: lineage chain

Draw:

git SHA → data/suite version → run (config+metrics) → artifact → registry alias → prod span(run_id)

If any link is missing, you cannot answer “why did prod change on Tuesday?”

Picking a decision metric

Secondary metrics inform; they should not all be bold in the dashboard.

Online meets offline

Offline golden sets catch regressions before ship. Online sampled traces catch distribution shift.

Link them with prompt_version / run_id on spans — OpenTelemetry for LLMs.

Artifact hygiene

• Version prompts as files, not screenshot pastes
• Store small failure slices, not full raw corpora with PII
• Alias staging / prod only after gates pass
• Keep eval code in git even if scores live in W&B

Tradeoffs summary

Checklist

• Primary decision metric agreed
• git SHA + suite id on every run
• Prompts/artifacts versioned
• PII redaction on tables
• run_id linked from prod traces
• Promote/reject note template used

Glossary

Debugging playbook (first hour)

Anti-patterns

1. Training-only culture — prompts/RAG never logged.
2. Ten bold metrics — no decision rule.
3. Prompt edited only in a UI — not in git.
4. Comparing different golden sets as if scores match.
5. Uploading full corpora with PII as artifacts.
6. Treating W&B as source of truth for code — eval code lives in git.
7. Promote without a written note — Slack folklore returns.

Security and privacy threat note

• Eval tables often contain user text — redact emails, phones, secrets.
• Artifacts can include API keys in notebook outputs — scrub before upload.
• Access control on projects: contractors should not see prod traces.
• Linkage from run_id → customer traffic is sensitive — scope carefully.

Interview prompts you should be able to answer

1. Draw lineage: git → data/suite → run → artifact → registry → prod span.
2. What belongs in config vs metrics vs artifacts for a RAG A/B?
3. How do offline golden sets and online sampled evals connect?
4. When would you prefer MLflow’s registry story vs W&B compare UI?
5. What is a decision metric and why does it matter more than dashboards?

End-to-end: promote a change safely

flowchart LR
  Idea[Prompt / chunk / LoRA change] --> Run[W&B run + config]
  Run --> Table[Eval table + primary metric]
  Table --> Note[Promote/reject note]
  Note --> Gate{Pass gate?}
  Gate -->|yes| Reg[Alias staging/prod]
  Gate -->|no| Fix[Iterate]
  Reg --> Prod[Serve]
  Prod --> Span[OTel span + run_id]
  Span --> Sample[Online eval sample]
  Sample --> Run

Minimum promote note: baseline metric, candidate metric, regressions found, suite id, git SHA, rollback alias.

Production readiness checklist

• Primary decision metric agreed in writing
• git SHA + suite id + prompt hash on every run
• PII redaction on tables/artifacts
• Promote/reject note template
• Registry aliases only after gate
• run_id / prompt_version on prod spans
• Eval code versioned in git
• Offline + online loop documented

How it works end-to-end (experiment lifecycle)

1. Start a run with config (prompt hash, model, retriever, seed).
2. Log metrics over steps (loss, eval scores, cost, latency).
3. Attach artifacts (prompt file, eval JSON, checkpoint, chunker config).
4. Compare runs on a fixed decision metric.
5. Promote winner to registry / config flag; record lineage.
6. Sample production traces → offline suite → new runs.

flowchart TD
  Idea[Change prompt/RAG/LoRA] --> Run[W&B run]
  Run --> Art[Artifacts]
  Run --> Met[Metrics table]
  Met --> Cmp[Compare to baseline]
  Cmp -->|win| Promo[Promote + version]
  Cmp -->|lose| Keep[Keep baseline]
  Promo --> Prod[Production]
  Prod --> Sample[Online sample]
  Sample --> Run

What to log for LLM systems (checklist)

If it influenced the score and isn’t logged, the run is folklore.

Promote gates (do not skip)

Define before experiments:

1. Primary metric + minimum delta vs baseline
2. No-go metrics (safety fail rate, p95 latency, cost)
3. Required artifact set
4. Reviewer / auto-gate in CI

W&B shows charts; you define the gate. Pair with Evals fundamentals.

W&B Tables for qualitative review

Log example rows: query, baseline answer, candidate answer, judge score, notes. Engineers spot systematic failures charts miss (tone, citation format, refusal quality).

Secrets and compliance

• Do not log raw PII prompts from production without redaction
• Restrict project access; separate prod vs research projects
• API keys in env, not notebooks committed to git
• Artifact stores may contain model weights — ACL them

Local surrogate when you lack an account

ShipAI labs can use a markdown/CSV surrogate:

runs/YYYYMMDD_prompt_v4.md  # config + metrics + links to artifacts/

Habits matter more than the vendor. Graduate to W&B/MLflow when collaboration needs dashboards.

Worked walkthrough: prompt A/B for support tone

1. Baseline prompt artifact prompt_v3.
2. Candidate prompt_v4 (shorter, citation-first).
3. Same golden suite support_v2; same model pin.
4. Log cost + faithfulness + user-rubric tone score.
5. Promote only if faithfulness ≥ baseline and tone +Δ ≥ threshold.
6. Tag production release with run URL.

FAQ (W&B)

Is W&B only for training?
No — prompts and RAG configs need runs too.

W&B vs MLflow?
Either; see MLflow for LLMOps. Pick one habit per team.

Can dashboards replace evals?
No. Dashboards display evals.

Deep dive: lineage for RAG launches

A releasable RAG change should point to: chunker version, embed model revision, index build id, prompt hash, model id, eval suite version, and W&B run id. Missing links make incidents un-debuggable.

Putting what / why / how together

Architecture that survives team use

flowchart TB
  Dev[Engineer laptop / CI] --> Run[W&B run]
  Run --> Proj[W&B project]
  Proj --> Reg[Model / artifact registry]
  Reg --> Deploy[Config flag / CD]
  Prod[Prod sampler] --> Offline[Eval workers]
  Offline --> Run

Separate projects: research-sandbox, rag-prod-candidates, llm-training. Permissions differ. Do not let every notebook write into the prod candidates project.

Config as code

Store canonical configs in git; W&B run config should match a git sha. “Edited in UI only” recreates folklore. Prefer: PR → CI run → W&B → human/auto gate → merge.

Decision metrics catalog (pick few)

One primary metric per experiment type. Ten equal metrics = no decision.

CI integration sketch

PR opened → run eval suite → log W&B run with git sha →
post summary to PR → fail if guardrail breached

Flaky judges need fixed seeds / majority vote — otherwise CI becomes noise.

Interview whiteboard

Draw lineage: git sha → run id → artifacts → prod flag. Ask: “If prod regresses, which run do we roll back to?”

Failure story bank

1. Best chart used a different golden set than baseline.
2. Prompt artifact missing → cannot reproduce.
3. Prod logging PII into W&B tables.
4. Everyone uses personal entities; team cannot find runs.

End-to-end lab checklist (do this once)

• Log prompt + model + suite ids
• Attach eval JSON artifact
• Compare two runs on one primary metric
• Write promote gate in README
• Redaction rule documented
• Optional: MLflow parity note

Production readiness checklist (experiment tracking)

• Projects split by trust level (sandbox vs prod candidates)
• Required config fields enforced in helper SDK
• Artifacts always include prompt/suite/model pins
• Primary metric + guardrails documented per experiment type
• PII redaction before logging prod samples
• CI can publish runs with git sha
• Promote path writes lineage into release notes
• Retention policy for artifacts/weights

What “good” looks like in a design doc

Lineage diagram, decision metrics, gate thresholds, who can promote, where prod reads its prompt/model versions from, and how online samples re-enter offline suites.

Common interview traps

• Charts without dataset version
• “W&B is our eval” (it’s storage/UI for evals)
• Logging secrets or raw PII
• Incomparable runs (different suites) sold as A/B

Online evaluation loop

Sample 1% of prod traces → redact → judge offline → log W&B run tagged online_sample → page if guardrail breaches. Without this, offline wins drift from reality.

Glossary addendum

Micro-project stretch

Create baseline + candidate prompt runs; attach a 20-row W&B table (or CSV surrogate) with side-by-side answers; write a promote/no-promote decision with the primary metric cited.

How to evaluate the tracking practice itself

Audit monthly: sample 10 prod changes; each must cite a run id.

Team norms that make W&B useful

1. One project naming scheme (team-domain-env)
2. Run names include hypothesis (rrf_k60_vs_vector)
3. Tags: baseline, candidate, prod-sample
4. Saved views for the decision metric

Tools don’t create culture — norms do.

Artifacts for LoRA and prompts

Fine-tunes: log base model revision, LoRA rank, dataset hash, eval table, adapter artifact. Prompts: plain text file artifact + hash in config. RAG: dump retriever YAML as artifact. Future you will thank present you.

Pairing with online telemetry

W&B is not a substitute for OpenTelemetry for LLMs. Use OTel for per-request prod traces; W&B for experiment batches and promotion evidence. Link via release_id / prompt_hash.

Putting what / why / how together

Comparing prompt, RAG, and LoRA runs fairly

Hold constant everything outside the hypothesis:

Contaminated comparisons waste weeks. Put the frozen fields in the run config automatically via a shared helper.

Registry promotion record

When promoting, write a tiny RELEASE.md fragment:

prompt_hash: ...
run_id: ...
suite: ...
metrics: ...
approver: ...

Store alongside the app config version. Incidents become bisectable.

FAQ addendum

Do I need W&B for a solo portfolio?
A markdown surrogate is fine; demonstrate the habits.

Can I log every prod request to W&B?
Usually too expensive/PII-heavy — sample + redact.

Related

Advanced Key Tech: MLflow for LLMOps, OpenTelemetry for LLMs. Core: Evals fundamentals. Advanced: Fine-tuning LoRA/QLoRA. Guided Deploy, cost, latency, observability, Build & serve your SLM.