Data flywheels

Production AI compounds only with deliberate feedback loops

Models do not automatically improve from user traffic. A data flywheel converts signals — thumbs down, support escalations, verify failures, edited outputs — into labeled datasets that feed eval expansion, fine-tuning, and prompt fixes. Without priority rules and poison guards, flywheels amplify noise, adversarial inputs, and demographic bias.

Design the loop before collecting terabytes of chat logs nobody will label.

Flywheel stages

Signal capture → Triage queue → Label / adjudicate → Dataset version → Train or eval → Deploy → measure

Each stage needs owners and SLAs — even in capstone, document intended owners.

Signals to capture:

• Explicit feedback (thumbs, stars)
• Implicit (user rewrote answer, abandoned session)
• Harness verify failures
• Human support corrections
• Red-team failures promoted to golden set

Store raw events in feedback_events.jsonl with run_id, not just aggregated counts.

Labeling and priority rules

Not all negatives are equal. Priority score example:

priority = severity_weight * frequency * label_confidence

Define adjudication: one human labeler for P3+, two for P0 policy cases.

Callout — Flywheel ≠ dump logs to fine-tune: Most raw chat is useless or harmful for FT. Curate subsets with clear objective.

Avoid poisoning fine-tune and eval sets

Poison sources:

• User injections in feedback text ("always say X")
• Model-generated labels without human review on high-stakes data
• Survivorship bias — only logging "successful" sessions
• Synthetic echo — model outputs relabeled as gold without filter (frontier module deepens)

Guardrails:

• Human review gate before train.jsonl append
• Separate eval_holdout never auto-ingested from production
• Version datasets (feedback_v3) with changelogs
• Deduplicate near-identical failures

Document in flywheel_policy.md.

Closing the loop to evals vs fine-tune

Capstone should pick one primary loop (usually eval expansion) — FT optional given module scope.

Metrics for flywheel health

Track:

• Time signal → labeled case
• % labels used vs discarded
• Eval pass rate trend after ingest
• Poison incidents (manual audit sample)

Labeling tooling minimal viable

Spreadsheet works for 50 cases; beyond that, lightweight labeling UI or Label Studio export — flywheel_policy names tool choice. Budget half-day to build import from feedback_events.jsonl to labeling queue.

Incentives for human labelers

Internal teams label faster when they see fixes ship — close loop publicly: "your thumbs-down became golden case refund-47." Product ops detail for flywheel adoption.

Negative flywheel guard

Monitor when automated ingest increases error rate after FT on feedback data — sign of poison or bias. Automatic halt rule: if post-FT eval drops >3%, revert dataset version.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Close the loop: prod interactions → labels/prefs → improve model/prompts/RAG.

Diagram — Flywheel

flowchart LR
  Users --> Logs --> Label --> Train/Eval --> Better --> Users

Precise definitions & mental model

Feedback types, sampling bias, consent.

Tradeoffs — when to use what

Human label cost vs noisy implicit feedback.

Failure modes (interview + on-call)

Training on thumbs without context; privacy violations.

Production & OSS practices

Governed pipelines; quality gates on new data.

Micro-project: Feedback → dataset design

Ship:

1. flywheel_policy.md with stages, priorities, poison rules.
2. Schema for feedback events + example JSONL rows.
3. Triage script: ingest events → prioritized queue CSV or markdown.
4. Simulate 10 events → show which enter golden candidate vs discard with reasons.
5. Capstone section: which signals you will actually collect at launch.

Acceptance: policy explains why one malicious feedback example is rejected.

Checklist

• flywheel_policy.md complete
• Feedback event schema committed
• Triage/priority script or documented manual process
• Poisoning guards explicit
• Capstone signal collection plan linked