Dataset curation and licenses

Your SLM is only as defensible as its data paperwork

Fine-tuning memorizes patterns in training rows — including toxic phrasing, private snippets, and license-incompatible text. Before LoRA touches a GPU, build a curated JSONL dataset with provenance, licenses, splits that prevent leakage, and quality gates. This is unglamorous work that separates hobby fine-tunes from models you can ship or discuss in compliance review.

Callout — if you cannot cite the license, do not train on it: Scraping Stack Overflow, internal Slack exports, or customer tickets without clearance is a career-limiting move.

JSONL format for instruction tuning

One JSON object per line, common chat template:

{"messages": [
  {"role": "system", "content": "Classify support tickets into billing, shipping, account, bug, other."},
  {"role": "user", "content": "Charged twice for order 9921"},
  {"role": "assistant", "content": "billing"}
]}

Alternatives:

• Alpaca style: instruction, input, output fields — map to chat template in training script.
• Completion only: raw text for continued pretrain (not this lesson's focus).

Keep consistent output format across rows — the model learns delimiter patterns.

Target 300–2000 rows for a narrow LoRA task; quality beats quantity below ~200.

Sourcing data

Maintain data/PROVENANCE.md:

| file | source | license | collected | notes |
| train.jsonl | synthetic via gpt-4o-mini | OpenAI ToS dev use | 2026-08-01 | teacher-labeled |
| val.jsonl | hand-labeled | internal OK | 2026-08-05 | no customer names |

Curation quality gates

Before accepting a row:

• Correct label/format — spot-check 10% manually; inter-annotator agreement if team.
• No PII — regex + manual scan for emails, phone, account ids unless synthetic.
• Diverse phrasing — avoid 50 near-duplicate templates.
• Balanced classes — oversample rare labels or use class weights in training.
• Length bounds — truncate outliers; log truncations.

Dedupe with normalized text hash — duplicate rows inflate metrics.

Train/validation/test splits

Random split is fine when rows are independent.

Leakage traps:

• Same customer email paraphrased in train and val → inflate val accuracy.
• Same document chunked into multiple labeled rows across splits.
• Synthetic data generator using same seed phrases.

Use group splits by source_id or author_id when duplicates cluster.

Typical ratios: 80/10/10 or 85/15 if small data; hold test untouched until final report card.

Never tune hyperparameters on test.

Callout — val reflects deployment: If production sees long emails, val must include long emails — not only short synthetic stubs.

Negative and edge examples

Include:

• Ambiguous cases with gold other
• Empty or gibberish input with safe default output
• Adversarial "ignore instructions" in user content (for robustness)

Document edge policy in system message consistently across rows.

Synthetic data with a teacher model

When hand labels are scarce, generate rows with a strong API model:

Given label billing, write 5 diverse customer emails about double charges, failed refunds, invoice questions.
Output JSONL messages format.

Risks and mitigations:

• Homogeneous phrasing — vary temperature, ask for typos/slang, mix languages if relevant.
• Label leakage in system prompt — teacher must not echo label in user content unrealistically.
• License — OpenAI/Anthropic ToS for generated data used to train competing models changes; read current terms.

Always keep teacher_model and generation date in PROVENANCE.md. Spot-check 10% of synthetic rows — garbage synthetic data teaches garbage patterns faster than no data.

Deduplication and near-duplicate detection

Beyond exact hash dedupe:

• MinHash / SimHash on character 3-grams catches template spam.
• Embedding cluster dedupe — if cosine > 0.95, keep one representative row per cluster.

Near-duplicates in train inflate val metrics when splits leak paraphrases — group by cluster id before splitting.

Label noise and adjudication

When two human labelers disagree on 10%+ of spot-check sample:

• Write adjudication guide with edge case rulings.
• Merge ambiguous classes if distinction not product-critical.
• Drop rows where gold label is genuinely unclear — noise teaches wrong boundaries.

Track label_version in dataset metadata when guide updates — retrain when guide bumps major version.

JSONL validation script

Ship validate_rows.py that fails CI on:

• Malformed JSON lines
• Missing required message roles
• Assistant content outside allowed label enum
• Rows exceeding max token length threshold

Validation script is cheap insurance before overnight GPU jobs.

Storage layout

m6/dataset/
  raw/                 # untouched exports
  processed/
    train.jsonl
    val.jsonl
    test.jsonl
  PROVENANCE.md
  stats.json           # class counts, token length histogram
  scripts/
    build_dataset.py
    validate_rows.py

validate_rows.py checks schema, enum labels, duplicate rate — run in CI.

Engineering problem (staff framing)

FT data quality/license dominates outcomes. Garbage in ⇒ confident garbage out.

Diagram — Data curation pipeline

flowchart LR
  Src[Sources] --> Lic[License check]
  Lic --> Clean[Dedup/PII scrub]
  Clean --> Split --> Train

Precise definitions & mental model

Licenses, PII, dedup, contamination vs eval.

Tradeoffs — when to use what

Synthetic scale vs human quality.

Failure modes (interview + on-call)

Training on eval; ToS-violating scrape; leaking customer data into weights.

Production & OSS practices

Datasheets; license matrix; retention policy.

Micro-project: JSONL dataset

In m6/dataset/:

1. Curate ≥300 train + ≥50 val rows for your worksheet task.
2. Complete PROVENANCE.md for every source file.
3. Group-aware split if leakage risk exists; document split strategy.
4. stats.json with class distribution and avg token length.

No training yet — that is lesson 6.3.

Checklist

• JSONL validates against schema script
• PROVENANCE complete
• No obvious duplicate leakage across splits
• Edge/ambiguous examples included