Data splits and leakage

Splits are a product decision

Before you touch a loss function or a training loop, you decide what the model is allowed to know when it makes a prediction. That decision is encoded in your train/validation/test split — and it is rarely as simple as shuffling rows and taking 80%.

A split answers one question: At deployment time, what information will be available? If your fraud model sees transaction history up to today but your test set includes transactions from users who already defaulted next month, you are not measuring fraud detection — you are measuring time travel.

Random 80/20 splits work when examples are independent and identically distributed (i.i.d.): each row is unrelated to the others, and the future looks like the past. That describes textbook datasets and almost nothing in production.

Scenario	Split strategy	Why random fails
Time series / changing policies	Split by time (train on past, test on future)	Future rows leak signal from patterns that no longer hold
Users, documents, patients	Split by group (same entity never in both train and test)	Model memorizes entity-specific quirks
Retrieval / RAG eval	Hold out queries and their near-duplicate chunks	Reranker "finds" answers it saw during training
A/B test analysis	Split by experiment cohort, not by row	Same user appears in both arms

The validation set exists to tune hyperparameters and decide when to stop training. The test set is touched once, at the end, to estimate generalization. If you repeatedly peek at test metrics while iterating, test becomes validation — and you lose your honest estimate.

Callout — validation is for decisions, test is for truth: Every time you change the model because validation got worse, you are making a valid workflow choice. Every time you change the model because test got worse, you are contaminating your final number.

How leakage inflates metrics

Leakage is any path by which information from the test distribution (or the label itself) enters training. Leakage does not always mean cheating on purpose. It usually means a pipeline step that seemed harmless — scaling features on the full dataset, joining a table with future timestamps, deduplicating after the split.

The symptom is suspiciously good offline metrics and embarrassing production behavior. A tabular model that hits 99% AUC on a holdout set but fails in staging almost always has leakage, distribution shift, or a label definition that changed between train and serve.

Think of leakage as label-adjacent information crossing the split boundary. The model is not learning the underlying phenomenon; it is learning a shortcut that will not exist at inference time.

Common leakage classes

Target leakage

Target leakage means your features contain the answer — or something that only exists after the label is determined.

Examples:

• Predicting loan default using days_since_default (only defined for defaulters).
• Predicting churn using cancellation_reason (collected after the user leaves).
• Predicting disease from treatment_prescribed (doctors prescribe based on diagnosis).

Fix: ask, for each feature, "Would I have this value at the moment I need to predict?" If not, drop it or engineer a point-in-time version.

Preprocessing leakage

Preprocessing leakage happens when you fit transformers on data that includes the test set.

Examples:

• StandardScaler.fit() on train + test combined.
• TF-IDF or embedding index built on the full corpus before splitting.
• Imputing missing values with global medians computed over all rows.

Fix: fit preprocessors on training data only, then transform validation and test. In sklearn, use Pipeline + cross_val_score or a dedicated ColumnTransformer inside each fold.

Duplicate and near-duplicate leakage

Two rows that are nearly identical — same user paraphrasing the same question, same document chunked twice, same image with different crops — should not land on opposite sides of the split.

Near-duplicates are especially dangerous in NLP and retrieval: the model sees a phrasing in training and gets graded on a rephrasing in test.

Fix: deduplicate before splitting, or split by a stable key (document ID, user ID) so duplicates stay together.

Group leakage

Group leakage occurs when correlated rows from the same entity appear in both train and test.

Examples:

• Multiple hospital visits from one patient in both splits.
• Ten photos of the same product in train and one in test.
• Session-level data where early clicks are in train and the conversion is in test.

Fix: group-aware splitting — assign entire groups to one split. Libraries like GroupKFold in sklearn implement this pattern.

Temporal leakage

Temporal leakage is a special case of group leakage where the group is time itself.

Examples:

• Training on 2024 data that includes outcomes from events that happened in 2025 test period.
• Using a feature refreshed nightly when your model only gets weekly batch inputs at serve time.

Fix: use a cutoff timestamp. All rows with event_time < cutoff go to train; everything after goes to test. For rolling deployments, use walk-forward validation.

Forensic workflow: finding leakage

When metrics look too good, run this checklist before celebrating:

1. List every feature and its collection timestamp relative to the label.
2. Trace the pipeline from raw data to model input — note every fit call and what data it sees.
3. Check for ID columns that encode label information (hashed IDs, row order, file paths with class names).
4. Plot feature importance — a single feature dominating is a red flag.
5. Ablation test — remove suspicious features; if metrics collapse, you found a leak.
6. Compare train vs test distributions for each feature; identical label-conditional distributions on "future" features suggest leakage.

Callout — leakage is a skill, not a bug hunt: Senior ML engineers spend real time on split design and forensic reviews. Catching leakage before launch is cheaper than explaining a demo that fails in production.

Engineering problem (staff framing)

Leakage creates fake metrics. Design splits that mirror deployment.

Diagram — Leakage paths

flowchart TD
  F[Future feature] --> L[Leak]
  R[Random user split] --> L
  S[Scaler fit on all] --> L
  L --> Fake[Fake metric]
  G[Time/group + fit train] --> Honest

Precise definitions & mental model

IID vs temporal vs group splits; leakage; contamination in LLM evals.

Tradeoffs — when to use what

Random (efficient, unrealistic) vs time/group (honest, needs data).

Failure modes (interview + on-call)

Target encoding with test; near-dup docs across RAG splits; tune on test.

Production & OSS practices

Document split in model card; hold out retrieval corpora too.

Deep dive (FAANG / OSS bar)

Push «data-splits-and-leakage» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Find planted leakage

In m2/leakage/:

1. Create (or download the course stub CSV if you add one) a small tabular dataset with at least one planted leak (e.g. a column correlated with the label via future info, or ID-like encoding).
2. Train a naive model that "wins" using the leak — log the suspicious metric (accuracy, AUC, F1).
3. Remove or fix the leak; retrain and show the metric drop. The drop should be dramatic enough to prove the leak was doing real work.
4. Document the forensic steps in REPORT.md: how you suspected the leak, how you confirmed it, and what a production-safe pipeline would do instead.

If you invent the dataset yourself, state exactly how you planted the leak so a reviewer can verify. Good planted leaks mirror real mistakes: a post-outcome column with an innocuous name, a scaler fit on the full dataset, or duplicate rows split across train and test.

Example planted leaks you can try:

• A column account_status_at_review that is only populated for fraud cases.
• Row IDs that sort by label because the CSV was exported that way.
• A feature computed with a global statistic that includes test rows.

Your REPORT.md should read like an incident postmortem: symptom, root cause, fix, and before/after numbers.

Checklist

• Before/after metrics recorded
• Leak named precisely (target, preprocessing, duplicate, group, or temporal)
• Fix described with enough detail that someone else could reproduce it
• Split strategy documented for how this model would actually be deployed