Classical ML literacy

Why LLM engineers still need this

RAG evals, fine-tune datasets, and agent scorers all fail the same way classical ML failed: leakage, overfit demos, and no baseline. Fast.ai and Ng both teach this early for a reason.

You can ship an impressive LLM demo and still have a scientifically invalid eval. Classical literacy is how you catch yourself.

Splits and leakage

• Train — fit parameters / prompts / index contents carefully
• Validation — tune hyperparameters and prompt variants
• Test / holdout — touch once for ship decisions

Leakage = future or target information sneaks into features or retrieval. Example: embedding the answer document into the corpus you retrieve for the same question in the eval set.

flowchart LR
  Raw[Raw data] --> Split[Split first]
  Split --> Train[Train / index]
  Split --> Val[Validate]
  Split --> Test[Final test]

LLM-shaped leakage examples

Classic leakage	GenAI analog
Target column in features	Answer text inside retrieved chunks for the eval question
Random split on time series	Training prompts on next week’s tickets
Duplicate rows train/test	Same FAQ paraphrases in gold and few-shot demos
Peer features from future	Using post-resolution fields in a “predict escalate” model

Ship rule: split users / time / documents the way production will see them — then build indexes and few-shots only from allowed slices.

Overfitting in GenAI clothes

Classic symptom	LLM analog
Memorizes train rows	Prompt overfit to golden demos
Val loss rises	Eval set polished; production queries fail
Complex model wins tiny data	Huge model + tiny LoRA set → brittle
Endless feature fiddling	Endless prompt fiddling without a freeze

flowchart TD
  Fit[Fit prompt / model / index] --> Val[Score validation]
  Val --> Gap{Train >> Val?}
  Gap -->|Yes| Reg[Simplify / more data / freeze demos]
  Gap -->|No| Test[Touch holdout once]
  Test --> Ship[Ship + monitor]

Baselines before LLMs

Always beat:

1. Keyword / BM25 search
2. Regex or rules for structured fields
3. A small classifier when labels exist
4. “Return top FAQ template” for support intents

If BM25 + FAQ template scores 0.82 and your RAG scores 0.84, celebrate carefully — measure cost and latency too.

Metrics literacy (minimum set)

Problem type	Start with
Classification	Precision / recall / F1; confusion matrix
Ranking / retrieval	Recall@k, MRR, nDCG
Generation	Task rubrics + faithfulness; not BLEU alone
Agents	Task success + illegal tool rate + cost/steps

Pair with Evals fundamentals.

How to build intuition

1. Train a tiny sklearn model; plot train vs val error as capacity grows.
2. Deliberately leak a feature; watch val score become dishonestly high.
3. Freeze a 50-example gold set before prompt hacking.
4. Compare BM25 vs embedding retrieval on the same questions.

Tools today

Layer	Examples
Classical ML	scikit-learn, XGBoost, LightGBM
Experiment tracking	W&B, MLflow
Data splits	Explicit time/user keys in your warehouse
LLM eval harnesses	Custom golden JSONL + CI

Failure modes

Failure	Fix
Tuning on test	Lock holdout; use val only
Prompt overfit to 10 demos	Larger gold; rotate demos
No baseline	Ship BM25/rules first
Metric mismatch	Optimize the metric users feel

Tradeoffs

• Simple models — debuggable, cheap; may miss nuanced language.
• LLMs — flexible language; expensive, harder to guarantee.
• Hybrid — LLM for language, classical model for calibrated risk scores.

Glossary

Term	Meaning
Leakage	Invalid information path that inflates scores
Holdout	Untouched test set for final decisions
Baseline	Simple system that defines the bar
Overfitting	Memorizing train quirks; failing new data
Calibration	Predicted probabilities match real frequencies

Micro-project

Train a tiny sklearn baseline and plot train vs val error. Then write one GenAI leakage example for a RAG eval you care about.

Related guided path

ML/DL literacy — data splits, loss curves, embeddings before LLMs, PyTorch loop. Next often Tokenization and Embeddings.