Classical ML era

Features, models, leaderboards

The classical machine learning era — roughly 1990s through early 2010s in industry mindshare, with gradient boosted trees still dominant on tabular data today — standardized a loop that remains the backbone of applied ML:

1. Collect labeled data — examples with known outcomes.
2. Engineer features — turn raw records into numeric or categorical columns the model can consume.
3. Fit a model — logistic regression, decision trees, SVMs, random forests, gradient boosting (XGBoost, LightGBM).
4. Evaluate on held-out data — never trust training accuracy alone.
5. Deploy if metrics and latency fit the product — then monitor drift.

Large language models did not delete this loop. They relocated work from manual feature engineering to data curation, prompt design, and fine-tuning — and they changed assumptions about label scarcity. Tabular fraud detection at a bank still looks like classical ML. Recommendation cold-start still uses embeddings plus GBMs. Responsible AI engineering means knowing when a sklearn baseline beats a $0.02/call LLM.

This lesson places classical ML in the historical arc (after symbolic walls, before representation learning at scale) and gives you hands-on metric discipline you will reuse in eval modules.

Historical placement

After symbolic AI hit coverage walls on perception and language, practitioners turned to statistics + optimization: learn patterns from data instead of encoding every pattern by hand.

Landmarks in industry (not exhaustive):

• 1990s — SVMs, kernel methods; spam filters with hand-tuned features.
• 2000s — ensemble trees; Netflix prize era; ML enters ads and search ranking.
• 2010s — XGBoost dominates Kaggle tabular; "feature stores" emerge in big tech.
• Still now — GBMs on structured data often beat generic LLM prompts on cost and latency.

Deep learning (next lesson) ate vision and much of NLP. Classical ML remained king on rows and columns — and on small data regimes where billion-parameter models overfit or overcost.

Callout — LLMs did not replace metrics: Product owners still ask "what is precision on fraud?" not "does the chatbot feel smart?" Train/test splits and leakage awareness from classical ML apply directly to LLM eval sets later.

The classical ML loop in detail

Features

Features are the interface between domain knowledge and math. Examples:

• User age, account tenure, transaction amount → fraud model.
• TF-IDF vectors over words → baseline text classifier before transformers.
• Aggregates: "count of logins in last 7 days."

Bad features cap performance no matter the algorithm. Good features make simple models competitive — a lesson teams forget when they default to LLMs for tabular problems.

Model choice (practical)

Start simple. A logistic regression that beats a random guess in an hour beats a week tuning a neural net on 500 rows.

Metrics beyond accuracy

Accuracy alone misleads on imbalanced classes (99% non-fraud → predict all legit → 99% accuracy). Always pair a primary metric with context:

Choose metrics that match product cost asymmetry: false negative on cancer screening vs. false positive on marketing email differ radically.

Train/test discipline (practical)

Holdout test set: Split data once (or use cross-validation on train only). Touch the test set only for final reporting — not for feature ideas, not for hyperparameter search.

Leakage — the silent killer:

• Including future information in features (predict churn using "days until cancel" computed after cancel).
• Duplicates across train and test (same user in both splits).
• Scaling/normalizing using full-dataset statistics before splitting.

The training literacy module goes deeper on splits and leakage. For now: if your test metric looks "too good," suspect leakage before celebrating.

Baselines: Always compare against a dumb baseline — majority class predictor, simple heuristic, or linear model. Beating baseline by 2 points with 10× complexity may not ship.

Why engineers still reach for classical ML

• Tabular business data with clear labels and stable schema.
• Tight latency/cost — microseconds vs. seconds for LLM calls.
• Regulatory preference for simpler, explainable models (GBM + SHAP still beats black-box 70B).
• Strong baselines before adding an LLM — if logistic regression solves it, do not burn tokens.

When someone proposes "fine-tune GPT for everything," ask whether the problem is actually rows, rules, and labels.

Engineering problem (staff framing)

Classical ML is still how tabular products ship. Master splits, leakage, and metrics that match business cost.

Diagram — Supervised learning loop

flowchart LR
  Raw --> Feat[Features] --> Split --> Fit --> Met[Metrics] --> Deploy --> Raw

Precise definitions & mental model

Supervised learning, bias–variance, leakage, strong tabular baselines (linear/GBM).

Tradeoffs — when to use what

Failure modes (interview + on-call)

Random split on time data; accuracy on rare fraud; uncalibrated probabilities.

Production & OSS practices

Feature stores, train/serve skew checks, model cards, always ship a baseline.

Deep dive (FAANG / OSS bar)

Push «classical-ml-era» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Tiny sklearn baseline

In m1/sklearn_baseline/:

1. Use a built-in dataset — sklearn.datasets.load_breast_cancer (binary classification) or load_digits (multiclass) keeps deps minimal.
2. Train/test split with fixed random_state for reproducibility (e.g., 80/20).
3. Fit a simple model — LogisticRegression or RandomForestClassifier.
4. Report accuracy and a second metric (F1 macro for digits, ROC-AUC or F1 for breast cancer).
5. Save metrics to metrics.json:

{
  "dataset": "breast_cancer",
  "model": "logistic_regression",
  "accuracy": 0.97,
  "roc_auc": 0.99,
  "train_size": 455,
  "test_size": 114
}

6. Write NOTES.md (5+ lines): would accuracy alone mislead a product owner on this dataset? Why did you pick the second metric?

Add requirements.txt or use your portfolio lockfile with scikit-learn pinned. Script should run offline after deps install.

Checklist

• Script runs offline after deps install
• Metrics printed and saved to metrics.json
• Short note on metric choice (NOTES.md or README)