Deep learning boom

Representation learning goes mainstream

Deep learning's industry boom — popularly dated from AlexNet winning ImageNet in 2012, though neural nets have a much longer history — was less "we discovered neurons" and more a convergence of scale, algorithms, and hardware:

• End-to-end features — models learn representations from raw-ish inputs (pixels, character sequences) instead of hand-crafted feature pipelines.
• GPUs — made dense matrix operations cheap enough to train large nets repeatedly.
• Better optimization — ReLU, batch norm, Adam, residual connections; tricks that made deeper networks trainable.
• Open toolkits — Theano, Caffe, TensorFlow, PyTorch lowered the barrier from research lab to laptop.

For language, recurrent networks, LSTMs, and seq2seq models set the stage through the mid-2010s. Word embeddings (Word2Vec, GloVe) showed that dense vectors capture semantic similarity. Transformers (next lesson) later unlocked the generative wave — but the deep learning mindset — learn the features, scale the data, watch the loss — already reshaped the field.

This lesson explains what changed conceptually and gives you a minimal training loop experience before the training literacy module and mini-LLM build go deeper.

What changed vs. classical ML

Representation learning means the model learns an internal space where similar inputs cluster — edges and textures in vision, phoneme patterns in speech, syntactic and semantic regularities in text. Downstream tasks attach heads to these representations or fine-tune the whole stack.

Callout — depth without scale underwhelms: A three-layer MLP on 500 tabular rows rarely beats XGBoost. Deep learning's advantage shows up when data and compute unlock hierarchical features — not everywhere by default.

The training stack (pieces you must name)

Whether you use PyTorch, Keras, or JAX, the same components appear:

Data pipeline

Load batches from disk or memory; apply transforms (normalize pixels, tokenize text). Shuffling, batching, and num_workers in PyTorch affect throughput. Bad pipelines GPU-starve your model.

Model

Layers: linear, conv, attention (later), activations, dropout for regularization. model.parameters() are what optimizers update.

Loss function

Scalar objective minimized during training — cross-entropy for classification, MSE for regression, next-token prediction for language models. The loss is the training signal. If loss is wrong for the task, nothing else saves you.

Optimizer

SGD, Adam, AdamW — updates weights using gradients from backpropagation. Learning rate is the hyperparameter everyone touches first; often too high (loss spikes) or too low (plateau).

Evaluation loop

Separate from training: forward pass on validation data without gradient updates. Track val loss / accuracy to detect overfitting — train loss drops while val loss rises.

You do not need a research degree for ShipAI's literacy modules. You do need to recognize these five pieces in every script you read or write.

Overfitting as default risk

Neural nets are flexible enough to memorize training labels, especially small datasets. Symptoms:

• Train accuracy 99%, val accuracy 70%.
• Val loss increases while train loss decreases.

Mitigations (conceptual — you will experiment in later modules):

• More data or data augmentation.
• Dropout, weight decay (L2 regularization).
• Early stopping when val loss stops improving.
• Smaller model capacity.

Always compare against your sklearn baseline mindset: did deep learning earn its complexity on val metrics?

A minimal training loop (pseudocode)

Every framework wraps the same loop — recognize it when you read PyTorch scripts in the mini-LLM module:

for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        optimizer.zero_grad()
        logits = model(batch.x)
        loss = loss_fn(logits, batch.y)
        loss.backward()
        optimizer.step()

    model.eval()
    with torch.no_grad():
        val_loss = evaluate(model, val_loader)
    log(epoch, train_loss, val_loss)

If you can explain each line in an interview, you are ahead of many "I fine-tune with a notebook" candidates. Gradients flow backward through the computation graph; the optimizer nudges weights to reduce loss. That is the entire deep learning training story at the engineer level — everything else is scale and architecture.

GPU vs. CPU for this lesson

Your micro-project should run on CPU in minutes. That is intentional: the literacy goal is loop comprehension, not ImageNet time records. When you move to the mini-LLM module, the same loop runs with different tensor shapes and a causal language modeling loss — the muscle memory transfers.

Engineering problem (staff framing)

Representation learning removed hand features but added data/compute/opacity. Know when DL beats classical baselines.

Diagram — Representation learning

flowchart TD
  Data --> Arch[CNN/RNN/Transformer] --> Rep[Representations] --> Head --> Loss --> Arch

Precise definitions & mental model

End-to-end learning, GPU+backprop, transfer learning, architectural inductive bias.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Tiny data + huge net; bench hacking; ignoring edge latency.

Production & OSS practices

Dataset versioning, checkpoints, mixed precision, sliced evals.

Deep dive (FAANG / OSS bar)

Push «deep-learning-boom» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Train a small NN

In m1/small_nn/ (PyTorch or Keras — pick one and pin versions):

1. Train a small MLP on a tabular or vision toy dataset, or a tiny CNN on MNIST subset / Fashion-MNIST. Keep it trainable on CPU in minutes.
2. Log train and validation loss per epoch (print or save to history.json).
3. Save final metrics to metrics.json (val accuracy or val loss).
4. Write COMPARISON.md:
   • If comparable to your sklearn baseline task, compare wall-clock and metric honestly.
   • If not comparable (different dataset), explain why and compare mindset instead: feature engineering vs. learned representations, tuning surface, overfitting risk.

Example history.json snippet:

{"epochs": [{"epoch": 1, "train_loss": 0.45, "val_loss": 0.38}, {"epoch": 5, "train_loss": 0.12, "val_loss": 0.22}]}

Add README with: how to run, hardware used, first hyperparameter you would change to improve val loss.

Callout — document the first knob: "I would halve the learning rate" or "add dropout 0.3" shows engineering taste. Reviewers learn more from your improvement plan than from a perfect score.

Checklist

• Training script + requirements pinned
• Final metrics saved (metrics.json or equivalent)
• Note: what would you change first to improve val loss?