Attention → Transformer → GPT lineage

The lineage that matters for builders

Research history is full of dead ends and parallel discoveries. As an AI engineer shipping products, you need a compressed lineage — not every paper, but the causal chain that explains why ChatGPT-class systems exist and what you will implement yourself in the mini-LLM module.

This lesson is the map. The mini-LLM module is the territory: you will code attention blocks, training loops, and sampling — a tiny slice of the stack below.

Seq2seq and the attention mechanism

Before transformers dominated, sequence-to-sequence models (encoder RNN → decoder RNN) handled machine translation and summarization. Bottleneck: cramming an entire source sentence into one fixed vector.

Attention (Bahdanau et al., 2014; refined in many forms) let the decoder look back at encoder hidden states with learned weights — "where should I focus when generating this token?" Alignment became soft and differentiable. Translation quality jumped; the idea generalized beyond RNNs.

Engineer takeaway: attention is a weighted lookup over positions. That intuition survives in every transformer block you will write.

Transformer (2017): "Attention Is All You Need"

Vaswani et al. removed recurrence for many tasks and stacked:

1. Self-attention — each token attends to other tokens in the same sequence (captures context without sequential processing).
2. Multi-head attention — parallel attention subspaces (syntax-ish, coreference-ish — learned, not hand-labeled).
3. Position encoding — inject order because self-attention alone is permutation-sensitive.
4. Feed-forward sublayers + residuals + layer norm — the standard block repeated N times.

Why builders care: Self-attention parallelizes across sequence positions on GPUs — training scales. RNNs serialized over time steps; transformers unlocked scale that later met data and compute.

Encoder, decoder, and hybrids

Not all transformers are GPT. Three architectural families matter:

Decoder-only GPT lineage won the generative API era for a pragmatic reason: predicting the next token is simple to scale, and at sufficient scale the same objective produces surprising general-purpose capability (the "emergent" behaviors debate continues, but the product impact is undeniable).

GPT scale and the generative turn

Landmarks (approximate, for mental anchoring):

• GPT-1 (2018) — proof that pre-training + fine-tune works on downstream tasks.
• GPT-2 (2019) — larger, coherent long-form text; release caution narrative.
• GPT-3 (2020) — in-context learning at scale; API business models accelerate.
• ChatGPT / RLHF era (2022+) — alignment layers (SFT, preference optimization) make raw LMs usable as assistants.
• Open-weight wave (Llama, Mistral, etc.) — self-host and fine-tune (your SLM module).

Post-training (SFT, DPO, tool-use fine-tuning) maps in advanced curriculum segments; practice arrives in agent and SLM modules.

Callout — you will build a slice, not a frontier model: The mini-LLM module implements tiny dimensions (small vocab, few layers). The goal is to demystify forward pass, loss, and generation — not to compete with GPT-4.

What happens inside one forward pass (preview)

Without math overload:

1. Tokens embed into vectors.
2. Each transformer block applies self-attention (causal mask in GPT — token i cannot see future tokens).
3. FFN layers mix per-position representations.
4. Final linear layer predicts logits over vocabulary; softmax → probabilities for next token.

Training: compare predicted distribution to actual next token (cross-entropy). Generation: sample or greedy-pick next token, append, repeat.

You will feel this loop in code. API-era work makes sense when you know temperature scales logits before sampling and context window is finite attention memory.

RAG and agents on the timeline (orientation)

After base LMs scaled, systems layers exploded:

• RAG (retrieval-augmented generation) — attach external knowledge without retraining all weights (domain RAG module).
• Tool use / function calling — model emits structured calls; runtime executes (agent modules).
• Multimodal stacks — vision/audio encoders feeding LMs (landscape lesson).

These are not separate universes — they are patterns on top of decoder-only cores (often).

Common misconceptions (quick debunk)

"Transformers understand language like humans." They model statistical co-occurrence at scale. Useful, not anthropomorphic — evals catch failures humans would not make and miss failures humans would catch.

"BERT and GPT are interchangeable." Encoder-only models excel at classification and embeddings with bidirectional context; decoder-only models excel at generation with causal masking. Fine-tuning BERT for chat is the wrong tool; prompting GPT for sentence classification is expensive vs. a small encoder.

"Attention IS the memory." Context window is finite. Long documents need chunking, retrieval, or summarization — patterns in the RAG module, not bigger attention by default.

"Scale fixes everything." Scale improves many benchmarks; it does not fix hallucination, stale knowledge, or tool misuse without system design.

Knowing these saves you from wrong architecture choices in design reviews.

Engineering problem (staff framing)

Attention→Transformer→GPT is the industrial lineage behind chat APIs. Literacy prevents cargo-cult prompting and bad latency math.

Diagram — Lineage timeline

timeline
  title Conceptual lineage
  2017 : Transformer self-attention
  2018 : BERT / GPT split
  2020 : GPT-3 few-shot scale
  2022+ : Chat post-train + tools

Precise definitions & mental model

Self-attention Q/K/V, transformer block, decoder-only LM, scaling interactions.

Tradeoffs — when to use what

Failure modes (interview + on-call)

BERT vs GPT API confusion; attention≠explanation; ignoring tokenizer/context limits.

Production & OSS practices

KV-cache, batching, speculative decoding assume this stack.

Deep dive (FAANG / OSS bar)

What "GPT lineage" means for APIs

When a vendor exposes chat completions, you are usually talking to a decoder-only model with chat templates and post-training. When they expose embeddings, you may be talking to an encoder or a truncated LM. Do not assume one weight stack behind every endpoint.

Systems consequence

Training parallelizes across sequence (attention) and batch. Inference is often memory-bandwidth bound on GPUs for decode — hence KV-cache and batching obsession in serving blogs from large companies.

Micro-project: Timeline artifact

Create m1/timeline/ai-lm-timeline.md (or export SVG/Excalidraw PNG linked from markdown):

Requirements:

• ≥8 dated milestones spanning symbolic/ML → deep learning → attention/transformer → GPT-class → RAG/agents (or multimodal).
• Each item: one sentence "why builders care."
• Explicitly mark which items you will build vs use as APIs in ShipAI (e.g., build mini attention stack; use hosted API for agent demos; fine-tune open weights in SLM module).

Example row format:

| 2017 | Transformer architecture | Parallelizable training → modern stack foundation | Use API; build tiny version in mini-LLM module |

Commit so it renders readably on GitHub. This artifact becomes a section in your Milestone 1 field-map write-up and a interview whiteboard aid.

Checklist

• Timeline committed and readable in GitHub
• Mini-LLM, SLM, and agent module items explicitly called out
• Each milestone has a builder-oriented "why care" sentence