Self-attention

The engineering problem

Early sequence models (RNNs/LSTMs) processed tokens mostly left-to-right; long-range dependencies were hard and training did not parallelize well over the sequence.

Self-attention (the Transformer’s core move) lets each position compute a weighted mix of all positions in the window. That buys:

• Parallelism over sequence length during training
• Direct paths for long-range dependencies
• A clear reason context is expensive: naive attention is O(n²) in sequence length

You do not need to derive every matrix calculus identity to ship LLM products — but you do need the cost and masking intuition, or Inference and “why is my prompt slow?” stay mysterious.

Mental model: Query, Key, Value

For each token embedding:

• Query (Q) — “what am I looking for?”
• Key (K) — “what do I contain?”
• Value (V) — “what do I contribute if selected?”

Scores ≈ how well each Query matches each Key; Softmax turns scores into weights; output is the weighted sum of Values.

flowchart TD
  X[Token embeddings] --> Q[Query]
  X --> K[Key]
  X --> V[Value]
  Q --> S[Scores QKᵀ / √d]
  K --> S
  S --> Soft[Softmax]
  Soft --> Mix[Weighted sum of V]
  Mix --> Out[Context-aware vectors]

A high attention weight from token i to token j means: when representing i, the layer leans heavily on j’s Value — e.g. a pronoun attending to its noun, or a ) attending to a matching (.

Multi-head and layers

Multi-head attention runs several Q/K/V projections in parallel so different heads can specialize (syntax, copy, coreference, code structure). Stacked layers deepen the representation; residual connections + LayerNorm keep training stable.

flowchart LR
  In[Hidden states] --> H1[Head 1]
  In --> H2[Head 2]
  In --> H3[Head N]
  H1 --> Cat[Concat + project]
  H2 --> Cat
  H3 --> Cat
  Cat --> FFN[Feed-forward]
  FFN --> Out[Next layer]

Causal masks (why chat models don’t “see the future”)

Autoregressive LLMs train with a causal mask: position i may only attend to positions ≤ i. That matches left-to-right generation. Encoder-only models (classic BERT-style) use bidirectional attention for understanding tasks; modern generative chat stacks are usually decoder-style with causality.

Step-by-step: one forward pass (intuition)

1. Embed tokens (+ positions / RoPE-style position info).
2. For each layer: build Q, K, V → attention → residual → FFN → residual.
3. Final hidden state → logits over vocabulary → next-token distribution.
4. At decode time, generate one token at a time; KV cache stores past K/V so you do not recompute the whole prompt every step (see Inference track).

Why context length hurts (product view)

Effect	Cause
Prefill latency ↑	Attention over long prompts is heavy
Memory ↑	Activations + KV cache grow with tokens × layers × heads
Cost ↑	Providers bill tokens; long RAG dumps burn money
Quality ≠ monotonic	More tokens can dilute attention (“lost in the middle”)

Ship rule: treat context as a scarce packing problem, not a dump truck — same lesson as RAG and agents tool observations.

How to build intuition (without a PhD)

1. Sketch Q·Kᵀ for a 4-token sentence; mark which pairs should be strong.
2. Implement tiny attention in NumPy/PyTorch in guided Build an LLM from scratch.
3. Profile a long vs short prompt’s TTFT on any API — feel prefill cost.
4. Read Inference pages on KV-cache and PagedAttention once attention “clicks.”

Tools / stacks that expose attention economics (2025–2026)

Area	Examples	Why it matters
Training / learning	PyTorch, Hugging Face Transformers	Implement & inspect
Serving	vLLM, SGLang, TensorRT-LLM	KV cache, batching
Long context	Model-native long windows, sliding/attention variants	Product limits
Observability	Prefill vs decode metrics	Capacity planning

Exact kernel tricks change; the O(n²) mental model still predicts bills and latency.

Failure modes and misconceptions

Myth / failure	Reality
“Attention = explanation”	Weights are not faithful user-facing explanations
“Longer context always better”	Noise and cost often win; pack deliberately
“KV cache is optional”	Without it, decode is unusably slow
Ignoring masks	Train/serve mismatch → garbage generations

Tradeoffs

• Full attention — strongest quality default; quadratic cost.
• Sparse / sliding / linearized variants — cheaper long context; different quality curves.
• More heads / layers — capacity vs latency and VRAM.

When to use this knowledge

• Designing RAG packers and agent tool budgets
• Choosing context windows and summarizing history
• Talking to infra about GPUs, batching, and TTFT
• Interview systems questions: “Why is long context expensive?”

Glossary

Term	Meaning
Self-attention	Attention where Q/K/V come from the same sequence
Multi-head	Parallel attention subspaces
Causal mask	Block attending to future tokens
KV cache	Stored keys/values for fast autoregressive decode
Prefill vs decode	Prompt processing vs token-by-token generation

Micro-project

Sketch Q·Kᵀ for a 4-token sequence and explain in one paragraph what a high weight means for one pair. Then measure TTFT for a 500-token vs 4k-token prompt on any model API.

Related guided path

• Guided Self-attention and Tiny GPT / mini-LLM end-to-end — implement attention and train a tiny GPT
• Tokenization — what the sequence is
• Inference — KV-cache / prefill / decode, PagedAttention
• Advanced — MoE sits beside attention blocks, not instead of them