Learn AI · Browse tracks
Inference
Serving stack — KV-cache, continuous batching, quantization, speculative decoding, vLLM, cost and latency routing.
How this relates Browse here for Hello Interview–style articles. The guided path below in the sidebar remains the project milestone sequence.
Articles in this track
- vLLMHigh-throughput LLM serving — PagedAttention, continuous batching, scheduling knobs, capacity planning, and when vLLM beats naive Hugging Face generate.Open
- KV-cache, prefill, and decodeWhy TTFT and tokens/sec are different jobs — KV growth math, prefix caching, chunked prefill, and scheduling implications for multi-tenant serving.Open
- Continuous batchingStatic batches waste GPU on short requests; continuous batching admits new sequences as others finish — scheduler knobs, fairness, and prefill/decode mixing.Open
- Quantization for inferenceAWQ, GPTQ, GGUF, FP8 — shrink weights/KV to raise concurrency; calibrate on prod-like data and measure task quality, not only VRAM.Open
- Speculative decodingDraft tokens with a small model, verify with the large one — speedups when acceptance rates stay high; when to enable per route.Open
- Cost and latency routingRoute easy traffic to cheap/fast models and hard traffic to frontier — with budgets, caches, SLOs, escalation, and FinOps logging.Open