Core Concepts

Tokenization

Tokens are the interface between UTF-8 text and model IDs — BPE, costs, chat templates, and why bad splits break products.

40 min

Also in guided path: Tokenization and BPE

Why engineers care about tokens

Models do not see characters or words. They see integer token IDs. Billing, context limits, RAG chunk sizes, streaming UIs, and many “weird” model failures are token problems in disguise.

If you only remember one sentence: the tokenizer is the API contract between your UTF-8 product text and the neural net.

flowchart LR
  Text[UTF-8 text] --> Tok[Tokenizer]
  Tok --> Ids[Token IDs]
  Ids --> Model[Transformer]
  Model --> OutIds[Output IDs]
  OutIds --> Detok[Detokenize]
  Detok --> TextOut[Text]

Mental model

  1. A vocabulary maps token strings ↔ IDs (tens of thousands of entries).
  2. Encode splits text into tokens using learned merge rules (plus special tokens).
  3. The model predicts the next ID; decode turns IDs back into text.
  4. Chat templates wrap roles (system / user / assistant / tool) with special tokens — wrong template ⇒ silent quality loss even with the “right” model.

BPE and friends (intuition)

Byte Pair Encoding (BPE) and cousins (WordPiece, Unigram LM) learn merges so common substrings become single tokens:

  • Frequent words/pieces → one token (cheap, stable)
  • Rare words, typos, IDs → many pieces (longer sequences, higher cost)
  • Code and non-English often tokenize poorly on English-heavy vocabs
  • Numbers and UUIDs can explode into many tokens — painful for SKUs and order IDs in RAG

Byte-level BPE (GPT-style) can represent any UTF-8 string via bytes, avoiding “unknown” characters — but multilingual efficiency still varies by training mix.

flowchart TD
  Corpus[Training text] --> Stats[Pair frequency stats]
  Stats --> Merges[Learn merge rules]
  Merges --> Vocab[Vocabulary + merges]
  Vocab --> Encode[Encode product text]
  Encode --> Cost[Token count / cost / context use]

How tokenization shows up in products

Surface What tokens control
Context window How much prompt + history + tools + RAG fit
Billing Prompt vs completion token meters
Latency More prompt tokens → longer prefill (TTFT)
Chunking RAG chunk size is often token-budgeted, not characters
Structured output Truncation mid-JSON is a token-budget bug
Multimodal Image patches / audio frames also consume “token-like” budget on many APIs

Step-by-step: what happens on one chat request

  1. Client sends messages (+ tool schemas).
  2. Server applies the model’s chat template → one token ID sequence.
  3. Prefill runs over that sequence; decode emits new IDs.
  4. Detokenizer streams text to the client (SSE) — sometimes buffering partial tokens.
  5. Usage counters report prompt_tokens and completion_tokens.

How to work with tokenizers (build literacy)

# Example exploration mindset (library names vary by model)
# Compare counts for the same string under tiktoken vs Hugging Face tokenizer

Practical checklist:

  1. Tokenize production-shaped prompts (system + tools + RAG), not toy sentences.
  2. Compare two tokenizers on the same paragraph — note weird splits on code, Hindi/CJK, URLs.
  3. Measure tokens per chunk after your chunker — not characters ÷ 4 folklore alone.
  4. Pin tokenizer + chat template version next to model version in config.

Tools today (2025–2026)

Tool Use
tiktoken OpenAI-style counting for many hosted models
Hugging Face tokenizers Open-weight models; inspect merges / special tokens
Provider token counters /tokenize or usage fields — source of truth for billing
vLLM / SGLang / Ollama Serving stacks that must match the model’s tokenizer
LangChain / LlamaIndex callbacks App-level token accounting (verify against provider)

Failure modes

Symptom Likely token cause Fix direction
Hit context “too early” Verbose system + tool schemas Slim tools; summarize history
Bill spike Long RAG dumps; no truncation Pack with token budget; rerank
Broken JSON / code Truncation mid-structure Max tokens + schema repair
Multilingual weirdness Vocab mismatch Better multilingual model/tokenizer
“Model got dumber” after swap Wrong chat template Match model card template
Agent loops blow context Huge tool observations Truncate/summarize tool results

Tradeoffs

  • Larger vocab — shorter sequences for covered languages; bigger embedding tables.
  • Aggressive compression — fewer tokens but worse rare-word behavior.
  • Character heuristics (“4 chars ≈ 1 token”) — fine for rough capacity planning, wrong for billing and non-English.

When to go deeper

Glossary

Term Meaning
Token Atomic model input/output unit (subword / byte piece / special)
BPE Merge-based subword tokenization
Special token Non-text markers for roles, EOS, tools
Chat template Serialization of messages → token IDs
Context window Max tokens model can attend over in one call

Micro-project

Tokenize the same paragraph with two tokenizers. Record: total counts, worst split (e.g. email, JSON key, non-English word), and estimated cost at a published $/1M rate.

Tokenization and BPE — train/inspect a tiny tokenizer in your portfolio. Browse is the concept; guided is the build. Also pair with Self-attention (why sequence length is expensive) and Embeddings (chunk token budgets for RAG).

Project checklist0/3 done