Chunking

Chunking is an eval problem disguised as text processing

Whole documents rarely fit retrieval use cases. A 40-page PDF contains many facts; embedding the entire file yields one vague vector that matches everything weakly. Chunking splits documents into retrieval units — paragraphs, fixed token windows, or structure-aware sections — so the index returns precise passages.

The trap is treating chunk size as a magic constant ( "512 tokens" copied from a blog post). Correct chunking depends on:

• How users ask questions (short fact lookup vs synthesis)
• Document structure (Markdown headers vs legal clauses)
• Embedding model context limits
• Downstream LLM context budget for top-k chunks

You choose hypotheses, run ablations, measure recall@k — not vibes.

Callout — overlap exists for boundary cuts: A fact split across two chunks may miss retrieval unless overlap or structure-aware splitting keeps it intact.

Chunking strategies

Start with structure-aware when metadata exists; fall back to fixed windows with 10–20% overlap.

Example fixed window with overlap (tokens):

def chunk_tokens(text: str, size: int = 256, overlap: int = 50) -> list[str]:
    tokens = tokenize(text)  # use same tokenizer family as embed model if possible
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + size
        chunks.append(detokenize(tokens[start:end]))
        start += size - overlap
    return chunks

Attach metadata to every chunk: source_id, chunk_index, heading_path, char_start, char_end.

Recall@k as the metric

For each eval question, you know which chunk(s) contain the answer (gold labels). After retrieval:

Recall@k = fraction of questions where at least one gold chunk appears in top-k results.

This measures retrieval quality independent of LLM generation — isolate the bottleneck.

Build eval/queries.jsonl:

{"id": "q1", "query": "What is the refund window?", "gold_chunk_ids": ["doc42#chunk3"]}

Run retrieval for chunk configs (size=128, overlap=0), (256, 50), (512, 100). Plot recall@5 and recall@10.

Hypothesis-driven ablation

Write hypotheses before running:

• H1: 256-token chunks with 50 overlap beat 512 with zero overlap on FAQ queries because answers are localized.
• H2: Header-based chunks beat fixed windows on internal wiki because questions name sections.

If data rejects H1, document why — maybe your FAQ answers span multiple paragraphs and need larger windows or parent-child chunk linking.

Advanced pattern parent-child indexing: retrieve small chunks, return expanded parent section to the LLM. Optional stretch goal; not required for the micro-project.

Domain tradeoffs

Your TRADEOFFS.md should tie results to your corpus — generic advice is insufficient.

Metadata that improves retrieval without bigger models

Enrich chunk text at index time with lightweight prefixes:

[source: handbook.md] [section: Refunds > International]
Actual chunk body here...

Users rarely ask with those tokens, but embeddings partially absorb structure and hybrid BM25 can match section titles in later lessons. Also store raw metadata fields (section, doc_date, product_line) for filtered search in the vector DB lesson.

For code corpora, prepend filepath: and symbol: lines so "function parseReceipt" queries align lexically even when body is implementation details.

Iterating after ablation

When recall@k plateaus:

1. Inspect miss queries manually — wrong chunk nearby in rank 6–10 suggests rerank (next lessons) not chunk resize.
2. Try parent-child: index small chunks, retrieve parent paragraph for LLM context.
3. Add synthetic FAQ rows that paraphrase common miss queries — cheap boost for demo corpora.

Document each iteration in TRADEOFFS.md with dated entries so you do not re-run abandoned experiments.

Eval query authoring tips

Good eval queries mirror real users:

• Mix question forms: WH questions, imperatives ("show me"), keyword dumps ("refund international wire").
• Include typos if your audience makes them — hybrid search lesson may rescue these.
• Tag expected chunk id after manual lookup — do not guess gold labels while tired.

Aim for 20–30 labeled queries minimum before trusting ablation numbers; smaller sets swing wildly with one miss.

Common failure signatures

• Right doc, wrong chunk: recall@k low but doc id in top results with wrong snippet → tune size/overlap.
• Duplicate chunks: excessive overlap floods top-k → dedupe or reduce overlap.
• Lost context: chunk lacks heading → prepend breadcrumb ## Billing > Refunds\n to each chunk text at index time.

Callout — tokenizer alignment: Chunk by the same tokenizer the embedding model uses when possible; character splits mid-token waste capacity.

Engineering problem (staff framing)

Chunk boundaries dominate recall. Bad chunks = unreachable facts.

Diagram — Chunking strategies

flowchart TD
  Doc --> Fixed[Fixed tokens]
  Doc --> Struct[By headings]
  Doc --> Sem[Semantic splits]
  Fixed --> Idx[Index]
  Struct --> Idx
  Sem --> Idx

Precise definitions & mental model

Overlap, structure-aware splits, parent-child / small-to-big.

Tradeoffs — when to use what

Small chunks ↑recall precision localization; large ↑context coherence.

Failure modes (interview + on-call)

Split mid-table; no overlap; chunk IDs lost for citations.

Production & OSS practices

Store offsets + source IDs; evaluate chunkers with retrieval metrics.

Micro-project: Ablation recall@k

In m5/chunking_ablation/:

1. Take ≥20 labeled query/chunk pairs from your semantic search corpus (or synthetic set).
2. Implement two chunking configs; build separate indexes.
3. Compute recall@5 for each; save results.json with configs and scores.
4. Write TRADEOFFS.md with hypothesis, outcome, and recommendation for your domain.

Checklist

• Gold chunk labels for eval queries
• At least two configs compared fairly (same embed model)
• Results reproducible via script
• Recommendation stated with evidence