Chunking and metadata

Chunking is a product decision

Embeddings see chunks, not whole PDFs. Too large → diluted similarity (many topics in one vector). Too small → missing context (answer split across neighbors). Overlap, headings, tables, and code fences change retrieval quality more than swapping Pinecone for Weaviate on a small corpus.

If you only remember one line from the Data track: fix chunking before you rewrite the stack.

flowchart TD
  Doc[Document] --> Clean[Clean / parse]
  Clean --> Split[Chunker strategy]
  Split --> Meta[Attach metadata + stable id]
  Meta --> Emb[Embed]
  Emb --> VDB[(Vector DB)]
  Q[Query] --> Filter[Metadata filters]
  Filter --> VDB
  VDB --> Pack[Pack adjacent / parent context]

One-sentence definition

Chunking is the process of cutting source documents into retrieval units sized for embedding quality, then attaching metadata so filters, citations, and packing can reconstruct useful LLM context.

What / why / how (staff three-liner)

Why chunking dominates vector DB choice (early)

You still need a solid vector database story — but many “RAG is broken” incidents are parse + chunk bugs. Vendor migrations do not fix nav-footer embeddings or tables split without headers.

Interview cue

If someone asks “Pinecone or Weaviate?”, answer with corpus shape, tenancy, and chunk eval results first. Brand second.

Strategies that show up in production

# Conceptual fixed chunker with overlap
def chunk_tokens(tokens: list[str], size=400, overlap=80):
    i = 0
    while i < len(tokens):
        yield tokens[i : i + size]
        i += max(1, size - overlap)

Overlap helps when the answer straddles a boundary; too much overlap wastes money and duplicates hits.

Choosing a default (practical)

1. Start structure-aware if you have Markdown/HTML headings
2. Else recursive with ~400–600 token target and ~10–15% overlap
3. Adopt parent–child when evals show precise hits but thin LLM context
4. Only then try semantic splitters if narrative docs dominate

Strategy decision tree (quick)

flowchart TD
  Start[Have headings / DOM?] -->|Yes| Struct[Structure-aware]
  Start -->|No| Rec[Recursive ~500 tok]
  Struct --> Eval{recall@5 OK?}
  Rec --> Eval
  Eval -->|Thin LLM context| PC[Add parent-child]
  Eval -->|Miss mid-boundary answers| Ov[Raise overlap 10-20%]
  Eval -->|Narrative digressions| Sem[Trial semantic split]
  Eval -->|Good| Ship[Version config + ship]

Tables, code, and PDFs

• Tables: Prefer row- or section-level chunks with the header repeated in each chunk; otherwise embeddings see gibberish columns.
• Code: Keep functions / classes intact when possible; include file path in metadata.
• PDFs: Layout extraction beats pdftotext for multi-column docs; bad parse → unrecoverable retrieval.
• Boilerplate: Strip nav/chrome before chunking or you retrieve the footer forever.
• Images / scans: OCR quality is part of chunking; garbage OCR → garbage vectors.
• Slide decks: One slide ≈ one chunk often works; include slide title in text.
• Spreadsheets: Sheet name + column headers + row window; never dump raw CSV as one vector.
• Email / tickets: Thread subject + message boundaries; speaker/role tags in text or metadata.

flowchart LR
  PDF[PDF] --> Layout[Layout-aware parse]
  Layout --> Clean[Strip chrome]
  Clean --> Struct[Detect headings/tables]
  Struct --> Chunk[Strategy-specific splits]

PDF failure gallery (recognize these)

Metadata that pays rent

{
  "text": "Refunds are issued within 5–7 business days…",
  "doc_id": "policy-42",
  "tenant_id": "acme",
  "heading": "Returns / Refunds",
  "updated_at": "2026-03-01T00:00:00Z",
  "acl_roles": ["user", "support"],
  "content_hash": "sha256:…",
  "chunk_index": 3,
  "parent_id": "policy-42#returns",
  "chunker_version": "struct-v3",
  "meta_schema_version": 2
}

Ship rule: if you cannot cite it, do not retrieve it. Metadata must carry stable ids through fusion and hybrid search / rerank.

Metadata that should not be embedded as free text

Do not paste secrets, raw session tokens, or full PII dumps into the embedded text field “for convenience.” Prefer structured payload fields that never hit the embed API when possible, and minimize what third-party embedders see. Companion: Privacy and data for AI.

Stable IDs and re-chunking

Re-chunking without stable ids → duplicate vectors and ghost citations.

When the splitter changes, delete all chunks for doc_id then re-ingest — do not hope indexes self-heal.

Idempotent ingest sketch

def upsert_doc(doc_id: str, chunks: list[Chunk], store):
    store.delete_by_filter({"doc_id": doc_id})
    for c in chunks:
        store.upsert(
            id=c.stable_id,  # e.g. f"{doc_id}:{c.content_hash}"
            vector=embed(c.text),
            payload=c.metadata | {"doc_id": doc_id, "text": c.text},
        )

Race note: concurrent workers on the same doc_id need a lease or “last writer wins” with a doc-level version — otherwise two deletes interleaved with upserts leave holes.

Packing context after retrieval

Top-k child chunks are not always what you paste into the LLM:

1. Retrieve children (small, precise)
2. Expand to parent section or ±1 neighbor chunk
3. Deduplicate overlapping windows
4. Enforce token budget; prefer diversity across doc_id
5. Keep citation ids aligned with expanded text

flowchart LR
  Hits[Top-k child hits] --> Expand[Expand to parent/neighbors]
  Expand --> Dedupe[Dedupe + budget]
  Dedupe --> Cite[Keep citation ids]
  Cite --> LLM[Prompt]

Packing budget sketch

def pack(hits, budget_tokens=3000):
    used = 0
    out = []
    seen_docs = set()
    for h in hits:  # already reranked
        parent = expand(h)
        if parent.tokens + used > budget_tokens:
            continue
        if h.doc_id in seen_docs and len(out) >= 3:
            continue  # encourage diversity after a few
        out.append(parent)
        seen_docs.add(h.doc_id)
        used += parent.tokens
    return out

Packing anti-patterns

Eval: ablate like an engineer

Hold out ~20–50 docs and 30+ questions with labeled relevant chunk ids (or answer spans).

Report recall@5, MRR, and a quick answer faithfulness sample. Winner is corpus-specific — that is the point.

flowchart TD
  Corpus[Frozen corpus + labels] --> A[Chunk config A]
  Corpus --> B[Chunk config B]
  A --> Emb[Same embedder]
  B --> Emb
  Emb --> R[recall@5 / nDCG]
  R --> Pick[Pick default + document why]

Eval harness checklist

• Frozen corpus snapshot (immutable object-store prefix)
• Question → gold chunk_id or answer span labels
• Same embedder + same ANN knobs across ablations
• Log config hash next to metrics
• Spot-check faithfulness on 20 answers (human or LLM-judge with care)
• Gate: do not ship a chunker change that drops recall@5 below your floor

Three measurable metrics (chunking staff bar)

1. recall@5 across chunk-size / structure ablation
2. % chunks that are boilerplate / nav (should be ~0 on monthly sample)
3. Duplicate-id rate after re-ingest (should be 0)

Add a fourth when multi-tenant: cross-tenant leak rate = 0 in isolation tests (filters on packed hits).

Interface contract (staff bar)

Two degrade modes

1. Structure parser fails → fall back to recursive splitter; flag docs for reparse
2. Retrieval empty after strict filters → broaden doc_type / freshness before letting LLM invent

Threat note

Chunks that include another tenant’s pasted content, or PII in metadata fields logged to third parties, create leaks. Catch with ACL on every chunk, PII minimization before embed, and cite-only packing. Prompt injection via retrieved docs is a packing + policy problem — sanitize and cite; do not trust retrieved HTML/scripts as instructions.

Failure modes

• One chunk = whole 40-page PDF
• Chunking after HTML chrome → nav embeddings
• No tenant_id / ACL on payload
• Re-chunking without stable ids → duplicate vectors
• Evaluating only chatbot vibes → blind to retrieval rot
• Tables split without headers → unreadable hits
• Parent expansion blowing past context budget → truncated mid-policy
• Multilingual docs chunked with English-only separators
• Mixed embedding models in one collection after “quick” re-embed
• Parser upgrade without delete-by-doc_id → ghost old chunks
• Semantic splitter thrashing on repetitive legalese
• Storing embeddings without chunk text → cannot cite or rebuild prompts

Debugging playbook

1. Dump top-5 chunk texts for a failing query — readable to a human?
2. Check parse output before chunking — chrome? columns scrambled?
3. Ablate one size step larger/smaller — recall move?
4. Verify metadata filters not over-narrowing
5. Confirm re-ingest deleted old doc_id chunks
6. Compare child hit vs packed parent — did expansion help or drown the answer?
7. Diff chunker_version / parser_version between staging and prod

flowchart TD
  Fail[Bad answer ticket] --> Dump[Dump top-k texts]
  Dump --> Parse{Parse clean?}
  Parse -->|No| FixParse[Fix parser / strip]
  Parse -->|Yes| Size{Ablate size}
  Size --> Filters{Filters too tight?}
  Filters --> Ids{Ghost ids?}
  Ids --> Pack{Packing budget?}

Production readiness checklist

• Chunker config versioned and stored with index / collection name
• Stable ids + delete-by-doc on refresh
• Metadata schema documented (esp. ACL)
• Ablation results attached to design doc
• Special handling for tables/code/PDF noted
• Packing + citation path tested
• Boilerplate strip verified on HTML sources
• Reparse triggers documented (parser / chunker / strip rules)
• Monthly random chunk sample for chrome regression
• Isolation tests for tenant filters on packed context

Interview prompts

1. Why might smaller chunks raise recall but hurt answer quality?
2. Explain parent–child retrieval.
3. What metadata is mandatory for multi-tenant RAG?
4. How do you evaluate a chunker change safely?
5. Walk through a safe chunker-version upgrade without ghost vectors.
6. How do you chunk tables so embeddings stay meaningful?
7. What goes in the vector payload vs Postgres system of record?

Chunk size intuition (rules of thumb, not laws)

Always ablate — these are ignition points for experiments.

Overlap math

For size (S) and overlap (O), stride = (S - O). Storage ≈ (S / (S - O)) times a no-overlap corpus. At (S=500), (O=100) you store ~1.25× chunks — usually acceptable; at (O=250) you pay 2×.

Also pay for:

• Embed API cost × storage multiplier
• ANN RAM / disk
• Duplicate hits at query time (packing must dedupe)

Ship rule: pick the smallest overlap that recovers boundary-straddle questions on your eval set.

Citation UX requirements

Downstream UI needs:

• doc_id, title, URL/page
• Optional highlight span offsets if you store them
• Version/timestamp for “as of” honesty

If packing expands to a parent, cite the parent id the user can open — not an orphan child index.

Citation integrity tests

Multilingual and Hinglish corpora

• Do not assume English sentence splitters
• Keep language metadata; filter or route embedders if you use per-language models
• Mixed Hinglish often works with strong multilingual embedders — verify with a labeled slice
• Digit / transliteration variants (“refund” vs “रिफंड”) may need hybrid keyword assist

Reparse triggers

Re-chunk when any of these change:

• Parser (HTML/PDF) version
• Chunker config version
• Boilerplate strip rules
• Embedding model (always re-embed; usually re-chunk only if splitter also changes)

Safe upgrade runbook

1. Freeze eval set; record baseline recall@5 / faithfulness sample
2. Build new chunks under a new collection or dual-write side index (…_chunkerv4)
3. Shadow-query production traffic (or replay logs) against both
4. Cut over app config to new version when gates pass
5. Keep old collection read-only for N days; then delete

Never “edit in place” with a new splitter and hope upserts overwrite every old id.

Parent–child storage sketch

parents:  { parent_id, text, doc_id, tokens, ... }
children: { child_id, parent_id, text, embedding, ... }

Retrieve children via ANN; hydrate parents by parent_id for the prompt. Deduplicate multiple children that map to the same parent before packing.

When parent–child is worth the glue

HTML chrome checklist

Before chunking HTML:

• Remove nav, footer, header, cookie banners
• Drop script/style
• Prefer main / article node
• Keep heading hierarchy
• Normalize whitespace
• Resolve relative links if you store url for citations
• Drop “related articles” side rails that pollute retrieval

Sample 20 random chunks monthly — if you see menus, your stripper regressed.

Worked ablation table (example)

Replace numbers with yours. The point is the process, not these fake scores.

Worked narrative (support policy corpus)

1. Corpus: 40 help-center articles + 5 PDF policies
2. Baseline: fixed 1000-token chunks → recall@5 = 0.74; answers vague
3. Heading-aware ~500: recall@5 = 0.88; refund timing questions land on the right section
4. Parent–child: children ~200 under section parents → faithfulness sample up; storage +30%
5. Ship: chunker_version=struct-v3, collection suffix _structv3_te3small

Document that story in the design doc so the next engineer does not “optimize” back to 1000-token fixed splits.

Metadata schema versioning

When you add a required field (e.g. language):

1. Bump meta_schema_vN in ingest config
2. Backfill old payloads or re-ingest
3. Reject query filters that assume fields missing on old points

Store schema version in the collection name or a side table.

Schema evolution examples

From chunker to vector upsert (contract)

@dataclass
class Chunk:
    stable_id: str
    doc_id: str
    tenant_id: str
    text: str
    metadata: dict
    content_hash: str

# chunker outputs Chunk; embedder adds vector; store upserts id+vector+payload

Keep the chunker pure (no I/O). Easier to ablate and unit test.

Suggested module boundaries

If chunk talks to the network, your ablations become flaky.

Observability for the chunking plane

Log at ingest:

doc_id, tenant_id, parser_version, chunker_version,
n_chunks, n_empty_dropped, n_boilerplate_dropped,
token_total, content_hash, duration_ms

Log at query / pack:

query_id, hit_child_ids[], packed_ids[], pack_tokens,
dropped_for_budget, diversity_skips, filter_hash

Alert on:

• Spike in empty or boilerplate-dropped ratio (parser regress)
• Ingest lag (docs updated but not re-chunked)
• Duplicate-id rate > 0 after refresh jobs
• Pack budget exhaustion rate (always truncating → answers suffer)

Cost model (back of envelope)

A “free” 20% overlap is not free. Measure dollars per recall point gained.

Consistency and freshness

Freshness SLOs belong next to query latency SLOs — stale policy chunks cause confident wrong answers.

Security deep dive (chunk payloads)

Anti-patterns (chunking edition)

• One vector per entire PDF “to keep it simple”
• Chunking before stripping chrome
• Evaluating only final chatbot answers
• Re-chunk upsert without delete-by-doc_id
• Mixing structure-aware and fixed splits in one collection without chunker_version
• Parent expansion without token budget
• Storing only vectors (no text) so citations are impossible
• English-only regex on Indic corpora
• Letting product managers set chunk size from a blog post without an ablation

Design-doc template (paste into your RFC)

1. Corpus types + volumes + growth
2. Parser choice + chrome strip rules
3. Default chunk strategy + ablation table
4. Metadata schema + ACL fields
5. Stable id scheme + re-ingest rule
6. Packing / citation behavior
7. Eval set location + gates (recall@5 floor)
8. Reparse / dual-write runbook
9. Cost sketch (embed + storage)
10. Open risks (tables, PDFs, multilingual)

If any bullet is missing, the design is not ready for production review.

Structure-aware chunking (implementation sketch)

For Markdown / HTML docs sites, prefer an AST walk over blind token windows:

def chunk_by_headings(blocks, max_tokens=600):
    """blocks: list of {heading_path, text} from a Markdown/HTML parse."""
    for b in blocks:
        toks = tokenize(b["text"])
        if len(toks) <= max_tokens:
            yield make_chunk(b["heading_path"], b["text"])
            continue
        # Oversized section: recursive split, keep heading on every piece
        for piece in recursive_split(toks, max_tokens):
            yield make_chunk(b["heading_path"], detokenize(piece))

Heading path in metadata (e.g. Returns > Refunds > Timing) helps rerankers and UI breadcrumbs even when the embedding only sees section body.

Separator priority (recursive baseline)

Typical order when structure is weak:

1. \n\n\n / thematic breaks
2. \n\n (paragraphs)
3. \n
4. . / ? / ! (sentence)
5. (last resort — avoid when possible)

Never split inside fenced code if you can detect fences; never split mid-HTML table row.

Late chunking and contextual embeddings (when available)

Some encoders let you embed a long document once, then derive per-span vectors that still “know” surrounding context (late chunking / contextualized chunk embeddings). That can beat naive independent chunk embeds on long narrative docs.

Ship rule: treat late chunking as an ablation arm, not a rewrite of your metadata/ACL story.

Tokenizers vs “words” (gotcha)

Chunk sizes quoted in tokens depend on the tokenizer (cl100k, embedder-specific, etc.). Mismatching “500 tokens” measured with a chat tokenizer vs the embedder’s tokenizer skews ablations.

Sliding windows on chat and transcripts

Support chats and call transcripts are not docs:

Always tag speaker / role / channel in metadata. Packing should prefer contiguous turns from the same thread when expanding neighbors.

Code repositories as a corpus

Do not embed minified bundles. Do not one-vector an entire src/ tree.

Tables: concrete patterns

# Good: header repeated per row-group chunk
Columns: Order ID | Status | Refund SLA
Row: 1842 | approved | 5-7 business days
Row: 1843 | pending | 5-7 business days

# Bad: split mid-table without headers
| approved | 5-7 business days |
| pending | …

For wide tables, chunk by row groups (10–30 rows) or by primary key entity, not by raw character count across the CSV.

End-to-end sequence: one query

sequenceDiagram
  participant U as User
  participant App as App
  participant Emb as Embedder
  participant VDB as Vector DB
  participant SoT as Postgres/SoT
  participant LLM as LLM
  U->>App: "How long do refunds take?"
  App->>Emb: embed query
  App->>VDB: ANN + tenant/ACL filters
  VDB-->>App: child hits
  App->>SoT: hydrate parents / neighbors
  App->>App: pack + cite under budget
  App->>LLM: prompt with packed spans
  LLM-->>U: answer + citations

Every failure mode in this article maps to a labeled arrow: bad parse before VDB, missing ACL on the VDB call, or pack blowing the budget before LLM.

Scoring beyond recall@5

Optimize the metric that matches the product failure: FAQ bots die on faithfulness; enterprise search dies on recall under ACL.

Capacity planning for chunk counts

Rough planning numbers:

Overlap at 20% adds ~25% more vectors — fold that into the vector database sizing sheet before you “just raise overlap.”

Team workflow (who owns what)

Chunking is not “an LLM prompt tweak.” It is a versioned data pipeline with owners.

Whiteboard prompt (practice)

Design chunking + metadata for a 30-tenant SaaS: help center (Markdown), PDF policies, and Zendesk tickets. Show id scheme, ACL fields, ablation plan, and how you upgrade the splitter without ghost vectors.

A strong answer hits: per-doc_type strategies, tenant_id+roles on every chunk, delete-by-doc re-ingest, recall@5 gate, packing/citations, dual-write on chunker_version bump.

FAQ (chunking)

Should every doc type share one chunk size?
No. Policies, code, and FAQs often need different strategies — route by doc_type.

Is semantic chunking always better?
No. It costs more and can over-split. Beat a strong heading-aware baseline before adopting it.

Where do I store raw PDFs?
Object store. Chunks and embeddings are derived; keep the raw bytes for reparse.

Can long-context models skip chunking?
They reduce how often you retrieve large spans; they do not remove ACL, cost, or citation needs. Chunking still defines what you index and what you can cite.

How do I pick overlap?
Ablate 0 / 10% / 20% on boundary-straddle questions. Stop when recall gains flatten relative to cost.

Do I embed the parent, the child, or both?
Usually embed children for ANN; store parents for packing. Embedding both doubles cost unless evals prove the need.

What if heading-aware splits yield huge sections?
Cap section size and recursively split oversized headings; keep heading metadata on every piece.

Glossary

Hands-on lab sequence

1. Take 20 docs (Markdown + one multi-column PDF).
2. Build fixed 200 / 500 / 1000 and one heading-aware config.
3. Label ≥30 questions with gold chunk ids.
4. Report recall@5 + a 10-answer faithfulness skim.
5. Add parent–child on the winner; re-measure answer quality.
6. Break chrome strip on purpose; watch menus enter top-k.
7. Re-ingest with a new chunker_version using delete-by-doc_id.
8. Write a half-page rationale for the shipped default.

That sequence teaches what / why / how better than reading splitter library docs alone.

Putting it together with the rest of the Data track

Chunking sits before brand debates and beside hybrid/rerank: bad units make every later stage look broken.

Common interview traps

Micro-project

Ablate 200 / 500 / 1000-token chunks on a 20-doc corpus; report recall@5 and pick a default with a one-paragraph rationale. Bonus: add a heading-aware arm and a parent–child packing comparison on the same labels.

Related

Guided Chunking; hybrid + rerank; vector databases; Privacy and data for AI for ACL fields.