Context engineering

Context is a scarce resource

Everything you stuff into a model call — system rules, RAG chunks, tool schemas, chat history, scratchpads — competes for a finite context window and for attention. Longer is not automatically smarter: cost and latency rise, and models can get distracted by irrelevant tokens (“lost in the middle”).

Context engineering is deliberate allocation, packing, and eviction of those tokens so the model sees the right evidence at the right time. It sits between prompt poetry and full agent architecture.

pie title Typical agent context budget
  "System + policies" : 15
  "Tool schemas" : 20
  "Retrieved evidence" : 35
  "Conversation memory" : 20
  "Scratch / plan" : 10

Ship rule: treat the context window like RAM in an OS — processes (sections) get budgets; the OOM killer is your truncator.

What goes in the window

Section	Job	Common failure
System / policy	Non-negotiable rules, persona, safety	Too long → crowds out evidence
Tool schemas	JSON/function specs	40 tools → thrash and wrong calls
Retrieved evidence	RAG / search hits	Low-precision dumps
Working memory	Recent turns, plan, TODOs	Unbounded history
Scratch	CoT, critiques	Leaks or burns tokens

flowchart TD
  In[User turn] --> Alloc[Budget allocator]
  Alloc --> Sys[System block]
  Alloc --> Tools[Tool subset]
  Alloc --> Ret[Retrieve + rerank]
  Alloc --> Mem[Summarize / select history]
  Sys --> Pack[Pack prompt]
  Tools --> Pack
  Ret --> Pack
  Mem --> Pack
  Pack --> LLM[Model call]
  LLM --> Out[Answer / tool calls]
  Out --> Store[External state: DB / Redis / files]

Core techniques

Hard budgets per section

Assign token caps (e.g. evidence ≤ 3k tokens). Drop lowest-score chunks first. Never “append until full” without ranking.

Summarize older turns

Keep the last (n) turns verbatim; compress the tail into a structured summary (entities, decisions, open questions). Re-summarize on a schedule, not every token.

Tool result stores

Large tool payloads (HTML, CSV, PDF text) do not belong inline forever. Write to Redis/S3/DB; pass ids + short excerpts back into context. See Redis for AI caching.

Skill / MCP loading

Load only the skill packs and MCP tool descriptors relevant to the intent — guided Skills, MCP, context engineering and Model Context Protocol.

Citations over dumps

Prefer precise spans with doc_id + offsets over pasting entire pages. Improves auditability and reduces distraction. Pair with RAG building blocks and chunking and metadata.

Structured packing order

A stable order helps models and debuggers:

1. System policy
2. Active tools
3. Retrieved evidence (ranked)
4. Memory summary
5. Recent dialogue
6. User message

Failure mode: “just use 128k / 1M”

Long-context models delay the packing problem; they do not erase:

Cost	Effect
$	Prefill charges scale with tokens
Latency	TTFT grows with prompt size
Quality	Irrelevant tokens still compete
Privacy	More retained PII in prompts/logs

Measure task success vs context size. Mid-size, well-ranked context often wins.

flowchart LR
  Q[Query] --> Rerank[Retrieve + rerank]
  Rerank --> Small[Top evidence only]
  Small --> Win[Higher success / $]
  Q --> Dump[Stuff 100 chunks]
  Dump --> Lose[Cost ↑ · focus ↓]

Architecture patterns that work

1. Two-stage retrieve — cheap recall → cross-encoder / LLM rerank → pack top-(k) (hybrid search and rerankers)
2. Agent with filesystem — write plans/artifacts outside the window; read slices on demand
3. Router to specialists — different agents with different tool subsets (multi-agent orchestration)
4. Prompt cache / prefix reuse — keep stable system prefixes hot when the provider supports it (KV cache)

Instrumentation

Log per turn:

• Tokens per section
• Which chunk ids were packed vs dropped
• Tool schema count
• Outcome (success, repair loop, human escalate)

Without these counters, “context engineering” stays vibes.

Checklist

1. Written budgets per section?
2. Eviction policy for history and RAG?
3. Large tool outputs externalized?
4. Tool/skill subsetting by intent?
5. Eval that varies context size on the same tasks?

Related guided path

Skills, MCP, context engineering. Core: prompt engineering, RAG building blocks; Advanced: reasoning (scratchpads compete for the same budget).