Context budgets

Context engineering is resource allocation

"Context engineering" is not a rebranded system prompt — it is allocating a finite token budget across every slice that competes in the model's attention window. When a long agent run fails — truncated history, missed tool result, "forgetting" a skill instruction — the root cause is usually budget exhaustion or wrong priority, not model stupidity.

Treat the context window like RAM on an embedded device: every subsystem has a reservation, a maximum, and a eviction policy when pressure hits.

Typical slices in an production-shaped agent:

If you do not measure each slice, you optimize blindly — usually by shortening the user message, which is the one slice users actually care about.

Measuring tokens accurately enough

Exact tokenization matches the provider's tokenizer (tiktoken for OpenAI-compatible APIs, model-specific for others). For budgeting, consistent approximation beats perfect counts on wrong tokenizer:

1. Use the same library your billing uses when available.
2. Count per slice before assembly, not one blob after.
3. Log both input tokens sent and provider-reported usage to catch wrapper overhead.

Instrument the host at assembly time:

budget = {
  "system": count_tokens(system),
  "skills": sum(count_tokens(s) for s in loaded_skills),
  "tools": count_tokens(serialize_tools(schemas)),
  "memory": count_tokens(memory_block),
  "retrieval": count_tokens(rag_block),
  "history": count_tokens(messages),
}
budget["total_input"] = sum(budget.values())

Persist budget JSON on every LLM call. When someone files "it broke on turn 47," plot totals over turn index — the cliff is usually obvious.

Callout — Tool results dominate: A single badly designed tool returning 8k of pretty-printed JSON can eat more than your skill library. Measure tool outputs separately from schemas.

Diagnosing a failing long run

Reproduce with the saved trajectory. For each turn, chart:

• Total input tokens
• Largest single slice
• Largest single message (often a tool result)

Common signatures:

Linear climb — History never compacts. Fix: rolling summary, turn cap, or archival memory outside context.

Step jumps — Retrieval or skill loads mid-run. Fix: router stability, cache skill loads within session.

Sawtooth — Aggressive trimming drops tool results the model still needs. Fix: pin critical messages, trim oldest non-essential turns first.

Ceiling flatline — Hitting model max; model silently loses early instructions. Fix: hard fail before send, or force compaction pass.

Your micro-project picks one failing trace from your agent and writes a budget autopsy: which slice violated policy, at which turn, and what user-visible symptom appeared.

Trimming policies that preserve correctness

Eviction order (safest last):

1. Old retrieval chunks superseded by newer fetch
2. Dropped skill candidates (never loaded bodies)
3. Oldest tool intermediate results where final answer already synthesized
4. Old user small-talk turns
5. Never first: active skill body, current-turn tool results, safety system rules

Implement pinned messages — tags like pin: true on system, active skill wrapper, and the last N tool exchanges relevant to the open subtask.

Summarization compaction: every K turns, replace turns 1..K-1 with a structured summary block:

<session_summary>
Goal: refund order 8842
Done: looked up order, confirmed eligibility
Open: awaiting user confirmation to issue partial refund
</session_summary>

Summaries must be generated with a checklist (goal, decisions, open items) — free-form "the user asked about stuff" summaries fail.

Set hard stops: if total_input + max_output > context_limit - margin, refuse new retrieval or ask user to start a fresh thread with exported summary.

Budget report as an engineering artifact

A token budget report is a one-page doc + JSON appendix:

• Scenario — Session ID, model, turn count, failure description
• Timeline table — turn → slice breakdown → total
• Violation — Which cap was exceeded (skill cap 3000, actual 4200)
• Remediation — Specific trim rule or cap change
• Verification — Re-run passes under same scenario

This report format transfers to on-call: ops should not need to read Python to understand context pressure.

Connecting budgets to product SLAs

Latency and cost correlate with tokens. Publishing internal SLAs helps:

• P95 input tokens per turn by route
• Max skill tokens per product surface
• Max tool result size (enforce at server)

When PMs ask for "load all documentation skills," you respond with a budget table showing tradeoffs — engineering leadership, not obstruction.

Engineering problem (staff framing)

Context is a scarce cache. Budget tokens across system, skills, retrieval, transcript.

Diagram — Context budget pie

flowchart TD
  Ctx[Context window] --> Sys[System]
  Ctx --> Sk[Skills]
  Ctx --> Rag[Retrieval]
  Ctx --> Hist[History]
  Ctx --> Reserve[Output reserve]

Precise definitions & mental model

Reservation for output; truncation policies; cache-friendly prefixes.

Tradeoffs — when to use what

More retrieval vs more history — measure task success.

Failure modes (interview + on-call)

Overflow silent truncate of instructions; no reserve for tools.

Production & OSS practices

Token accountant in driver; alerts on budget pressure.

Micro-project: Token budget report

In your portfolio:

1. Add budget instrumentation to your agent host (per-slice counts logged every LLM call).
2. Run a deliberately long session (≥15 turns with tools) until quality degrades or you hit limits.
3. Produce budget_report.md with timeline, diagnosis, and chosen trimming policy.
4. Implement at least one trim rule (summary compaction, tool result cap, or history window).
5. Re-run the same scenario; show before/after token chart in README.

Acceptance: report identifies dominant slice; after fix, same scenario completes without silent instruction loss.

Checklist

• Per-slice token logging on every LLM call
• Long-run reproduction trace saved
• Budget report with timeline and remediation
• Trimming policy implemented and documented
• Before/after comparison committed