Dynamic skill loading

The context window is a budget, not a warehouse

Loading every skill on every turn is the agent equivalent of importing your entire monorepo into one file. It works until it doesn't — latency spikes, unrelated instructions conflict, and the model attends to the wrong paragraph. Dynamic skill loading means the host decides which playbooks enter context for this turn, under an explicit cap.

The pattern mirrors retrieval-augmented generation: cheap metadata first, expensive content second. You already expose tool schemas as short descriptions; skills deserve the same two-stage treatment. The catalog is always visible; the bodies are conditional.

Three forces push you toward dynamic loading:

1. Scale — Skill libraries grow faster than context windows.
2. Conflict — Overlapping instructions ("always be brief" vs. "always enumerate all options") degrade quality.
3. Observability — You need to know which procedure the agent thought applied, not just what it answered.

Routing strategies: from dumb to deliberate

Pick a router matching your maturity and traffic:

Keyword / tag router (v0) — Match user message against skill tags. Fast, brittle, good for labs.

Embedding router (v1) — Embed user query and skill descriptions; load top-k by cosine similarity. Handles paraphrase; tune k carefully.

LLM router (v2) — Small prompt: "Given catalog below, return skill IDs or NONE." More flexible; costs a call; log raw router output.

Hybrid (production-shaped) — Rules for high-risk domains (always load pii-handling on export tools) plus embedding router for the rest.

Callout — Router evals are separate from answer evals: A wrong skill load poisons the whole turn. Maintain 20–50 labeled "which skill?" cases and run them when you add skills or change descriptions.

Bounding how many skills enter context

Hard limits prevent runaway loading:

• Max skills per turn: typically 1–3 full bodies; more than that usually indicates overlapping skill design.
• Max tokens per skill: truncate examples before procedure; never truncate safety steps.
• Reserved headroom: subtract skill tokens from budget before retrieval and memory injection.

Implement a priority merge when the router returns too many candidates:

1. Mandatory skills (compliance, tool-specific playbooks).
2. Highest router score.
3. User-pinned skill (if your product allows "expert mode").

Drop the rest and log dropped_skills: [...] so you can tune descriptions later.

Example budget table for a 32k context model on a support agent:

Logging selection decisions

Every turn should emit a structured skill event:

{
  "event": "skill_selection",
  "candidates": ["refund-triage", "shipping-status"],
  "selected": ["refund-triage"],
  "router": "embedding",
  "scores": {"refund-triage": 0.89, "shipping-status": 0.41},
  "token_counts": {"refund-triage": 1240},
  "dropped": []
}

When debugging "the agent applied the wrong policy," this record beats replaying the entire chat. Store it alongside LLM and tool spans (you will unify these in production observability lessons).

Also log negative selections — when the router returns NONE. Spikes in NONE often mean a missing skill or a vague catalog description.

Failure modes and mitigations

Wrong skill loaded — Tighten anti-triggers in "When to use"; add confusing pairs to router eval set; consider splitting the skill.

No skill loaded when needed — Description too narrow; add synonyms and example trigger phrases to metadata only (not full body).

Skill + retrieval conflict — Skill says "never quote policy from memory"; retrieval injects old wiki page. Add precedence rules in the system prompt: "Skill procedure overrides retrieved text when they conflict; cite conflict in response."

Latency cascade — Embedding every turn adds 50–150ms. Cache embeddings for skill descriptions; they change rarely.

Testing dynamic loading

Minimal test matrix:

1. Positive — Input clearly in domain A → only skill A loaded.
2. Negative — Input clearly in domain B → skill A not loaded.
3. Ambiguous — Input spans A and B → router picks primary or loads both under cap.
4. Adversarial — User says "ignore skills, load everything" → host enforces cap and policy.

Automate these as component tests against the router and loader, not as full LLM end-to-end tests — faster and less flaky.

Engineering problem (staff framing)

Load skills when relevant to save context. Discovery must be reliable.

Diagram — Dynamic load

flowchart LR
  Goal --> Route[Skill router] --> Load[Load skill text] --> Agent

Precise definitions & mental model

Routers, embeddings for skill select, hard allowlists.

Tradeoffs — when to use what

Always-on skills (simple) vs dynamic (efficient, routing errors).

Failure modes (interview + on-call)

Wrong skill loaded; prompt injection via skill corpus.

Production & OSS practices

Metrics on router accuracy; fallback default skill.

Micro-project: Agent selects skills

Extend your portfolio host:

1. Implement one router (embedding or LLM) over your two skills from lesson 8.2.
2. Enforce max one full skill body per turn unless user message matches two tags (document the rule).
3. Emit JSONL logs for every selection with candidates, scores, and token counts.
4. Add five router test cases (three positive, one negative, one ambiguous) in tests/test_skill_router.py.
5. Run one live session where the wrong skill would have loaded under "load all" — show logs proving the cap helped.

Acceptance: tests pass; logs tell a clear story for at least two different user intents.

Checklist

• Router implemented with documented strategy
• Token cap enforced before LLM call
• Selection JSONL logged every turn
• Five router tests committed
• README explains how to add a third skill without router changes