Caching and latency

Latency budgets die in tool fan-out and cold prompts

Agent p95 latency is often dominated by: sequential tool calls, cold system+skill prompts resent every turn, retrieval embedding round-trips, and provider queue time. Caching attacks the repeatable parts — identical prompt prefixes, identical retrieval queries, identical tool results within TTL.

Caching is not free: stale policy answers, wrong-user data leaks, and non-deterministic model outputs complicate result caches.

Layers to cache

Layer	Key	TTL guidance
Prompt prefix	hash(system+skills+tools schema)	until deploy
Embedding	hash(query text)	hours–days
Retrieval results	hash(query+corpus_version)	minutes–hours
Tool read results	hash(tool+args)	seconds–minutes
Full LLM response	hash(all inputs)	risky — see below

Provider-native prompt caching (Anthropic/OpenAI prefix caching) reduces cost and latency for static prefixes — measure before building custom.

Measuring p50/p95

Benchmark script:

1. Warm vs cold: 20 requests, same session vs new session.
2. Record TTFB (first SSE token) and total time.
3. Report p50/p95 with and without cache enabled.

Store benchmarks/latency_YYYYMMDD.json. One graph in README beats adjectives.

Callout — Agent latency is path-dependent: Router misroute to heavy worker hurts p95 more than mean. Tag benchmarks by route.

Correctness risks

Never cache without scoping:

• User-specific tool results (get_account) — key must include user_id.
• Mutating tool outputs — generally do not cache.
• LLM full responses when tools or retrieval must be fresh — stale answer cache is silent bug.

Use corpus_version in retrieval cache keys — reindex busts cache automatically.

Document cache_policy.md: what is cached, key shape, TTL, bust conditions.

Negative caching

Cache "document not found" for short TTL to prevent retrieval storms on typo queries — but TTL must be short to avoid delaying new doc availability.

Interaction with streaming

Cached prefix may skip prefill latency — first token faster. Log cache_hit: true on spans for observability.

Cache stampede protection

When hot cache key expires, thundering herd hits origin — all requests miss simultaneously. Use single-flight (one recomputation per key) or stale-while-revalidate: serve stale value while background refresh. Especially important for embedding cache on popular queries.

Semantic vs exact cache

Exact hash cache hits only identical prompts — rare in chat. Semantic cache (embedding similarity above threshold) increases hit rate but risks wrong answer on near-duplicate questions with different intent. If experimenting with semantic cache, require high threshold + same user scope + log near-miss review sample.

Latency SLO budgeting

Decompose p95 budget: 200ms gateway, 1500ms LLM TTFT, 2000ms tools, 500ms overhead — if tool budget blown, caching tool reads first before upgrading model. Publish internal SLO table; regressions become actionable tickets not vague "slow today."

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Caches cut cost/latency: exact, semantic, KV/prefix. Incorrect cache ⇒ wrong answers.

Diagram — Cache layers

flowchart TD
  Req --> Exact[Exact cache]
  Exact -->|miss| Prefix[Prompt prefix / KV]
  Prefix -->|miss| Model
  Model --> Store[Fill caches]

Precise definitions & mental model

Idempotent keys, TTL, semantic cache risk, provider prompt caching.

Tradeoffs — when to use what

Hit rate vs staleness/safety.

Failure modes (interview + on-call)

Caching personalized answers globally; huge keys.

Production & OSS practices

Cache metrics; poison controls; key includes model+prompt SHA.

Micro-project: Prompt cache experiment

Ship:

1. Implement one cache layer (prefix embedding or tool read — pick one).
2. Benchmark p50/p95 before/after on fixed scenario (≥20 runs).
3. cache_policy.md with correctness rules.
4. Demonstrate safe bust (corpus_version bump or TTL expiry).
5. Log cache hit/miss on cost or trace records.

Acceptance: benchmark JSON committed; p95 improvement or honest negative result documented.

Checklist

• One cache layer implemented with explicit key schema
• p50/p95 benchmark before/after
• cache_policy.md covers scoping and TTL
• cache hit/miss logged per request
• Correctness bust scenario demonstrated