Redis for AI caching

Why Redis shows up in AI stacks

LLMs are expensive and slow. Many product turns are repeatable: same FAQ, same system prompt + tool schema, same embedding of a hot document. Redis is the default low-latency store for:

1. Exact response cache (hash of messages → completion)
2. Semantic cache (embedding → nearest prior answer)
3. Rate limits / quotas (token buckets per tenant)
4. Short-term agent memory (conversation / scratchpad)
5. Job queues / streams (async ingest, eval workers)
6. Idempotency keys (dedupe chat POSTs)
7. Feature flags / config fingerprints tied to prompt versions
8. Query-embedding cache (identical strings → reuse vectors for a short TTL)

It sits in front of model calls — and sometimes in front of vector search — not as a replacement for vector databases or Postgres.

flowchart LR
  Req[Chat request] --> Exact{Exact key hit?}
  Exact -->|yes| Hit[Return cached]
  Exact -->|no| Sem{Semantic hit?}
  Sem -->|yes + high score| Hit
  Sem -->|no| LLM[Call model / RAG]
  LLM --> Store[Write Redis]
  Store --> Resp[Response]

One-sentence definition

Redis in AI stacks is a hot path store for deduplicating expensive model/retrieval work, enforcing quotas, and holding ephemeral session state — never the compliance system of record.

Why not “just make the model faster”?

Serving optimizations (vLLM, quantization, speculative decoding) help misses. Cache helps repeats. For FAQ bots, support macros, and internal copilots with a long tail of identical turns, hit rate often beats another 20% tokens/sec on the origin. Ship both: a fast origin and a correct cache layer.

Interview cue: Caching is a correctness problem first and a latency problem second. A wrong cached refund policy is worse than a 2s miss.

Mental model: layers in front of the LLM

Think of three caches with different risk profiles:

Always re-check ACL when hydrating chunk text from the system of record. Cached ids are hints, not authority.

flowchart TD
  Gateway[API gateway] --> RL[Rate limit Redis]
  RL --> Exact[Exact cache]
  Exact -->|miss| Sem[Semantic cache optional]
  Sem -->|miss| RagCache[Optional rag id cache]
  RagCache --> ANN[Vector / hybrid search]
  ANN --> LLM[Model]
  LLM --> Write[Write caches]

Ship rule: fail open on Redis for read caches (call the model); fail closed or shed load on rate-limit Redis if abuse is the bigger risk — document which path you chose.

Exact-match caching

Key design:

cache:v1:{tenant}:{model}:{hash(system + messages + tools + temperature + response_format + prompt_ver + corpus_ver)}

import hashlib, json, redis

r = redis.Redis()

def cache_key(model: str, tenant: str, payload: dict, versions: dict) -> str:
    blob = json.dumps(
        {"payload": payload, "versions": versions},
        sort_keys=True,
        separators=(",", ":"),
    )
    h = hashlib.sha256(blob.encode()).hexdigest()
    return f"cache:v1:{tenant}:{model}:{h}"

def get_or_generate(model, tenant, payload, versions, generate):
    key = cache_key(model, tenant, payload, versions)
    hit = r.get(key)
    if hit:
        return json.loads(hit), True
    out = generate(payload)  # includes usage
    r.setex(key, 3600, json.dumps(out))
    return out, False

Rules that prevent silent wrong answers:

• Include model id, temperature, tool schemas, and response_format in the hash
• Include prompt/template version and retrieval corpus version when answers depend on them
• Include tenant_id (and locale / policy version when relevant)
• TTL by product (FAQ: hours; personalized: minutes or never)
• Store token usage for cost accounting
• Invalidate when prompts, tools, or retrieval corpora change — bump prefix (cache:v2:)
• Never cache across tenants

Canonicalization pitfalls

Two requests that feel identical often hash differently:

Log the canonical blob hash (or truncated key) on every turn so support can reproduce hits/misses.

What to store in the value

Prefer a small envelope:

{
  "completion": { "role": "assistant", "content": "..." },
  "usage": { "prompt_tokens": 120, "completion_tokens": 80 },
  "model": "…",
  "cached_at": "2026-08-12T06:00:00Z",
  "cache_version": "v3",
  "fingerprint": "abc…"
}

Do not store secrets, raw tool credentials, or full upstream provider debug dumps.

Singleflight / stampede control

When a popular key misses, many workers may stampede the LLM:

SET lock:cache:{hash} NX EX 30
→ winner generates; losers wait or short-poll

Or use request coalescing in the app (in-process + Redis). Stampede protection matters more than micro-optimizing TTL.

sequenceDiagram
  participant A as Worker A
  participant B as Worker B
  participant R as Redis
  participant M as Model
  A->>R: GET cache
  R-->>A: miss
  B->>R: GET cache
  R-->>B: miss
  A->>R: SET NX lock
  R-->>A: OK
  B->>R: SET NX lock
  R-->>B: fail
  A->>M: generate
  M-->>A: answer
  A->>R: SET cache + DEL lock
  B->>R: GET cache (retry)
  R-->>B: hit

Losers should jitter sleep (50–200 ms) and cap wait time; on timeout, either call the model (accept duplicate cost) or return 503 with Retry-After.

Semantic caching

Embed the user query (or full turn). Lookup nearest neighbor in a Redis vector index (Redis Query Engine / RedisVL). Accept hit only if:

1. Similarity ≥ threshold (tune on evals; start conservative, e.g. 0.95 cosine for FAQs)
2. Metadata matches: tenant_id, locale, policy_version, model

flowchart TD
  Q[User text] --> E[Embed]
  E --> ANN[Redis vector search]
  ANN --> T{score >= thr AND meta OK?}
  T -->|yes| Return[Cached answer]
  T -->|no| Miss[Full LLM path]

Risks:

• Near-miss answers that look “close” but differ on price, ID, or entitlement
• Poisoning if you cache unvalidated model output
• Drift when the underlying knowledge base updates but cache keys do not
• Threshold tuned on English FAQs failing on other locales
• Embedding-model upgrade changing neighborhoods without re-index

Ship rule: semantic cache is for stable knowledge domains; disable for money movement, medical, and PII-heavy turns. Prefer exact cache + good RAG over aggressive semantic hits.

Semantic vs vector DB — do not confuse them

Threshold engineering

Treat threshold like a model hyperparameter:

1. Freeze a paraphrase set with gold answers (include adversarial near-misses: “refund in 7 days” vs “refund in 30 days”).
2. Sweep 0.90–0.99; plot accept rate vs critical-error rate.
3. Pick the highest accept rate under your error budget (often near-zero for policy/money).
4. Re-run after every corpus, prompt, or embed-model bump.
5. Feature-flag per route — FAQ on, checkout assistant off.

Poisoning and write policy

Only write semantic entries when:

• Response passed policy / groundedness checks (or is from a curated FAQ pack)
• Route is marked cacheable
• Tenant isolation fields are present
• You are willing to serve this answer for the TTL without re-generation

Never write speculative or tool-failed turns into the semantic index.

Rate limits and budgets

INCR tenant:{id}:tokens_minute
EXPIRE tenant:{id}:tokens_minute 60

Or Lua / Redis Cell–style token buckets. Separate:

Return 429 with Retry-After. Pair with cost/latency routing and Networking for AI apps.

flowchart TD
  Req[Request] --> RL{Under quota?}
  RL -->|no| Reject[429 Retry-After]
  RL -->|yes| Cache{Cache hit?}
  Cache -->|yes| Hit[Cached response]
  Cache -->|no| LLM[Model / RAG]

Soft vs hard limits

On Redis outage for rate limits: decide explicitly — fail open (cost spike risk) vs fail closed (availability risk). Most consumer apps fail open with a process-local emergency cap; most billed APIs fail closed.

Session and agent scratchpads

Use hashes/lists/streams for short-lived state:

• Conversation turns (trim to max tokens / turns)
• Tool results by id (keep prompts small)
• Distributed locks so one worker owns an agent run
• Prefill/prefix fingerprints if you coordinate with serving-side prefix cache
• “Pending tool call” markers for crash recovery of a single run

agent:{run_id}:scratch → hash of tool_id → JSON blob
session:{user_id}:turns → LIST (LPUSH + LTRIM)
lock:agent:{run_id} → SET NX EX 120

Do not treat Redis as durable system of record — snapshot important state to Postgres.

Session trim rules

Agent memory that must survive process death and audits belongs in Postgres / object storage; Redis holds the hot window.

Idempotency keys

SETNX idemp:{tenant}:{key} 1 EX 86400

Chat POST retries should not double-charge or double-send side effects. Store the first response body under the same key (or a sibling idemp:{tenant}:{key}:body).

Idempotency ≠ caching: keys are client-supplied (or derived from a business id), TTLs are long (hours–days), and hits return the original response even if prompts changed mid-flight for that request id.

Caching the retrieval path (optional)

Hot documents or hot query→chunk-id lists can be cached to skip ANN:

rag:v2:{tenant}:{hash(query)} → JSON[chunk_ids]

Invalidate on corpus version bumps. Still enforce ACL when hydrating chunk text from Postgres — never trust cached ids without re-checking auth.

flowchart LR
  Q[Query] --> C{rag cache?}
  C -->|hit| IDs[chunk ids]
  C -->|miss| ANN[Vector / hybrid search]
  ANN --> IDs
  IDs --> ACL[Hydrate + ACL check]
  ACL --> Pack[Pack prompt]

Query embedding cache

Identical strings waste embed API money:

emb:v1:{model}:{sha256(text)} → float[] or binary

Short TTL (minutes) is usually enough. Do not share across embedding models or prefixes (query: vs document:).

Latency and cost math

If p50 model call is 2s / $0.01 and exact cache hit rate is 30% on a hot FAQ bot, you cut both roughly by that fraction — often the highest-ROI “model optimization,” ahead of speculative decoding.

Log:

Worked example

10M turns/month, 25% exact hit, $0.008/miss:

• Without cache: (10M \times 0.008 = $80k)
• With cache: (7.5M \times 0.008 = $60k) → $20k/month saved before latency gains

Semantic cache adding another 10% accepted hits is tempting — only ship it if evals show no critical near-miss errors.

Second worked example (latency)

p50 miss = 1800 ms, p50 hit = 40 ms, hit rate = 35%:

[ p50_{blend} \approx 0.35 \times 40 + 0.65 \times 1800 \approx 1184\ \text{ms} ]

Exec slides should show blend latency and $/turn, not only “cache is 40 ms.”

Invalidation strategies

Prefer version prefixes over surgical key deletes when blast radius is unclear.

Config fingerprint pattern

fingerprint = sha256(prompt_pack + tool_schemas + corpus_ver + policy_ver)
key = cache:v3:{tenant}:{model}:{fingerprint}:{request_hash}

On deploy, fingerprint changes → natural misses. Optionally schedule UNLINK of the old prefix under memory pressure.

Architecture that survives production

A durable design usually separates:

flowchart TB
  subgraph Edge
    Exact[Exact Redis]
    RL[Rate-limit Redis]
  end
  subgraph Data
    PG[(Postgres SoT)]
    VDB[(Vector DB)]
  end
  Client --> RL --> Exact
  Exact -->|miss| VDB
  VDB --> PG
  Exact -->|miss| LLM[Model]

Anti-pattern: one Redis DB with allkeys-lru holding both FAQ answers and rate-limit counters — eviction silently disables abuse protection.

Ops concerns product engineers hit

Redis deployment shapes

For AI caches, managed + failover bypass path beats heroic self-hosting until you have a dedicated platform team.

Interface contract (staff bar)

Three measurable metrics

1. Exact hit rate (overall + per route)
2. Semantic accept rate and stale/wrong-answer ticket rate
3. p95 end-to-end latency with vs without cache (and $/turn)

Two degrade modes

1. Redis down → bypass cache, call model; shed load with stricter rate limits
2. Semantic false-positive spike → feature-flag semantic cache off; keep exact only

Threat note

Caching a privileged answer under a key missing tenant_id or ACL version leaks data on the next similar query. Catch with tenant in every key, no global semantic pool, and security tests. See Privacy and data for AI.

Failure modes

• Hash omitting tools/temperature → wrong cached completion
• Semantic threshold too low → confident wrong FAQ
• No version prefix after prompt change → serving yesterday’s policy
• Caching personalized or entitlement-gated answers globally
• Redis as only store for compliance-critical transcripts
• Unbounded session lists → OOM
• Rate-limit keys without TTL → permanent blocks
• Stampede without lock → cost cliff on viral post
• Caching tool-using agent turns that triggered side effects
• Shared Redis with LRU deleting idempotency keys mid-retry
• Multi-region sticky-session break → confusing miss storms
• Writing unvalidated model output into semantic index (poison)

Debugging playbook

1. Log cache_key version + hit/miss + layer on every turn
2. Reproduce miss locally with same canonical JSON dump
3. For semantic: dump top neighbors + scores for the failing query
4. Confirm Redis memory / eviction policy (allkeys-lru surprises)
5. Check whether corpus version bumped but app still writes v1
6. Diff fingerprints between two “identical” requests that miss
7. Verify tenant hash tags on Cluster when MULTI fails
8. Correlate stale-answer tickets with deploy timestamps

Production readiness checklist

• Exact cache keys include model, tools, temperature, prompt version, corpus version, tenant
• TTL + memory policy documented; cache vs limits separated
• Hit/miss metrics and $/turn dashboards
• Semantic cache behind flag + high threshold + eval gate
• Redis HA / failover behavior tested (bypass path)
• Idempotency on mutating chat endpoints
• Session trim limits enforced
• Stampede / singleflight on hot keys
• Security tests for cross-tenant cache hits
• Invalidation runbook (prefix bump) in the design doc

Interview prompts

1. Exact vs semantic cache — risks and when each is appropriate?
2. How do you prevent cache stampedes on a viral FAQ?
3. What belongs in Redis vs Postgres vs the vector DB?
4. How do you invalidate after a policy doc update?
5. Redis is down — do you fail open or closed, and why?
6. Design keys for a multi-tenant SaaS with prompt versions shipping daily.
7. How do you evaluate a semantic cache before enabling it in production?

Key namespace taxonomy

cache:v{N}:{tenant}:{model}:{hash}     # exact completions
semcache:v{N}:{tenant}                 # vector index / doc prefix
ratelimit:{tenant}:tpm:{yyyyMMddHHmm}  # budgets
session:{user}:turns                   # short-term chat
idemp:{tenant}:{key}                   # POST dedupe
rag:v{N}:{tenant}:{qhash}              # retrieval id lists
emb:v{N}:{model}:{thash}               # query embeddings
lock:cache:{hash}                      # singleflight
lock:agent:{run_id}                    # agent ownership
fingerprint:deploy                     # optional current config hash

Use hash tags {tenant} when you need multi-key ops on Cluster.

Eviction and memory policies

Separate Redis DBs or instances for cache vs sessions/queues/limits when eviction would delete locks or rate-limit keys unexpectedly.

Sizing sketch

Rough order-of-magnitude:

• Average cached completion envelope: 2–8 KB
• 1M hot FAQ keys × 4 KB ≈ 4 GB payload (+ Redis overhead)
• Semantic index: embeddings × dim × bytes + graph/HNSW overhead

Measure with INFO memory and a load test; do not size from vibes.

Semantic cache eval protocol

1. Collect 200 FAQ paraphrases with gold answers (+ 50 adversarial near-misses)
2. Sweep thresholds 0.90–0.99
3. Plot accept rate vs wrong-answer rate (split by severity)
4. Pick threshold with wrong-answer rate below product tolerance (often near-zero for money/policy)
5. Re-run after every corpus, prompt, or embed-model version bump
6. Shadow-mode first: compute would-hit decisions without serving them
7. Canary one locale / one route before global on

Multi-region notes

Sticky users to a region when using local Redis caches. Global semantic caches amplify stale and privacy risk — prefer regional exact caches + shared SoT.

Document cache warm after regional failover: expect temporary cost/latency spikes.

Exact cache + RAG interaction

Order of operations that usually works:

1. Exact cache (full answer) — strongest win
2. Else retrieval (optionally cache chunk-id lists)
3. Else LLM generate
4. Write exact cache if response is cacheable (public FAQ, not personalized)

Do not cache answers that include user-specific balances, PII, or entitlement-gated clauses under a key that another user can hit.

flowchart TD
  Req --> E{Exact?}
  E -->|hit| Done[Return]
  E -->|miss| R{Need retrieval?}
  R -->|yes| ANN[Search + ACL]
  R -->|no| Gen[Generate]
  ANN --> Gen
  Gen --> W{Cacheable?}
  W -->|yes| SET[SET exact]
  W -->|no| Done
  SET --> Done

Observability fields

cache_layer: exact|semantic|rag_ids|emb|miss
cache_version: v3
tenant_id, route, model_id
hit: true|false
semantic_score: optional
fingerprint: optional
latency_ms_saved: optional estimate
stampede_wait_ms: optional
bypass_reason: redis_down|flag_off|uncacheable

Finance loves hit rate × cost/miss. Eng loves stale-answer correlation with version bumps. Join cache spans to LLM spans with one trace_id.

Security deep dive

Ship rule: treat cache keys like authorization decisions — review them in security design reviews, not only in perf reviews.

Streaming and UX

• Cache the final assembled message (and usage), not every token delta.
• On hit, you may still fake-stream the cached text for UX consistency — optional product choice.
• Do not claim “streaming savings” in cost decks if you only skip the model; be honest about TTFB vs total.

Partial streams on cancel: usually do not write incomplete answers to exact/semantic caches.

Tool-calling and agents

Caching a turn that already charged a card or sent an email is a product bug, not a perf win. Use idempotency keys for retries; use caches for pure Q&A.

Glossary

Cacheability policy matrix

Write the matrix in the design doc before enabling semantic cache.

Lua sketch for atomic get-or-lock

Conceptually: attempt GET; on miss SET NX lock; winner generates; SET value; DEL lock. Losers sleep/retry with jitter. Keep Lua small and reviewed — bugs here cause thundering herds or stuck locks.

Production tips:

• Lock TTL ≥ p99 generate time (or heartbeat extend)
• Always DEL lock in finally
• Cap loser wait; escalate to generate or 503
• Metric stampede_wait_ms and lock_steal_timeouts

Pairing with inference optimizations

Cache hits beat speculative decoding and quantization for repeated FAQs. Still:

• Keep origin fast (vLLM) for misses
• Route cheap models on cache miss for simple intents (cost routing)
• Do not double-count savings in exec decks
• Prefix / KV cache on the server is complementary to Redis answer cache — different layers

Redis is complementary to serving-stack work, not a substitute for a slow origin.

Worked walkthrough: FAQ bot with Redis

1. Define cacheability: public help-center routes only; account balance routes excluded.
2. Ship exact cache with tenant + model + fingerprint; TTL 1h; singleflight.
3. Dashboards: hit rate, $/turn, stale tickets.
4. Eval semantic offline on 200 paraphrases; threshold 0.97; shadow for a week.
5. Canary semantic on en-US FAQ only; kill switch ready.
6. Incident drill: kill Redis → confirm bypass + tighter RPM.

Anti-patterns (Redis AI edition)

• Global semantic pool across tenants
• Caching personalized balances under FAQ keys
• One Redis with LRU for both answers and rate limits
• No fingerprint → prompt hotfix never reaches users
• Semantic on without adversarial near-miss evals
• Treating Redis transcripts as the compliance archive
• Hashing non-canonical JSON (mystery miss rate)
• Infinite session lists “for better memory”
• Writing every agent scratchpad into semantic cache
• Claiming 90% cost reduction from a 5% hit-rate demo

Consistency models you will actually hit

Document freshness SLOs for answer cache separately from RAG ingest lag.

Interview whiteboard: design the cache layer

Prompt: “Cut LLM spend 30% for a 20-tenant support bot without wrong refund answers.”

A strong answer covers:

1. Exact cache first; keys with tenant + fingerprint
2. Cacheability matrix (no money movement)
3. Singleflight + metrics
4. Semantic only after eval gate; high threshold; flag
5. Redis split: cache vs limits
6. Fail-open bypass + rate shed
7. Invalidation via version prefix
8. Postgres remains SoT; vector DB remains retrieval

Weak answers only say “add Redis” or “use semantic similarity 0.8.”

FAQ (Redis AI caching)

Is semantic cache the same as RAG?
No. Semantic cache reuses answers. RAG retrieves evidence for the model to read. See vector databases and hybrid search.

Should I cache streaming responses?
Cache the final assembled message (and usage). Replaying token streams is rarely worth it; optional fake-stream for UX.

What if Redis is a single point of failure?
Fail open to the model with tighter rate limits, or run Redis HA. Never fail closed on answer cache alone for critical paths unless product requires it. Quotas may differ.

Can Redis replace my vector database?
No for full-corpus RAG. Redis vector features can power a small semantic answer cache or hot-path ANN, not your compliance-grade document index.

How big should TTL be?
Product-driven: public FAQ hours; policy after fingerprint change natural-miss; personalized seconds or never. Prefer version bumps over heroic TTL tuning.

Do I need Cluster on day one?
No. Start single primary + replica; graduate when memory or QPS demands it — but design key tags early if you expect Cluster.

Deep dive: versioned invalidation at org scale

Large teams ship prompt changes daily. Without versions:

• Support sees yesterday’s refund policy in cached answers
• Security patches to system prompts do not apply
• Tool schema changes yield tool-call JSON that no longer validates

Use a config fingerprint (hash of prompt templates + tool schemas + corpus version) inside every cache key. On deploy, fingerprint changes → natural misses. Optionally UNLINK the old prefix in a controlled job if memory pressure demands it.

Operational checklist for a prompt train:

1. Compute new fingerprint in CI
2. Deploy app (writes new keys)
3. Watch hit rate dip then recover
4. Optional: async delete old prefix after N hours
5. File a change note linking fingerprint ↔ git SHA

Deep dive: stampede math

Suppose a key expires and 500 concurrent requests arrive in 200 ms; generate takes 2s:

• Without singleflight: ~500 model calls, cost cliff, possible provider 429s
• With singleflight: 1 generate + 499 waits (~lock TTL / poll)

Always load-test expiry storms (align TTLs intentionally or add jitter to TTL: base + random(0, 60s)).

Deep dive: multi-tenant key design

# good
cache:v4:{tenantA}:gpt-x:fp9f3c:{req}

# bad
cache:v4:gpt-x:{req}          # cross-tenant collision risk
cache:{userEmail}:{req}       # PII in keys; rotation pain

Put non-PII tenant ids in keys. Put emails in values only if you must — usually you must not.

Observability dashboards that matter

What “good” looks like in a design doc

Staff-level cache design usually includes:

• Cacheability matrix by route
• Key schema + fingerprint contents
• TTL + eviction + instance split
• Singleflight plan
• Semantic eval protocol (or explicit “not now”)
• Fail-open / fail-closed decisions
• Metrics + alerts
• Invalidation runbook
• Threat model (tenancy, poison)
• Cost model: expected hit rate × $/miss

If any bullet is missing, the design is not ready for production review.

Hands-on next steps

1. Guided Caching and latency
2. vLLM for origin serving on misses
3. Cost / latency routing
4. Vector databases — do not confuse with semantic answer cache
5. Chunking and metadata — better RAG beats reckless semantic cache

End-to-end lab checklist (do this once)

1. Put exact-match cache in front of one /chat FAQ route.
2. Log hit/miss + key version; verify a prompt bump misses.
3. Add singleflight; load-test an expiry storm.
4. Break it: omit tenant from key — write a failing security test, then fix.
5. Shadow semantic cache; plot threshold curves; do not enable until wrong-answer rate is acceptable.
6. Kill Redis in staging; confirm bypass + metrics.
7. Write a half-page invalidation runbook.

Micro-project

Add exact-match cache in front of one /chat endpoint: key design (tenant + model + fingerprint), TTL, hit/miss metrics, singleflight, and a cache:v2 bump procedure documented in three bullets.

Related guided path

Caching and latency. Pair with vLLM for origin serving vs edge cache, and cost routing.