Memory

Memory is state with evals — not an infinite transcript

Agents that append every message to context eventually hit token limits, cost cliffs, and confusion from stale facts. Memory stores distilled facts outside the window and retrieves relevant slices per turn — with explicit update, forget, and budget policies.

This lesson adds SQLite long-term memory to your agent: structured rows, retrieval into prompt context, and tests proving memories persist across sessions without unbounded growth.

Callout — memory ≠ chat history: Summarize and store facts ("user prefers email receipts") — do not dump 400 turns into SQLite verbatim.

Memory layers

This lesson focuses long-term personal/user memory in SQLite — portable, zero ops, good for course portfolio.

Schema design

CREATE TABLE memories (
  id INTEGER PRIMARY KEY,
  user_id TEXT NOT NULL,
  key TEXT,                    -- optional canonical key
  content TEXT NOT NULL,
  embedding BLOB,              -- optional for semantic recall
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  importance REAL DEFAULT 0.5,
  expires_at TEXT              -- nullable TTL
);
CREATE INDEX idx_mem_user ON memories(user_id);

Alternative: key-value (preference_timezone) vs free-text bullets — hybrid works.

Write path: when to memorize

Triggers:

• Explicit user command: "remember that I prefer SMS"
• Tool save_memory(content, key) the model may call after confirmation
• Post-turn extractor (small prompt): "List durable facts from this turn"

Avoid auto-saving every assistant sentence — noise accumulates.

Implement upsert by key: same user_id + key updates row instead of duplicating.

Read path: retrieval with budget

Before each model call:

1. Fetch candidate memories for user_id (recent N or semantic top-k if embeddings stored).
2. Rank by importance, recency, relevance to current query (keyword overlap OK for v1).
3. Format block:

[Memory mem_12] User prefers refund to original payment method.
[Memory mem_8] Account tier: pro (updated 2026-07-01)

4. Truncate to token budget (e.g. 500 tokens) — drop lowest rank first.

Log which memory ids injected for debugging.

Update and forget

Update: update_memory(id, content) refreshes updated_at; optionally decay old importance.

Forget:

• User says "forget my phone number" → delete by key or soft-delete
• TTL expiry job removes expires_at < now
• forget_memory(id) tool with confirmation for mutating class

Test cases required:

• Save → new session → memory still retrieved
• Update key → read returns new value
• Forget → memory absent next turn
• Budget → many memories, only top fits

Callout — GDPR-ish hygiene: Support delete-by-user even in toy projects — documents intent.

Conflict handling

Two memories contradict ("prefers email" vs "prefers SMS") — prefer newer updated_at or higher importance; inject note "conflicting memories resolved to latest."

Optional: one active value per key prevents most conflicts.

Without embeddings first

v1: SQL ORDER BY importance DESC, updated_at DESC LIMIT 10.

v2 stretch: embed memory content, cosine rank against query — reuse embedding code from RAG module.

Memory poisoning and hygiene

Users may say "remember that refunds are always approved" — adversarial or mistaken memories persist. Mitigations:

• Importance threshold — auto-save only when extractor confidence high.
• User-visible memory list — list_memories command deletes bad rows.
• Separate user vs system memory — system memories admin-only.

Run eval: inject malicious remember command, verify policy blocks or flags for review.

Semantic memory retrieval (stretch)

When keyword retrieval fails ("what did I say about my dog's name?" with no keyword overlap):

1. Embed query and memory rows offline or on write.
2. Cosine top-k memories above threshold 0.75.
3. Fall back to recency if no semantic hit.

Reuse embedding utilities from RAG module — same code, different table.

Session vs long-term boundary

On session end, optionally promote durable facts:

def promote_session_memories(session_id):
    for fact in extract_durable_facts(session_messages):
        upsert_memory(user_id, fact)

Without promotion, long-term memory only updates via explicit tools — simpler but misses implicit preferences. Document which policy your agent uses; eval both.

Integration with agent loop

def build_messages(user_id, user_text, session_messages):
    mem_block = fetch_memory_block(user_id, user_text, budget_tokens=500)
    system = base_system + "\n\n" + mem_block
    return [{"role": "system", "content": system}, *session_messages, {"role": "user", "content": user_text}]

Memory tools registered alongside domain tools from lesson 7.2.

Memory eval cases

Add to test suite:

1. Persistence: save fact, new session, ask question requiring fact.
2. Update: change preference, verify old value gone.
3. Forget: explicit delete command.
4. Budget: 50 memories inserted, only top-N appear in prompt block.
5. Cross-user isolation: user A memory invisible to user B query.

Passing memory tests without LLM proves storage layer; one integrated agent run proves retrieval prompt works.

Engineering problem (staff framing)

Memory is state: scratchpad, episodic, long-term retrieval. Unbounded history blows cost and confuses models.

Diagram — Memory tiers

flowchart TD
  Short[Working context] --> Comp[Summarize/truncate]
  Ep[Episodic store] --> Ret[Retrieve]
  Ret --> Short
  Long[Long-term facts] --> Ret

Precise definitions & mental model

Working memory vs vector memory vs durable profile; write policies.

Tradeoffs — when to use what

Full transcript fidelity vs summary compression.

Failure modes (interview + on-call)

Storing secrets; memory poisoning; never expiring stale facts.

Production & OSS practices

TTL, encryption, user reset, eval memory correctness.

Deep dive (FAANG / OSS bar)

Push «agent-memory» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: SQLite long-term memory

In m7/memory/:

1. SQLite schema + CRUD helpers.
2. Agent integration with retrieval budget.
3. test_memory.py — save/update/forget/cross-session tests (no LLM required for most).
4. One logged agent run showing memory influencing response in README.

Checklist

• Memories persist across process restart
• Forget and update tested
• Context budget enforced with logged ids
• No unbounded table growth in demo scenario