Building a ChatGPT-like product: streaming, tools, and memory

Chat is a product surface, not an architecture

Public descriptions of consumer assistants (ChatGPT-class products) hide a stack: streaming UX, tool calling, memory, moderation, rate limits, and model routing. Copying the UI without these layers produces a brittle demo.

This post is a pattern map — inspired by how large assistants are discussed publicly — not a claim about any vendor’s private design.

Request lifecycle

sequenceDiagram
  participant U as User
  participant FE as Client
  participant API as Chat API
  participant Mod as Moderation
  participant Orch as Orchestrator
  participant M as Model
  participant T as Tools
  U->>FE: message
  FE->>API: POST /chat (stream)
  API->>Mod: input check
  API->>Orch: build messages + memory
  Orch->>M: generate (stream)
  M-->>Orch: tool_call?
  Orch->>T: execute with policy
  T-->>Orch: tool result
  Orch->>M: continue
  M-->>FE: token deltas
  FE-->>U: render

Streaming contracts

Users judge quality by time-to-first-token and smooth deltas.

Ship:

• SSE or chunked HTTP with clear event types: delta, tool_start, tool_result, done, error
• Client cancellation → abort upstream generation
• Heartbeats so proxies don’t kill idle streams
• Never buffer the full answer “for convenience” on interactive paths

Failure modes: double-send on reconnect; tool JSON partially streamed into the visible transcript; incomplete done leaving the UI spinning.

Tools (function calling) done safely

Tools turn an LLM into an agent-shaped system. Treat them as privileged RPC:

flowchart TD
  Model[Model proposes tool_call] --> Val[Schema validate]
  Val --> Pol[Authz / allowlist]
  Pol --> Sand[Sandbox / timeout]
  Sand --> Res[Normalize result]
  Res --> Model2[Model continues]

Hard rules

• JSON Schema validation before execution
• Timeouts and payload size caps
• Tenant-scoped credentials — never ambient cloud roles
• Deterministic error strings the model can recover from
• Human approval for irreversible actions (payments, deletes, emails)

Memory tiers

Pattern: don’t dump entire history forever. Summarize with a revision id, and let users see/edit profile memory.

Latency and cost budgets

A single user turn may include: moderation → retrieval → model → 0..N tools → model again.

Budget explicitly:

TTFT budget:  e.g. ≤ 1.5s interactive
Tool budget:  ≤ 3 calls / turn default
Token budget: hard max_output + context trimmer

Route “small talk” to a cheaper model; reserve frontier models for tool-heavy or high-stakes turns.

What “done” looks like for a v1

• Streaming chat with cancel
• 2–5 tools behind schema + authz
• Thread store + optional summary job
• Moderation on input (and optionally output)
• Per-user rate limits and spend caps
• Trace each turn: model, tokens, tool timings

Tradeoffs

Ship advice

Start with excellent streaming + one safe tool + evals on tool selection. Memory and multi-agent orchestration are force multipliers only after the turn loop is reliable.