Networking for AI apps

Why networking is an AI topic

LLM calls are long, variable, expensive RPCs. A “simple chat API” inherits every distributed-systems footgun — plus token streaming, provider rate limits, and multi-second tails. Teams that treat the model as fetch() with optimism ship flaky products.

This page is the AI-engineering cousin of classic networking essentials: enough TCP/HTTP/streaming to design resilient clients and gateways.

Latency budget (typical chat turn)

sequenceDiagram
  participant U as User
  participant FE as Frontend
  participant GW as API gateway
  participant LLM as Model provider / GPU
  U->>FE: Send message
  FE->>GW: HTTPS POST /chat
  GW->>LLM: Upstream generate
  Note over LLM: Prefill + decode (seconds)
  LLM-->>GW: Token stream (SSE/WS)
  GW-->>FE: Forward chunks
  FE-->>U: Render tokens

Ship rule: budget TTFT and end-to-end separately. Users forgive slow answers more than a blank spinner. Pair with Serving and streaming.

HTTP patterns that matter

Timeouts are layered

• Connect timeout — fail fast on dead hosts
• TTFT timeout — abort if first token never arrives
• Idle timeout — abort if stream stalls mid-decode
• Total deadline — hard cap for the whole turn

Never use a single 60s socket timeout for streaming chats — you will kill healthy slow generations or hang forever on stalls.

Retries need idempotency keys

Retrying POST /chat after a gateway timeout can double-bill and duplicate side effects (emails, tickets). Use:

1. Client-generated idempotency key
2. Server-side dedupe window
3. Retry only on safe failures (connect errors, 408/429/503) — not on 400 validation errors
4. Exponential backoff with jitter

// Pseudocode — resilient LLM client shape
const res = await fetch("/v1/chat", {
  method: "POST",
  headers: {
    "Idempotency-Key": crypto.randomUUID(),
    "Accept": "text/event-stream",
  },
  body: JSON.stringify({ messages, stream: true }),
  signal: AbortSignal.timeout(45_000),
});

Rate limits are product design

Providers return 429 + Retry-After. Your gateway should:

• Queue or shed load with clear UX (“busy — retry in 8s”)
• Prefer token bucket per tenant over silent drops
• Separate interactive vs batch quotas
• Propagate remaining quota to clients when useful

Streaming: SSE vs WebSockets vs long poll

Watch for proxy buffering: nginx/proxy_buffering on turns a stream into a one-shot dump. Disable buffering for LLM paths.

flowchart LR
  Eng[Inference engine] --> GW[Gateway]
  GW -->|SSE chunks| FE[Frontend]
  GW -.->|buffered by mistake| Dump[One big blob]

Agents multiply network surface

One user message can fan out to search, DB, calendar, and another model. Design:

flowchart LR
  User --> Agent
  Agent --> LLM
  Agent --> Tools
  Tools --> Search
  Tools --> DB
  Tools --> Slack

• Per-tool timeouts and circuit breakers
• Max steps so loops cannot burn unbounded RPCs (Agents and ReAct)
• Trajectory logs with latency per hop (OpenTelemetry)
• Bulkheads: tool failures should not take down the chat gateway

TLS, auth, and data plane

• Prefer mTLS or signed service tokens between gateway and workers
• Never put long-lived provider API keys in browsers
• Strip PII at the edge when policy requires — networking is where DLP hooks live (Privacy and data for AI)
• Prefer private connectivity to vendors for sensitive tenants

How to build a resilient client (checklist)

1. Separate TTFT vs total deadlines
2. Idempotent retries with backoff + jitter
3. Explicit streaming transport + anti-buffering config
4. Per-tenant rate limits with UX
5. Trace every hop (gateway → model → tools)
6. Cancel upstream when the user hits Stop
7. Distinguish user-errors (4xx) from retryable (429/5xx)

Failure modes

Tradeoffs

• Aggressive retries — better availability; risk of amplified load and duplicate side effects.
• SSE — simple chat streaming; weaker for bidirectional agent protocols.
• Long deadlines — tolerate slow models; worse resource holding under load.

Glossary

Micro-project

Document timeout + retry policy for one LLM client: connect, TTFT, idle, total, which status codes retry, idempotency key lifetime.

Related guided path

Deploy, cost, latency, observability — Retries and rate limits, streaming APIs, runbooks. Pair with Serving and streaming.