OpenAI and Anthropic APIs

What these APIs are (plain English)

Most product traffic still goes through hosted chat APIs: messages in, tokens out, optional tools and structured JSON. OpenAI and Anthropic are the two contracts you will see constantly in industry code.

Learn one deeply; the other is a mapping exercise (roles, tool schema shape, streaming events).

Analogy: these APIs are SQL drivers for intelligence — your app should talk to a gateway, not sprinkle vendor SDKs in every feature flag path.

Interview cue: Talk about gateway patterns, token accounting, and failure modes — not just “call chat.completions.”

The problem they solve

Self-host (vLLM, Ollama) still often keeps the same client shape — OpenAI-compatible — so gateway skills transfer.

How they fit LLM apps

flowchart TB
  FE[Frontend] --> GW[Your gateway]
  GW --> Router{Router}
  Router --> OAI[OpenAI]
  Router --> ANT[Anthropic]
  Router --> Local[vLLM / Ollama]
  GW --> Tools[Tool runtime]
  GW --> Obs[OTel + cost logs]

Your gateway owns: auth, tenant quotas, PII redaction, model routing, retries, and observability. Provider SDKs are drivers — not the product architecture.

Architecture: the client contract

sequenceDiagram
  participant App
  participant GW as Gateway
  participant P as Provider
  App->>GW: POST /chat (stream)
  GW->>P: messages + tools
  P-->>GW: token deltas / tool_calls
  GW->>GW: maybe execute tools
  GW-->>App: SSE chunks

Messages and roles

Anthropic’s Messages API names differ slightly; map carefully and keep a provider-agnostic internal message type.

Tools and structured outputs

// Shape — one interface, many providers
type ChatRequest = {
  model: string;
  messages: Msg[];
  tools?: ToolDef[];
  responseFormat?: "text" | "json_schema";
  stream?: boolean;
};

Validate model JSON again on your side (zod/pydantic). Providers reduce errors; they do not eliminate them. See Structured outputs.

Mapping OpenAI ↔ Anthropic (mental cheat sheet)

Keep one internal Msg / ToolCall type; adapt at the edge.

How to use (resilient client sketch)

// Pseudocode — timeouts, idempotency, token logging
const res = await provider.chat({
  model: "gpt-4.1-mini", // pin; don’t use floating aliases in prod without intent
  messages,
  stream: true,
  headers: { "Idempotency-Key": idempKey },
  signal: AbortSignal.timeout(45_000),
});
// log: model, input_tokens, output_tokens, ttft_ms, request_id

Streaming

• Prefer SSE for chat UIs
• Separate TTFT timeout vs idle/total deadlines (see Networking for AI apps)
• Disable proxy buffering on LLM paths
• Heartbeat / comment frames if proxies idle-timeout

sequenceDiagram
  participant UI
  participant GW
  participant P as Provider
  UI->>GW: POST stream
  GW->>P: stream=true
  P-->>GW: first token
  Note over GW: TTFT clock stops
  GW-->>UI: SSE delta
  P-->>GW: more tokens
  GW-->>UI: SSE delta
  P-->>GW: done + usage
  GW-->>UI: final + metrics

Retries and idempotency

Ship rule: never retry a mutating tool blindly after a timeout — you may have succeeded server-side.

Gateway responsibilities (checklist)

1. AuthN/AuthZ + tenant quotas
2. Model allowlist + pin
3. PII redaction before logs
4. Timeout budgets (TTFT vs total)
5. Token + cost accounting
6. Tool policy + HITL hooks
7. Provider failover / routing

Cost/latency routing deep dive: Cost, latency, routing. Observability: OpenTelemetry for LLMs.

Walkthrough: tool loop behind a gateway

1. UI streams a user goal to your gateway.
2. Gateway calls provider with tools enabled.
3. Model returns tool_calls → gateway validates + executes.
4. Observations appended; second model call may finalize.
5. Trace stores provider request_id, tokens, tool latencies.

Same loop as Agents and ReAct — providers only supply the model step.

Alternatives

Production gotchas

• Floating model aliases change behavior under you — pin + changelog
• Double billing on retries without idempotency
• Tool loops without max steps / budgets
• Logging prompts that contain secrets or regulated PII
• SDK version skew across services → different defaults
• Assuming JSON mode = valid business objects — still validate
• Proxy buffering — users see multi-second blank TTFT
• Cross-provider prompt reuse — system prompts tuned to one model fail on another
• Ignoring usage fields — finance cannot attribute cost

flowchart TD
  Incident[Prod LLM incident] --> Q{Likely cause}
  Q -->|Blank spinner| Buf[Proxy buffering / TTFT]
  Q -->|Wrong answers after deploy| Pin[Model alias moved]
  Q -->|Duplicate refunds| Idem[Retry without idempotency]
  Q -->|Cost spike| Loop[Unbounded tools / retries]

Failure modes checklist

1. Browser holds the API key
2. latest model tag in prod
3. One global 60s timeout; TTFT never measured
4. Tools executed in the frontend
5. Provider error string shown raw to end users

Hands-on next steps

1. Wrap OpenAI + Anthropic behind one interface.
2. Add TTFT + total timeouts, retry policy, token logs.
3. Stream to a UI; verify nginx won’t buffer.
4. Guided Talk to models in the real world + Structured outputs.

Micro-project

Implement ChatClient with:

1. openai and anthropic adapters
2. Pin model ids via env
3. Stream SSE to a tiny HTML page
4. Log ttft_ms, input_tokens, output_tokens, request_id
5. Inject a deliberate 429 and prove backoff + Retry-After

Interview whiteboard: the gateway

Five boxes in order:

1. Client UI
2. Your gateway (auth, quotas, redaction, routing)
3. Provider adapters
4. Tool runtime
5. Observability (tokens, TTFT, request ids)

If someone draws the browser talking to OpenAI directly, correct them — that is a security answer, not a product answer.

Token accounting that finance accepts

Log per request:

• provider, model, input_tokens, output_tokens
• ttft_ms, total_ms, request_id
• tenant_id, feature, prompt_version

Roll up daily. Catch tool-loop cost spikes with per-run budgets. See Cost, latency, routing.

Structured outputs in the real world

Providers offer JSON mode / schema constraints. Still:

1. Validate with zod/pydantic
2. Retry once on schema failure with repair prompt — bounded
3. Fall back to safe error for users
4. Never execute tool args that failed schema

Deep dive: Structured outputs.

Multi-provider routing heuristics

Routers without evals are just randomizers.

Tradeoffs summary

Checklist

• Keys only on server
• Models pinned
• TTFT and total timeouts separated
• Idempotency on mutating paths
• JSON validated client-side
• Proxy buffering disabled on stream routes
• Usage logged with tenant + feature

Glossary

Debugging playbook (first hour)

Anti-patterns

1. Browser holds the API key — always server-side.
2. SDK calls from every feature flag path — use one gateway.
3. One 60s timeout for everything — TTFT and total differ.
4. Trust JSON mode alone — validate business objects.
5. Retry POSTs that already executed tools — double side effects.
6. Show raw provider errors to users — map to safe UX.
7. Copy-paste system prompts across providers — retune per model.

Security and privacy threat note

• Keys in client bundles or mobile apps = compromise.
• Prompt logs often contain PII — redact before OTel/W&B.
• Tool args are untrusted input — schema + allowlist + HITL for money/PII.
• SSRF via tools that fetch URLs — restrict egress.
• Multi-tenant: never reuse conversation state across tenants.

Interview prompts you should be able to answer

1. Draw the gateway between UI and providers — what does it own?
2. How do OpenAI tool_calls map to Anthropic tool_use conceptually?
3. When is a retry safe after a streaming timeout?
4. What do you log so finance can attribute LLM spend?
5. Why pin model ids instead of gpt-4o-latest-style aliases?

End-to-end: one product request

sequenceDiagram
  participant UI
  participant GW as Gateway
  participant P as Provider
  participant T as Tool runtime
  participant Obs as OTel / cost
  UI->>GW: stream chat + tenant
  GW->>GW: auth, quota, redact, pin model
  GW->>P: messages + tools
  P-->>GW: tool_calls
  GW->>T: validate + execute (idempotent)
  T-->>GW: observation
  GW->>P: continue
  P-->>GW: tokens + usage
  GW->>Obs: ttft, tokens, request_id, run ids
  GW-->>UI: SSE + final metrics

Production readiness checklist

• Keys only on server; rotate documented
• Model allowlist + pins
• TTFT vs total vs idle timeouts
• Idempotency on mutating tools
• JSON schema validated client-side
• Proxy buffering disabled on stream routes
• Usage logged: tenant, feature, model, tokens, request_id
• Max tool steps + $ budget per run
• Provider failover tested with chaos 5xx

How it works end-to-end (request lifecycle)

1. Client builds messages (+ tools / response format).
2. Gateway authN/authZ, injects server-side API keys.
3. Router picks provider/model (cost, latency, capability).
4. Outbound call with timeouts; stream or JSON.
5. On tool_call: validate → execute → append observation → continue.
6. Validate structured output; meter tokens; emit traces.

sequenceDiagram
  participant App
  participant GW as Gateway
  participant P as Provider
  participant Tools
  App->>GW: chat request
  GW->>P: messages + tools
  P-->>GW: tool_call or text
  alt tool_call
    GW->>Tools: execute
    Tools-->>GW: result
    GW->>P: continue
  end
  GW-->>App: stream/final + usage

OpenAI vs Anthropic mapping (practical)

Learn the shape, not every SDK method name. Pin SDK versions.

Retries, idempotency, and partial streams

Never retry non-idempotent tool executions without dedupe.

Prompt caching and cost controls

Providers offer prompt caching / prefix benefits for large stable system prompts. Practical rules:

• Keep stable prefixes identical byte-for-byte when possible
• Put volatile user content at the end
• Log cache hit rates when the API exposes them
• Still validate outputs — cache ≠ correctness

Pair with Redis for AI caching for response caches at your edge.

Safety and data handling

See Privacy and data for AI and Guardrails.

Observability fields (minimum)

provider, model, request_id, ttft_ms, total_ms, prompt_tokens, completion_tokens, cached_tokens (if any), finish_reason, tool_names, route_reason, tenant_id.

Wire to OpenTelemetry for LLMs.

Worked walkthrough: dual-provider support bot

1. ChatClient interface with OpenAI + Anthropic drivers.
2. Router: default cheap model; escalate on low confidence / hard intents.
3. Tools: search_docs, create_ticket with HITL on create.
4. Structured JSON for ticket fields; re-validate in app.
5. Chaos: force 5xx on primary → failover secondary.
6. Dashboard: cost/day, TTFT p95, tool error rate.

FAQ (hosted APIs)

Should every service call OpenAI directly?
No — one gateway.

OpenAI-compatible local models?
Yes for Ollama/vLLM — still keep the interface.

Is Anthropic “better”?
Task-dependent. Route with evals, not Twitter.

Deep dive: message roles and tool loops

Roles exist so models distinguish policy, user intent, and tool observations. Stuffing tool JSON into user messages without role discipline causes brittle agents. Keep tool results as tool role / blocks; truncate aggressively.

Putting what / why / how together

Architecture that survives production

flowchart TB
  Clients[Web / workers / agents] --> GW[API gateway]
  GW --> Limiter[Rate limit + quotas]
  Limiter --> Router[Model router]
  Router --> OAI[OpenAI]
  Router --> ANT[Anthropic]
  Router --> Local[vLLM OpenAI-compatible]
  GW --> Cache[Optional Redis exact cache]
  GW --> Obs[Traces + usage warehouse]

Gateway owns: auth, quotas, routing, redaction, retries, tool execution sandbox, and billing meters. Feature code calls ChatClient.complete(...) only.

Streaming UX contracts

• Flush first token quickly (TTFT)
• Heartbeats so proxies don’t kill idle SSE
• Final usage message after stream
• Client abort cancels upstream when possible

Buffering the entire completion in nginx “for convenience” destroys the product feel.

Tool-calling contract (provider-agnostic)

1. Declare tools with tight JSON Schema.
2. Model returns tool name + args.
3. Gateway validates args again.
4. AuthZ + timeout + byte cap.
5. Return observation; continue until stop or step limit.

Anthropic tool_use vs OpenAI tool_calls differ in wire format — normalize inside the gateway so agents see one shape.

Quotas and unfair traffic

When shedding, prefer degrading model class before hard failing interactive chat — if product allows.

Interview whiteboard

Boxes: Client → Gateway → Router → Providers; side boxes Tools, Cache, Metering. Red line: keys never leave gateway.

Failure story bank

1. Browser key → scraped → $10k bill.
2. Infinite tool loop without step cap.
3. Retry without idempotency → double refund tool.
4. Router sticky on outage → 100% errors instead of failover.

End-to-end lab checklist (do this once)

• Dual drivers behind one interface
• Stream TTFT logged
• 429 backoff honored
• Tool validate + deny path
• Usage rows exportable for a fake invoice

Production readiness checklist (gateway)

• Keys only on server; rotation tested
• Model ids pinned; logged on every trace
• Timeouts, retries, Retry-After honored
• Tool schema validation + authZ + step caps
• Structured outputs re-validated in app
• Token + $ meters per tenant
• Streaming path not buffered away
• Provider failover / degrade tested
• Redaction policy for logs and W&B exports
• Load shed behavior documented

What “good” looks like in a design doc

Gateway diagram, provider list, routing policy, tool allowlists, idempotency, quota model, privacy mode (retention), and SLOs (TTFT/error/$). SDK snippets alone are insufficient.

Common interview traps

• Putting API keys in mobile apps
• “We’ll just call the SDK in the React app”
• No answer for 429 storms
• Treating streamed partial JSON as final structured output

Multimodal and audio notes

Image/audio parts change token accounting and PII risk (faces, documents). Pin modalities per route; reject unexpected MIME types at the gateway. Eval multimodal separately — text golden sets won’t catch OCR failures.

Glossary addendum

Micro-project stretch

Add a forced-failover chaos switch and a Redis exact-match cache in front of the gateway; prove hit/miss metrics and that cache keys include model + tool schema hash.

How to evaluate gateway quality (not vibes)

Build a synthetic canary every minute: tiny chat + one tool dry-run. Canaries catch DNS/key/quota issues before users do.

Provider outage playbook

1. Detect via canary + provider status
2. Router shifts traffic; announce degrade if quality drops
3. Disable non-critical batch jobs to save quota
4. Incident doc: request_ids, route_reason, error bodies (redacted)

Deep dive: Anthropic extended thinking vs OpenAI reasoning models

Reasoning/thinking modes change latency and cost dramatically. Treat them as different model classes in the router: only escalate intents that need them; never make them the default for “hi.” Log thinking-token usage separately when exposed.

Comparing with open-weight serving

Keep the same gateway. Swap router target to vLLM OpenAI-compatible for sensitive tenants. Evals must compare task quality at equal latency budgets — not Elo from social media.

Putting what / why / how together

Related

Core: Structured outputs, Serving and streaming, Networking for AI apps. Guided Talk to models in the real world. Advanced Key Tech: OpenTelemetry for LLMs. Key tech: Ollama. Inference: Cost, latency, routing.