API + streaming

Product UX expects streams

Users tolerate seconds of wait when they see progress — tokens appearing, tool status updates, step labels changing. Batch JSON responses feel broken for chat and agent UIs. Serving your agent behind FastAPI with Server-Sent Events (SSE) is the minimal production-shaped interface: one HTTP POST to start, one long-lived stream of typed events until done.

Streaming also decouples orchestrator runtime from client timeouts — proxies may kill idle connections, but active token streams stay alive.

FastAPI service shape

Minimum routes:

• POST /v1/chat — accept { "messages": [...], "run_id": optional }, return StreamingResponse
• GET /health — liveness for load balancers
• GET /v1/runs/{id} — optional status for durable jobs from workflow lessons

Keep business logic out of route handlers — call AgentService.stream_turn() so CLI and API share core.

Example SSE event types:

event: token
data: {"text": "Checking"}

event: tool_start
data: {"name": "lookup_order", "id": "tc_1"}

event: tool_end
data: {"id": "tc_1", "ok": true, "summary": "found order"}

event: error
data: {"code": "RATE_LIMIT", "message": "..."}

event: done
data: {"run_id": "...", "usage": {"input_tokens": 1200}}

Clients parse event type + JSON payload — not raw token soup only.

Callout — SSE vs WebSockets: SSE is one-way server→client over HTTP/1.1, simpler behind corporate proxies. WebSockets fit bidirectional low-latency games; most agent UIs only need SSE.

Implementing SSE in FastAPI

Pattern:

async def event_generator():
    async for chunk in agent.stream(...):
        yield f"event: {chunk.type}\ndata: {json.dumps(chunk.data)}\n\n"

return StreamingResponse(event_generator(), media_type="text/event-stream")

Headers: Cache-Control: no-cache, Connection: keep-alive, disable buffering on nginx (X-Accel-Buffering: no).

Handle client disconnect — cancel agent task to avoid orphan tool calls (request.is_disconnected() loop).

Streaming non-token events

Agent streams should include tool and harness events, not only LLM tokens:

• Validation blocked
• Skill loaded (name, version)
• Verify pass/fail

UIs render structured progress; logs correlate with traces (next lessons).

Client usage documentation

README must show:

curl -N -X POST http://localhost:8000/v1/chat \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

Plus minimal Python/JS client parsing SSE lines. Stranger test: teammate runs curl without reading source.

Auth and limits (stub)

Production adds API keys or JWT on POST /v1/chat. Stub Authorization: Bearer dev-key and document where gateway replaces it.

Error handling mid-stream

Errors after 200 OK start must become SSE error events, then done — do not truncate TCP silently. Partial answers should include partial: true in done payload.

Backpressure and queue depth

When agent turns take 30–90 seconds, naive thread-per-request models exhaust workers under burst traffic. Cap concurrent agent runs per process; return 503 with Retry-After when saturated rather than accepting requests that will timeout. Queue depth metric helps distinguish overload from slowness.

For portfolio FastAPI, asyncio.Semaphore around agent entry is sufficient demonstration — document production would use dedicated worker pool or job queue for long runs.

SSE reconnection and idempotency

Clients on flaky mobile networks drop SSE mid-stream. Support Last-Event-ID or client-supplied run_id to resume status polling via GET /v1/runs/{id} when stream dies — durable job patterns from workflow module apply. Idempotent POST with client_request_id prevents duplicate side effects if user retries submit.

Contract testing the API

Publish OpenAPI or JSON schema for request/response event types; consumer tests in frontend or CLI assert parser handles all event types your server emits. Breaking SSE payload shape is a silent client bug — version events with schema_version field during rapid iteration.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Streaming improves UX but complicates cancellation, billing, and partial failures.

Diagram — Streaming tokens

sequenceDiagram
  Client->>API: request
  loop tokens
    API-->>Client: delta
  end
  API-->>Client: done

Precise definitions & mental model

TTFT, TPOT, backpressure, cancel semantics.

Tradeoffs — when to use what

Buffer complete JSON vs stream tokens.

Failure modes (interview + on-call)

Billing on partial; no cancel; buffering entire stream in gateway.

Production & OSS practices

SSE/WebSocket standards; idle timeouts; trace partials.

Micro-project: FastAPI + SSE

Ship:

1. FastAPI app wrapping your best agent/workflow with SSE stream.
2. ≥3 event types (token, tool_*, done).
3. Client disconnect cancels in-flight work (log cancel).
4. README curl example + expected output snippet.
5. Health route for smoke checks.

Acceptance: uvicorn start → curl receives streaming tokens and tool events on a tool-using prompt.

Checklist

• POST /v1/chat streams SSE with typed events
• Agent logic shared with non-HTTP entrypoint
• Client disconnect handling implemented
• README curl documentation verified
• Health endpoint returns 200