Retries, rate limits, idempotency

Production traffic is hostile

Providers return 429, 502, truncated JSON; tools timeout; DNS blips. A resilient client wraps model and tool calls with retries, backoff, rate-limit awareness, and idempotency for mutating operations. Without it, agent workflows flake in CI and duplicate charges in prod.

Boring infrastructure beats clever replanning when HTTP fails.

Retry policy design

Retry only idempotent or safely repeatable operations:

Use exponential backoff with jitter: base * 2^attempt + random(0, jitter).

Cap max_attempts (3–5). Log every retry with attempt, delay_ms, error_class.

Callout — Retries multiply load: Circuit break when error rate spikes — lesson 9.5 — or you DDOS your own dependencies.

Rate limits

Track client-side token bucket per provider key:

• Requests per minute
• Tokens per minute (TPM)

When bucket empty, wait or degrade (smaller model, queue job). Surface rate_limit_wait_ms in spans.

For multi-tenant SaaS, isolate buckets per customer tier.

Idempotency for mutating tools

Pattern:

def issue_refund(order_id, amount, idempotency_key):
    if store.has(idempotency_key):
        return store.get(idempotency_key)
    result = payment_api.refund(..., idempotency_key=idempotency_key)
    store.put(idempotency_key, result)
    return result

Generate keys from (run_id, step_name) for workflow retries — same key on replay returns same result, no double refund.

Document which tools are idempotent in tools/IDEMPOTENCY.md.

Unified resilient client API

await resilient.call(model.chat, ..., retry_policy=READONLY)
await resilient.call(tools.issue_refund, ..., idempotency_key=key, retry_policy=MUTATING)

Centralize metrics: retry count, final error rate, latency inflation from backoff.

Testing resilience

Use mock server:

• First two calls 429, third 200 — assert success after backoff.
• Mutating call invoked twice with same idempotency key — assert one side effect.

Wire into CI as fast unit tests — no LLM required.

Provider-specific rate limit headers

OpenAI-compatible APIs return x-ratelimit-remaining-requests and token headers — parse and log them proactively before 429. Adaptive client slows down when remaining fraction drops below 10% — smoother than crash-retry loops.

Dead letter for exhausted retries

After max retries, write request to dead-letter queue (file or table) with full context for manual replay — do not drop silently. Ops runbook includes DLQ replay procedure with idempotency keys intact.

Testing under load

Optional k6 or locust script hammering mock provider — verify rate limiter prevents ban and p95 stays bounded. Portfolio can document planned test without full load infra if time-boxed.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Retries without jitter amplify outages; respect 429s and idempotency.

Diagram — Backoff

flowchart LR
  Call -->|429/5xx| Wait[Exp backoff + jitter]
  Wait --> Call
  Call -->|ok| Done
  Call -->|budget| Fail

Precise definitions & mental model

Idempotency keys, Retry-After, bulkheads, queueing.

Tradeoffs — when to use what

Aggressive retry vs fail-fast UX.

Failure modes (interview + on-call)

Retry POST with side effects; thundering herd.

Production & OSS practices

Shared rate limiter; chaos test 429s.

Micro-project: Resilient client

Ship:

1. Wrapper around model + tool HTTP with backoff and 429 handling.
2. Idempotency store for one mutating tool.
3. Tests for retry success and idempotent replay.
4. Span attributes for retries and rate-limit waits.
5. Document policies in README.

Acceptance: mock 429 test passes; duplicate mutating call does not double-charge mock ledger.

Checklist

• Retry with jitter on transient errors
• Rate limit wait or degrade documented
• Idempotency on ≥1 mutating tool
• Automated tests for retry and idempotency
• Retry metrics visible in traces or logs