Failure isolation

Chaos drills belong in agentic systems too

Distributed systems engineers run chaos monkeys. Agentic workflows need the same discipline: kill a worker, stall a tool, corrupt one message — and prove the orchestrator recovers with bounded blast radius. Failure isolation means one bad node or tool does not poison the entire run or duplicate irreversible side effects.

Agents multiply failure modes: model timeouts, malformed tool JSON, rate limits, partial worker outputs, and cascading replans. Without isolation, orchestrators retry blindly until budget exhaustion.

Blast radius vocabulary

Define tiers for your workflow:

Each tier maps to policy: retry count, circuit breaker, compensation, escalate.

Callout — Fail closed on money and privacy: When in doubt on L3/L4, stop the run and preserve audit log. "Helpful" partial completions are liability in regulated domains.

Killing a worker safely

Orchestrator responsibilities when worker heartbeat stops:

1. Mark worker task lost after lease timeout (not instant — avoid false positives).
2. Check idempotency: can step be reassigned?
3. Reassign to fresh worker or fail run with structured error.
4. Never merge partial worker output without status: complete flag.

Chaos test script:

# after worker starts step 2
kill -9 $WORKER_PID
# expect orchestrator to requeue or fail within T seconds

Log worker_lost, reassign_count, final_outcome.

Tool failure isolation

Wrap every tool call:

with timeout_ms(5000):
    try:
        result = call_tool(...)
    except ToolTimeout:
        return {"error": "TIMEOUT", "retryable": True}
    except ToolValidation as e:
        return {"error": "VALIDATION", "retryable": False}

Circuit breaker: After N failures on payment_api, open circuit — short-circuit calls for cooldown period; route to ask-user or human.

Prevent error message injection: tool errors returned to model should be sanitized — no stack traces, no internal hostnames.

Timeouts at every layer

Orchestrator enforces parent timeout; children cannot extend silently.

Compensating actions

When step 3 fails after step 2 mutated state:

• Saga-style compensate: call undo_reserve_inventory if charge fails.
• Forward fix: create support ticket with run artifact.
• Mark dirty: flag entity for human reconciliation.

Document compensation in workflow spec — models should not invent compensations ad hoc unless tool explicitly allows.

Timeouts vs. retries interaction

Retries multiply load during outages. Cap total retries per run; use jittered exponential backoff; respect Retry-After headers from APIs.

Orchestrator as bulkhead

Assign each worker pool resource limits: max concurrent workers per tenant, max queue depth, priority lane for interactive vs batch jobs. When queue full, fail fast with OVERLOADED rather than unbounded latency — protects shared dependencies (payment API, vector DB).

Bulkheading is failure isolation at capacity planning layer — complements kill/recover tests.

Observability for failures

Dashboard counters: worker_lost_total, tool_circuit_open, compensation_invoked. Alert when worker_lost rate exceeds baseline — may indicate deployment bug not infra flake. Link spans with error_class attribute for triage.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Contain blast radius: one bad tool/agent must not corrupt the whole workflow.

Diagram — Bulkheads

flowchart TD
  Sup[Supervisor] --> A1[Agent sandboxed]
  Sup --> A2[Agent sandboxed]
  A1 --> DLQ[Dead letter]
  A2 --> Out

Precise definitions & mental model

Bulkheads, timeouts, circuit breakers, compensation.

Tradeoffs — when to use what

Fail-fast vs retry-with-backoff — know which errors are transient.

Failure modes (interview + on-call)

Shared credentials across agents; catch-all retries on 4xx.

Production & OSS practices

Per-tool SLOs; isolation by queue; chaos tests.

Deep dive (FAANG / OSS bar)

Push «failure-isolation» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Kill a worker; recover

In portfolio:

1. Multi-worker workflow with heartbeat or step lease.
2. Chaos script kills worker mid-step; orchestrator recovers or fails cleanly.
3. Tool wrapper with timeout + circuit breaker on one flaky mock tool.
4. One compensation path documented and tested (even if toy undo_*).
5. Write failure_report.md with blast radius table and observed behavior.

Acceptance: chaos test reproducible from README; no duplicate mutating side effects on recovery.

Checklist

• Worker kill test passes with logged recovery path
• Tool timeouts and sanitized errors implemented
• Circuit breaker or retry cap on flaky dependency
• Compensation or fail-closed documented for one mutating step
• failure_report.md committed