Shared state and handoffs

Multi-agent without a protocol is distributed spaghetti

When two LLM nodes can both read and write shared memory, you have a distributed system — minus the decades of lessons learned. Shared state and handoffs must be explicit: typed messages, immutable snapshots where possible, and tests that prove worker B can resume after worker A crashes.

Hidden global mutable state (current_user_context = {} imported everywhere) makes demos fast and production impossible. Every handoff should be serializable JSON you can log, replay, and assert in unit tests.

Design a message protocol first

Before adding agents, define the envelope every node sends:

{
  "run_id": "uuid",
  "from": "orchestrator",
  "to": "research_worker",
  "type": "task.assign",
  "payload": { ... },
  "correlation_id": "uuid",
  "timestamp": "ISO8601"
}

Message types should be a closed enum: task.assign, task.result, task.error, handoff.user, control.cancel. Extensibility comes from versioned payload schemas, not ad-hoc string fields.

Rules:

• Orchestrator owns run lifecycle; workers never mutate global run state directly.
• Results are append-only events on a log; derived state is computed.
• Errors include retryable: bool and error_class for harness decisions.

Callout — Event log vs. mutable blob: Prefer an event log (runs/{id}/events.jsonl) over a single JSON file workers overwrite. Overwrites race; logs replay.

Handoffs that are testable

A handoff is not @other_agent please continue. It is a function:

def handoff(from_node: str, to_node: str, packet: TaskPacket) -> HandoffRecord:
    validate_schema(packet)
    append_event(HandoffRecord(...))
    return record

Test matrix:

1. Valid packet → event appended, worker receives exact payload.
2. Invalid schema → rejected before side effects.
3. Duplicate correlation_id → idempotent no-op or explicit error.
4. Worker reads handoff cold start (no in-memory globals).

Document what crosses the boundary vs. what stays local (tool credentials stay with worker process, not in packet).

Avoiding hidden global soup

Anti-patterns:

• Module-level dict updated by "the research agent"
• Implicit context via thread-locals without logging
• Shared ORM session across async workers

Replacements:

• Pass run_id through every call chain
• Store working memory in namespaced keys: run:{id}:research:notes
• Use read-only snapshots for downstream nodes (research_result_v3 immutable)

When nodes need shared knowledge, publish facts with provenance:

{"fact": "Policy version 2024-03 applies", "source": "policy_search", "confidence": 0.92}

Downstream nodes cite facts; they do not re-fetch silently and diverge.

State size and context pressure

Handoff payloads must fit your context budget from the skills module. Large artifacts (PDF text, big JSON) go to object storage or a run-scoped blob store; packets carry handles:

{"artifact_ref": "runs/uuid/research.md", "summary": "..."}

Orchestrator merges summaries; full artifacts load only when a node declares need.

Versioning and compatibility

When payload schemas evolve:

• Bump protocol_version in envelope.
• Workers accept N and N-1 during rollout.
• Reject unknown versions with explicit error, not silent field drop.

Document breaking changes in protocol/CHANGELOG.md — same discipline as API changelogs.

Schema evolution without breaking runs

When you add fields to task.assign payloads, old workers still in deployment must not crash on unknown keys. Forward-compatible rules: receivers ignore unknown fields; senders always include protocol_version; breaking renames require new message type (task.assign.v2) rather than silently changing meaning of objective string field.

Maintain a compatibility matrix in protocol docs: which worker versions accept which packet versions. In portfolio scope, two versions max — but document the pattern as if ten existed.

Debugging handoffs with replay

Replay should be deterministic given the same event log and tool mocks. Store tool results in the log when side effects are expensive to reproduce — optional tool.result events enable offline replay without hitting APIs. Redact PII in stored tool payloads; use handles for large blobs as noted earlier.

Callout — Replay is your best integration test: If replay cannot reconstruct orchestrator state, your protocol is incomplete.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Handoffs need typed state schemas. Stringy message passing loses fields and creates he-said-she-said bugs.

Diagram — Typed handoff

sequenceDiagram
  participant A as Agent A
  participant S as State store
  participant B as Agent B
  A->>S: write typed fields
  A->>B: handoff pointer
  B->>S: read state

Precise definitions & mental model

Shared blackboard, reducers, schema evolution, ownership of fields.

Tradeoffs — when to use what

Pass-by-message vs shared store — auditability vs convenience.

Failure modes (interview + on-call)

Race conditions; clobbering fields; unversioned schemas.

Production & OSS practices

Pydantic state; optimistic locking; handoff tests.

Deep dive (FAANG / OSS bar)

Push «shared-state-handoffs» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Message protocol

In your portfolio:

1. Define protocol.md + JSON Schema for ≥4 message types.
2. Implement event log append + replay that reconstructs run state.
3. Wire orchestrator → worker handoff using only serialized packets (no globals).
4. Write tests: valid handoff, schema rejection, replay after simulated restart.
5. Diagram one run as sequence of message types.

Acceptance: kill worker mid-run, restart, replay log, continue from last task.result.

Checklist

• Protocol doc + JSON Schema committed
• Event log append/replay implemented
• Handoff tests pass including restart scenario
• No module-level mutable run state in agent code
• Module README documents envelope fields and versioning policy