Long-running / checkpoint / resume

Industry-shaped agents survive restarts

Demo agents live in one Python process. Production agents run for minutes or hours — research pipelines, batch migrations, long tool chains — and processes die: deploys, OOM kills, spot instance reclaim, laptop sleep. Durability means persisting enough state to resume from the last committed step without redoing side effects or corrupting data.

Checkpoint/resume is not optional for agentic workflows once mutating tools enter the graph. You already have an event log from handoff lessons; durability adds step boundaries and idempotent replay semantics.

What to checkpoint

At each durable step boundary, persist:

{
  "run_id": "uuid",
  "step": 3,
  "step_name": "fetch_invoices",
  "status": "completed",
  "input_hash": "...",
  "output_ref": "runs/uuid/step_3.json",
  "started_at": "...",
  "finished_at": "..."
}

Checkpoint after external side effects succeed or before irreversible mutations — pick per step:

• Read-only steps — checkpoint after completion (safe to retry whole step).
• Mutating steps — checkpoint intent before call; mark completed only after idempotency key confirmed.

Store checkpoints in SQLite, Postgres, or S3 + index — course portfolios often use SQLite for simplicity.

Callout — At-least-once execution: Assume every step may run twice. Design tools with idempotency keys (refund_idempotency_key=run:3) or dedupe tables.

Resume algorithm

On worker start:

1. Load latest checkpoint for run_id.
2. If status == in_progress, apply stale detection (timeout → mark failed or retry).
3. Continue from step + 1 unless output_ref missing (replay step).

Never trust in-memory step counters alone. The checkpoint store is source of truth.

def run_job(run_id: str):
    state = load_checkpoint(run_id) or init_run()
    for step in STEPS[state.step:]:
        mark_in_progress(run_id, step)
        result = execute_step(step, state)
        save_checkpoint(run_id, step, result, status="completed")

Process kill test

Your proof artifact is a test or script:

1. Start multi-step job (≥5 steps with simulated delay).
2. After step 2 completes, kill -9 the process.
3. Restart worker with same run_id.
4. Assert steps 1–2 not re-executed (mock counters) and job finishes through step 5.

Without this test, "we checkpoint" is wishful thinking.

Long-running UX

Users need run status outside the LLM thread:

• Expose GET /runs/{id} with step list and percent complete.
• Push notifications or webhook on terminal states.
• Allow cancel → write control.cancel event; worker polls between steps.

Do not stream 40 minutes of token output — stream progress events.

Comparison to workflow engines

Temporal, Dagster, or Celery provide durability primitives. Hand-rolling teaches the contract; migrating later is easier when your step model maps cleanly:

Failure modes

• Double refund — Missing idempotency on mutating replay.
• Skipped step — Off-by-one resume index after partial write.
• Zombie in_progress — Worker dies after mark in_progress; needs lease timeout.

User-visible progress for long jobs

Durability without UX feels broken — users submit job, silence for five minutes. Emit progress events: {step: 3, total: 8, label: "Fetching invoices"}. SSE from API module can stream these separately from chat tokens. Store latest progress in checkpoint row for polling clients.

Align step labels with support macros — reduces "is it stuck?" tickets.

Storage sizing and retention

Checkpoint + event logs grow with step count and tool payload refs. Define retention: completed runs archived to cold storage after 30 days; failed runs kept 90 days for postmortem. Compress JSONL with gzip in object storage for portfolio cost discipline.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Long agent runs must checkpoint and resume like workflows — process crashes are normal.

Diagram — Checkpoint / resume

flowchart LR
  Step --> Ckpt[Checkpoint state]
  Ckpt --> Crash([Crash])
  Crash --> Resume[Resume from ckpt]
  Resume --> Step

Precise definitions & mental model

Durable execution, idempotent effects, exactly-once illusions vs at-least-once + dedupe.

Tradeoffs — when to use what

Sync request/response vs async job+poll/webhook.

Failure modes (interview + on-call)

Non-idempotent tools on retry; huge checkpoints; no TTL.

Production & OSS practices

Temporal/queues patterns; store step hashes; user-visible progress.

Deep dive (FAANG / OSS bar)

Push «durable-jobs» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Durable multi-step job

In portfolio:

1. Implement ≥5-step job with SQLite (or file) checkpoints.
2. Wrap ≥1 mutating step with idempotency key.
3. Ship test_resume_after_kill that passes on CI or local documented command.
4. Add CLI: job run --id X and job resume --id X.
5. Document checkpoint schema in README.

Acceptance: kill test passes; logs show exactly one execution per completed step.

Checklist

• Checkpoint schema documented and persisted each step
• Resume continues from correct step after kill -9
• Idempotency on at least one mutating step
• Automated durability test committed
• Module README explains stale in_progress handling