Trace and debug trajectories

If you cannot replay it, you cannot fix it

Agent bugs hide in trajectories — sequences of model decisions, tool args, and observations. When a run fails (wrong answer, bad tool args, max_steps, denied HITL), you need to replay from logged JSONL, pinpoint the offending step, patch tool validation or prompt, and add a regression fixture so the bug never returns silently.

This lesson treats trajectories as test artifacts, bridging hand-rolled loops to production tracing (OpenTelemetry spans in later modules).

Callout — log raw model payloads: Summaries help humans; raw tool_call JSON helps diff across prompt versions.

Trajectory format recap

Standardize runs/<run_id>.jsonl — one JSON object per line:

{"type": "user", "content": "Refund order 8812"}
{"type": "assistant", "tool_calls": [{"name": "get_order", "args": {"id": "8812"}}]}
{"type": "tool", "call_id": "tc1", "result": {"status": "shipped"}}
{"type": "assistant", "tool_calls": [{"name": "issue_refund", "args": {"order_id": 8812}}]}
{"type": "error", "message": "validation: order_id must be string"}
{"type": "final", "content": "Could not complete refund."}

Include metadata file runs/<run_id>.meta.json:

• model, prompt_version, started_at, outcome, total_steps

Replay tooling

replay.py modes:

python replay.py runs/failed_042.jsonl --summary    # human timeline
python replay.py runs/failed_042.jsonl --step 3     # zoom one step
python replay.py runs/failed_042.jsonl --reexecute 3  # rerun tool with recorded args

Summary view prints:

Step 2: get_order(id="8812") → ok
Step 3: issue_refund(order_id=8812) → VALIDATION FAIL (expected string id)

Reexecute calls real tool with logged args — verifies whether bug was transient data vs systematic schema mismatch (int vs string).

Do not replay mutating tools against prod without dry-run flag.

Debugging workflow

1. Classify failure — validation, wrong tool choice, bad observation, policy deny, max_steps.
2. Isolate step — first step where gold path diverges (compare to successful run on similar input).
3. Hypothesis — e.g. "model passes integer because schema said number not string."
4. Fix — tighten schema, add few-shot, pre-validate with coercion, or block tool until arg type match.
5. Regression — add fixture from this run's user input + expected tool args or final outcome.

Callout — compare to golden trajectory: Store fixtures/golden/run_refund_ok.jsonl for diff tools.

Regression fixtures

fixtures/regression/issue_refund_type.json:

{
  "id": "reg_042",
  "source_run": "failed_042",
  "user_input": "Refund order 8812",
  "assertions": [
    {"step_tool": "issue_refund", "args_schema_valid": true},
    {"final_contains": "refund"}
  ]
}

Test runner simulates tool responses from recorded trajectory (deterministic) or mocks backend — assert agent behavior after fix without live LLM in CI optional; live nightly if affordable.

Minimum: fixture file + manual rerun note in README.

Common failure signatures

Diff two trajectories

Compare successful vs failed run on same input:

python replay.py --diff runs/ok_12.jsonl runs/failed_042.jsonl

Highlight first divergent step — faster than manual scan on long runs. Store diff output in POSTMORTEM appendix.

Team workflow

When multiple engineers debug agents:

• Name runs with git branch + author: failed_042_ak_react
• Link run id to Linear/Jira ticket
• Attach minimal repro user input in fixture, not entire session history

Regression fixtures become shared contract — failing CI blocks merge on agent prompt changes.

Building a minimal replay TUI

Optional terminal UI: arrow keys step through trajectory, t reexecute tool, q quit. Even a 50-line curses or rich panel accelerates debugging vs raw JSON — portfolio stretch goal.

Export subset to share in bug reports: replay.py --export-step 3 failed_042.jsonl.

Trajectory schema versioning

Add schema_version: 1 to meta.json when trajectory format evolves. Replay tool migrates v0 → v1 or fails loudly — silent schema drift breaks regression fixtures stored in git history.

Document breaking changes in m7/replay/CHANGELOG.md.

Linking replay to prompt version

Include prompt_version and model in meta.json — when replay shows step 2 tool arg failure, bump prompt version and re-run same fixture to verify fix without re-discovering input manually.

Golden vs failed run library

Organize trajectories:

m7/replay/library/
  golden/
  failed/
  regression_fixed/

Move run from failed/ to regression_fixed/ after patch — visual progress for portfolio reviewers.

Automated replay in CI

Nightly job replays library/regression_fixed/ with mocked tools — ensures replay.py never rots when trajectory schema bumps. Fast unit test replays one golden file on every PR.

From JSONL to spans (preview)

Production adds trace ids linking LLM spans, tool spans, retrieval spans. Your JSONL is a minimal compatible artifact — fields map 1:1 to span events later.

Engineering problem (staff framing)

Without trajectories you cannot debug agents. Traces are the stack dumps of LLM systems.

Diagram — Trace spans

flowchart TD
  Req[Request] --> Spans[LLM/tool spans]
  Spans --> Store[Trace store]
  Store --> UI[Debug UI]
  Store --> Eval[Offline replay]

Precise definitions & mental model

Span attributes, redaction, replay, failure taxonomies.

Tradeoffs — when to use what

Full prompt logging vs privacy.

Failure modes (interview + on-call)

No correlation IDs; logging secrets; unreproducible tool side effects.

Production & OSS practices

OpenTelemetry-style traces; sampled full prompts; replay harness.

Deep dive (FAANG / OSS bar)

Push «trace-debug-trajectories» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Replay failed run

In m7/replay/:

1. replay.py with summary + step zoom (+ optional reexecute).
2. Take one failed run from prior lessons; document root cause in POSTMORTEM.md.
3. Add regression fixture derived from failure.
4. After fix, link passing rerun id in POSTMORTEM.

Checklist

• Replay tool reads standard JSONL format
• Bad step identified with evidence
• Regression fixture committed
• POSTMORTEM explains fix