Tracing tool/LLM spans

Observability for agents is span-shaped

Logs tell you what happened; traces tell you when and how long each LLM call, tool invocation, and harness stage took. Agent debugging without spans is printf archaeology across async tasks.

Adopt OpenTelemetry concepts even if you only ship JSONL spans initially: trace_id, span_id, parent_span_id, name, start, end, attributes.

Span model for agents

Minimum span types:

• llm.chat — model, tokens, finish_reason
• tool.call — name, args_hash, ok/error
• harness.validate|verify
• retrieval.search — query hash, k, top_score
• skill.load — name, version, tokens

Example JSONL line:

{"trace_id":"t1","span_id":"s3","parent":"s1","name":"tool.call","start_ms":100,"end_ms":450,"attrs":{"tool":"lookup_order","ok":true}}

One user request = one trace_id linking all spans.

Callout — Waterfall reveals parallel lies: Orchestrator "parallel" workers that actually await sequentially show up instantly in spans.

Emitting spans without heavy SDK

Wrap primitives:

with span("tool.call", tool=name) as s:
    result = await call_tool(...)
    s.set("ok", True)

Flush span on context exit to append-only traces/YYYY-MM-DD.jsonl. Async-safe: use contextvars for parent span stack.

Correlate with cost logs via shared request_id.

Simple waterfall renderer

CLI script trace_waterfall.py traces.jsonl --trace t1:

0ms    ████ llm.chat (router) 120ms
120ms  ████████████ tool.lookup_order 330ms
450ms  ██████ llm.chat (main) 180ms

ASCII bars from (start_ms, duration_ms) — no fancy UI required. Highlight span >30% of total trace in red.

Finding a slow tool

Exercise:

1. Seed one tool with sleep(2) mock.
2. Run request; generate waterfall.
3. Document fix: parallelize, cache, timeout, or remove tool from hot path.

Write slow_span_postmortem.md — template for on-call.

Sampling and PII

Production traces sample (e.g. 10%) for cost — always sample errors and slow traces (>p95). Strip PII from span attributes; use hashes.

Trace context propagation

Pass trace_id from HTTP header through agent, MCP subprocess, and background tasks via contextvars or explicit parameter — broken propagation splits one user request into orphan spans. Standardize header name (traceparent W3C or simple X-Trace-Id).

Sampling strategy

100% trace in dev; sample in prod with tail sampling: always keep errors, slow traces (>p95), and canary run_ids; sample 5% of happy path. Reduces storage 10–20× while preserving debuggability for incidents.

From JSONL to OpenTelemetry

JSONL spans you build now map cleanly to OTel export later: span name → otel span name; attrs → attributes; parent → context. Document field mapping so migration is hours not weeks when you adopt Jaeger or Honeycomb.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Distributed traces across gateway/model/tools find latency and errors.

Diagram — Span tree

flowchart TD
  Root[HTTP request] --> LLM[llm.span]
  Root --> Tool[tool.span]
  LLM --> Tok[tokens]

Precise definitions & mental model

Trace/span IDs, baggage, redaction, sampling.

Tradeoffs — when to use what

Full vs sampled logging.

Failure modes (interview + on-call)

Missing child spans; PII in attributes.

Production & OSS practices

OTel conventions emerging for LLM; exemplars on errors.

Micro-project: Simple waterfall from JSONL

Ship:

1. Span instrumentation on LLM + ≥2 tools.
2. JSONL trace file per session or day.
3. trace_waterfall.py rendering one trace.
4. slow_span_postmortem.md from seeded slow tool.
5. Link trace_id in SSE done event from API lesson.

Acceptance: waterfall clearly shows dominant span; README documents trace_id lookup flow.

Checklist

• JSONL spans with parent links for full turn
• Waterfall script committed and runnable
• Slow tool identified via waterfall exercise
• trace_id exposed to client or logs
• PII policy for span attributes documented