OpenTelemetry for LLMs

What OpenTelemetry is

OpenTelemetry (OTel) is the open standard for traces, metrics, and logs. For LLM products it is how you stitch gateway → retrieve → model → tools into one timeline so “the agent failed” becomes a debuggable trajectory with cost and latency attached.

Vendor UIs (Datadog, Honeycomb, Grafana Tempo, Jaeger, LangSmith-style) come and go; OTLP + semantic conventions are the portable layer.

flowchart TB
  subgraph trace [Trace: chat.turn]
    S1[span: http.server]
    S2[span: rag.retrieve]
    S3[span: gen_ai.chat]
    S4[span: tool.search]
    S1 --> S2
    S1 --> S3
    S3 --> S4
  end

Interview cue: Draw a chat.turn root with retrieve + LLM + tool children, and name the GenAI attributes you’d attach for cost and repro.

The engineering problem

An agent failure is rarely “the model is dumb.” It is usually wrong tool args, retrieval miss, timeout, prompt regression, or a budget trip. Without traces you cannot:

Logs alone are not enough once you have multi-hop agents (agents and ReAct).

Architecture: signals you need

flowchart LR
  App[App / agent] --> SDK[OTel SDK]
  SDK --> Coll[Collector]
  Coll --> Tempo[Traces backend]
  Coll --> Prom[Metrics]
  Coll --> Logs[Log store]

Export via OTLP to a collector; don’t hard-bind your app to one SaaS SDK forever.

Propagation across boundaries

Broken propagation is the #1 reason “we have tracing” still feels useless for agents.

GenAI semantic conventions (shape)

Emerging OTel GenAI attributes (names evolve — pin a version in your repo):

Also record your attributes: tenant_id, prompt_version, retriever_id, eval_suite_id, cache_hit, agent_step, tool_name, engine_version.

# Shape — wrap one LLM call
from opentelemetry import trace

tracer = trace.get_tracer("shipai.agent")

def chat(messages: list[dict], model: str) -> dict:
    with tracer.start_as_current_span("gen_ai.chat") as span:
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("prompt_version", "support-v12")
        resp = client.chat(messages=messages, model=model)
        span.set_attribute("gen_ai.usage.input_tokens", resp["usage"]["input"])
        span.set_attribute("gen_ai.usage.output_tokens", resp["usage"]["output"])
        return resp

Span hierarchy that works

1. Root — chat.turn (user request id)
2. Children — rag.retrieve, rag.rerank, gen_ai.chat, each tool.*
3. Events — truncated prompts / outputs (PII policy first)
4. Links — optional link from online sample → offline eval run id

flowchart TD
  Root[chat.turn] --> Ret[rag.retrieve]
  Root --> LLM[gen_ai.chat]
  LLM --> T1[tool.search]
  LLM --> T2[tool.ticket.create]
  Ret --> Pack[context.pack]
  Pack --> LLM

Never log raw secrets. Hash or redact API keys, emails, and regulated fields. Prefer storing large payloads in object storage with a span attribute pointer, not a 50KB attribute blob.

What not to put on spans

Metrics alongside traces

Ship rule: optimize and alert on cost per successful task and p95 TTFT, not only tokens/sec on an empty cluster.

How it fits the product stack

Framework “tracing UIs” are fine for demos; production teams usually standardize on OTel so gateway and LLM share one trace.

Evals ↔ traces

When an offline eval fails, open the stored trace id for that fixture. Production online evals should sample traces into a review queue — same IDs in both worlds. Pair with evals fundamentals and MLflow for LLMOps / Weights & Biases for experiment lineage.

flowchart LR
  Offline[Offline eval fail] --> Tid[trace_id]
  Tid --> Tempo[Trace UI]
  Online[Online sample] --> Review[Human review queue]
  Review --> Track[MLflow / W&B run]

Sampling strategy

Sampling too aggressively means you never see rare tool failures; sampling everything bankrupts the observability bill.

Failure modes

• PII in spans — treat attributes as production data
• High-cardinality explosion — don’t put raw prompts as metric labels
• Missing propagation — broken parent/child across HTTP/Kafka
• Sampling too aggressive — rare tool failures invisible
• Token math wrong — streaming / tool rounds under-count cost
• Framework auto-instrument only — still add prompt_version yourself
• Clock skew across workers — weird span timings; NTP matters

Production checklist

1. Collector + OTLP from every service that touches a turn.
2. GenAI attributes + prompt_version + engine_version pinned.
3. Redaction policy reviewed like a data store.
4. Propagation tested across HTTP and Kafka.
5. Dashboards: TTFT, cost/task, tool error rate, cache hit.
6. Eval fixtures store trace_id.
7. Runbook: “open this trace when judge fails.”

Alternatives and complements

Micro-project

1. Add a parent span per chat turn with children for retrieve + generate + one tool.
2. Attach prompt_version + token usage; verify in your backend.
3. Propagate trace_id onto a Kafka embed job message.
4. Guided Tracing tool/LLM spans.

Related

Guided Deploy, cost, latency, observability. Key Tech: Weights & Biases. Advanced: Kafka for evented AI, MLflow for LLMOps. Inference: vLLM.