Observability for LLM apps: traces, evals, and user feedback
Logs aren’t enough. Wire traces, offline eval suites, and closed-loop feedback so quality regressions are detectable like latency pages.
Framed from public engineering talks, blogs, and OSS patterns. Not confidential internals or invented quotes.
Why LLM systems feel “unpageable”
Classic services page on error rate and latency. LLM features fail semantically while returning HTTP 200. Public practice from teams running assistants in production: treat quality as a first-class SLO, backed by traces + evals + feedback.
Three loops
flowchart LR
subgraph Online
Trace[Distributed traces]
FB[User feedback]
end
subgraph Offline
Gold[Golden sets]
Judge[LLM/human judges]
end
Trace --> Store[Trace store]
FB --> Store
Store --> Curate[Curate hard cases]
Curate --> Gold
Gold --> Judge
Judge --> Ship[Release gate]
Ship --> Trace
What to put on a trace
Every user turn should carry a span tree:
- Input messages (redact PII per policy)
- Retrieved chunk ids + scores
- Model id, temperature, token counts
- Tool calls: name, latency, status
- Output + safety flags
- Cost estimate
Correlation id from client → gateway → tools. Without it, “the bot lied on Tuesday” is archaeology.
sequenceDiagram
participant C as Client
participant A as App
participant M as Model
participant T as Tool
C->>A: turn (trace_id)
A->>M: generate
M->>A: tool_call
A->>T: execute
T-->>A: result
A->>M: continue
A-->>C: answer
Note over A: export spans + scores
Evals: offline gates
Minimum viable suite:
| Type | Example metric | Blocks release? |
|---|---|---|
| Task accuracy | Exact match / rubric | Yes for critical paths |
| Groundedness | Citation support | Yes for RAG |
| Safety | Policy violations | Yes |
| Latency | TTFT P95 | Yes for interactive |
| Cost | $/task | Soft then hard |
Run on every prompt, model, or retriever change. Store fixtures in git.
Online feedback
- Explicit: thumbs, “report,” reason codes
- Implicit: regenerate, abandon, edit distance to final user text
- Sampled human review queues for high-severity
Feed disagreements into the golden set weekly — that’s how evals stay alive.
LLM-as-judge (use carefully)
Judges drift and collude with the generator family. Calibrate against humans; use pairwise comparisons; freeze judge prompts like production code.
Failure modes
- Logging prompts to a vendor without DPA
- Metrics that reward verbosity
- Eval sets too small / too English-only
- Alerting only on 5xx while faithfulness collapses
What to ship
- OpenTelemetry-style traces for the orchestrator
- Redaction middleware before export
- CI job: eval suite on PRs touching prompts
- Feedback → issue queue with trace deep links
- Weekly quality review with slices (tenant, locale, intent)
Staff-level framing
Observability for LLMs is closing the loop between production reality and the fixtures you trust. Traces without evals are diaries; evals without production sampling are museum pieces.