UX of uncertainty, citations, feedback

Users experience probability

LLM products output distributions, not guarantees — but UIs often present answers as facts. Uncertainty UX covers how products communicate confidence, cite sources, offer corrections, and recover from errors without destroying trust. This is product engineering adjacent to evals: your judge may know the answer is shaky; the user must see it too.

Bad uncertainty UX causes: over-reliance on wrong answers, abandonment when answers hedge too much, and no signal for your data flywheel when users disagree silently.

Critique framework

For each product (pick 3 AI assistants, search+answer tools, or copilots you actually use), score:

1. Epistemic humility — Does it say "I don't know" when appropriate?
2. Citations — Clickable, verifiable, tied to specific claims?
3. Confidence signaling — Explicit (high/medium/low) or implicit (wording only)?
4. Correction path — Thumbs down, edit, escalate to human?
5. Failure recovery — After tool error, clear next step?
6. Consistency — Same question, refresh — wild variance handled how?

Document in ux_critique.md with screenshots or quoted UI text (respect ToS — no scraping at scale).

Callout — Citations as trust anchors: Agentic RAG without visible sources trains users to audit manually — most won't. Surface sources inline or in panel.

Citation patterns that work

Pattern	Pros	Cons
Inline footnotes [1]	Precise claim mapping	Clutter in chat
Side panel sources	Clean reading	Disconnect from claim
Expandable "Sources" block	Mobile friendly	Easy to ignore
Quote snippets	Verifiable	Token heavy

For capstone, pick one pattern and implement consistently — mixing styles confuses.

Feedback loops for UX and evals

Thumbs up/down should capture:

• message_id, run_id, optional category (wrong fact, tone, incomplete)
• Optional free text
• Never block user on feedback — friction kills signal

Route negatives to:

• Review queue for golden set candidates
• Weekly aggregate dashboard (flywheel lesson)

Pair with ask-user route from agentic RAG — clarifying question beats confident hallucination.

Mapping to capstone

capstone_ux.md section:

• Chosen uncertainty pattern (when to hedge, when to cite)
• Feedback UI wireframe or ASCII mock
• Which eval cases validate UX (low retrieval score → must show uncertainty)
• Accessibility note — don't rely on color alone for confidence

Connect to guardrails: blocked PII answers need user-visible explanation, not silent fail.

Anti-patterns

• Fake confidence after retrieve miss
• Citations to irrelevant docs (worse than none — audit in component evals)
• Feedback black hole — users never see impact

Progressive disclosure

Show summary first, citations expandable, raw tool JSON hidden unless "developer mode." Reduces cognitive load for merchants and support reps — capstone_ux.md should specify disclosure levels per persona.

Error copy as UX

When guardrail blocks, user message beats generic "something went wrong" — "We can't process pasted credit card numbers; remove card data and retry." Map red-team failures to copy templates in advance.

Measuring UX with evals

Add judge criterion user_clarity — would non-expert know next step? Pairs with uncertainty UX — hedged answer without next step fails clarity even if factually cautious.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Probabilistic UX: stream, cite, confidence, undo, escalate.

Diagram — Uncertainty UX

flowchart LR
  Ans --> Cite
  Ans --> Conf[Confidence]
  Conf --> Undo
  Conf --> Escalate

Precise definitions & mental model

Calibrated display, progressive disclosure, recoverable actions.

Tradeoffs — when to use what

Honesty vs user trust theater.

Failure modes (interview + on-call)

Fake confidence bars; irreversible silent actions.

Production & OSS practices

UX review with failure corpus screenshots.

Micro-project: Critique 3 products

Ship:

1. ux_critique.md with 3 products × 6 criteria + overall lesson learned.
2. capstone_ux.md mapping patterns to your project.
3. Mock or stub feedback endpoint logging events.
4. One golden eval asserting citation present when policy answer given.
5. README UX principles list (≤5 bullets) for capstone.

Acceptance: capstone_ux.md specifies concrete UI behaviors, not "we'll be user-friendly."

Checklist

• Three product critiques committed
• capstone_ux.md with citation and feedback design
• Feedback logging stub implemented
• Eval case for citation/uncertainty behavior
• UX principles in README