Shopify — production merchant agent

Production merchant agents vs demo chatbots

Shopify's engineering posts on production agentic systems (Sidekick-class merchant assistants) highlight tool-grounded actions, eval loops with LLM judges, and merchant-specific policy — not open-ended creativity. Demos optimize for wow; production optimizes for task completion rate, policy compliance, and measurable regressions when prompts change.

This lab builds a merchant-shaped toy agent (inventory, orders, store settings) with tools + calibrated judge eval loop — mirroring the eval module in a product context.

Reading extraction

Capture in industry/shopify/notes.md:

• Tool catalog design for merchant workflows
• When to retrieve vs call Admin API tools
• Eval culture — golden tasks, judges, canaries before prompt rollout
• UX patterns — confirming destructive actions, showing what changed
• Difference between internal ops agents and merchant-facing agents

Callout — Judge in the loop: Shopify-class systems use LLM judges in eval pipelines — same instrument you calibrated; production still needs deterministic guards on money-moving tools.

Tool-calling merchant agent

Minimum tools (mock or Shopify dev store if available):

• get_product, update_product_title
• list_orders, get_order
• draft_discount (non-destructive) vs publish_discount ( gated )

Agent must ** cite tool results** in answers — no inventing SKUs.

Skills module patterns apply: merchant policy skill loaded dynamically (returns, discount limits).

LLM-judge eval loop

Build evals/merchant/ with ≥10 scenarios:

• Happy path product update
• Policy violation attempt (discount too deep)
• Ambiguous product reference → should ask
• Tool failure → graceful message

Judge rubric criteria: correctness, policy, clarity — reuse calibration practices from eval module.

CI target: make eval-merchant on PR touching prompts/tools.

Tradeoffs vs demo agents

Write tradeoffs.md:

Dimension	Demo agent	Production-shaped
Tools	read-only wiki	mutating with authz
Evals	manual vibe check	judge + golden set
UX	free text	confirmations on writes
Cost	unbounded	routed models, caches

Honest section: what your lab still skips (multi-tenant isolation, real Shopify scale).

Merchant trust and confirmations

Merchant-facing agents destroy trust when they mutate catalog without confirmation. Pattern: draft_* tools free; publish_* requires explicit user click or typed confirm string in UI — agent proposes draft, UI gate publishes. Eval cases assert draft-only on ambiguous prompts.

Seasonal traffic and evals

Merchant traffic spikes Black Friday — evals should include load-sensitive cases (inventory low, discount stacking). Notes.md: production teams re-run golden set before peak events; adopt cadence for capstone before demo day.

Sidekick vs general chat

Sidekick is task-bounded to admin API surface — general chit-chat disabled or routed away. Router first message classification saves tokens and reduces policy surface — link agentic RAG routing concepts.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Study public engineering patterns around Shopify: Merchant copilot. Rebuild the idea, not scraped proprietary text — focus on transferable architecture.

Diagram — Shopify pattern (conceptual)

flowchart LR
  Merchant-->Agent-->ShopData-->Action

Precise definitions & mental model

Extract 3 transferable patterns from Shopify's public tech narrative on Merchant copilot; map each to a ShipAI module artifact.

Tradeoffs — when to use what

Lens	Question
Scale	What breaks at 10× traffic?
Safety	Where are human/policy gates?
Cost	Where do tokens/GPUs dominate?

Failure modes (interview + on-call)

Cargo-culting brand names without metrics; inventing fake citations; cloning UI not architecture.

Production & OSS practices

Write a lab README: hypothesis, architecture diagram, eval, cost model, what you'd ask their eng in an interview.

Micro-project: Tool agent + judge eval loop

Ship:

1. Shopify reading notes.
2. Merchant tool agent with ≥4 tools and confirmation on one mutating action.
3. Judge rubric + ≥10 eval cases with report JSON.
4. tradeoffs.md vs demo agent.
5. One canary case tagged for merchant policy.

Acceptance: eval report shows pass rate; policy violation case scored fail on policy criterion.

Checklist

• Reading notes on production agent patterns
• Merchant agent with tool grounding
• Judge eval loop ≥10 cases
• tradeoffs.md documented
• Mutating action requires confirmation or authz