Cost accounting

Agents multiply cost — accounting is a feature

A single agent turn may invoke: router LLM, main LLM ×3, embedding calls, judge model, tool APIs billed per call. Without per-request cost accounting, finance discovers the bill before engineering discovers the loop bug.

Treat cost like latency — measure every request, tag dimensions, aggregate, alert.

Per-request logging schema

Emit one cost record per user-facing request (or per run_id for durable jobs):

{
  "request_id": "uuid",
  "run_id": "uuid",
  "route": "support/refund",
  "model_calls": [
    {"model": "gpt-4o-mini", "input_tokens": 800, "output_tokens": 120, "usd": 0.00034}
  ],
  "embedding_tokens": 400,
  "tool_usd": 0.001,
  "total_usd": 0.00134,
  "latency_ms": 2340
}

Compute USD from a price table in config (pricing.yaml) — update when providers change rates; version the table.

Include attribution dimensions your PM cares about: customer_tier, feature_flag, environment.

Callout — Estimated vs invoiced: Provider dashboards lag; internal logs are real-time estimates. Reconcile weekly; drift >5% triggers pricing table audit.

Rolling up daily cost

Batch job or SQL query:

SELECT date, route, model, sum(total_usd), count(*)
FROM cost_log
GROUP BY 1,2,3

Surface in markdown report or Grafana — course portfolios can commit reports/cost_YYYY-MM-DD.md from a script.

Watch tail spend: p99 cost per request often reveals runaway tool loops before mean moves.

Alert thresholds

Define static and anomaly alerts:

• Hard cap — single request > $X → log cost_anomaly + optional kill switch
• Daily budget — staging env max $Y/day, block non-essential evals
• Spike — 2× 7-day rolling average on route research/*

Alerts go to Slack/email stub — document in runbook.

Cost-aware product patterns

Engineering levers tied to accounting data:

• Route simple intents to smaller model when router confidence high
• Cache embeddings and prompt prefixes (lesson 11.3)
• Cap max_steps and judge frequency in CI vs prod

Show one cost regression story in README: change X increased mean $/req by Z%.

Privacy in cost logs

Do not store full prompts in cost tables — IDs and hashes only. Join to trace store on debug.

Unit economics for product decisions

Translate total_usd into unit economics: cost per successful task, cost per active user per day, margin if product is paid. Example: support agent at $0.04/success × 10k tickets/month = $400 inference — compare to human handle time savings. Numbers anchor build-vs-buy conversations from synthesis module.

Anomaly detection on spend

Simple rules before ML: flag requests where total_usd > 10× rolling median for route; flag accounts exceeding daily cap; flag loops where model_calls.length > 20. Tie anomalies to trace_id for immediate replay — often reveals missing max_steps or runaway judge loop.

FinOps handoff

Monthly report template for finance: total spend by environment, top routes by cost, forecast from weekly growth rate. Engineering owns accuracy of estimates; finance owns budget — document reconciliation SLA in runbook.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Attribute token spend to features/customers or margins die silently.

Diagram — Cost attribution

flowchart LR
  Req --> Feat[feature_id]
  Feat --> Cust[customer_id]
  Cust --> Bill[Cost ledger]

Precise definitions & mental model

Token prices, cached tokens, tool overhead, unit economics.

Tradeoffs — when to use what

Gross tokens vs successful-task cost.

Failure modes (interview + on-call)

No feature tags; ignoring embedding/rerank costs.

Production & OSS practices

Ledger + budgets per tenant; anomaly detection.

Micro-project: Per-request cost log

Ship:

1. Instrument all LLM/embedding calls with token counts.
2. pricing.yaml + USD computation per request.
3. JSONL or SQLite cost_log append.
4. Script producing daily rollup by route/model.
5. Document one alert threshold and trigger a test alert.

Acceptance: three sample requests show different routes/models in rollup; README explains reconciliation.

Checklist

• Per-request cost JSON with model breakdown
• pricing.yaml versioned in repo
• Daily rollup script committed
• Alert threshold documented and testable
• No raw prompts in cost log tables