Safety, regulation, dual-use

Engineers still need a threat model when lawyers own compliance

AI regulation (EU AI Act framing, sector rules for finance/health, export controls on dual-use models) evolves faster than university syllabi. Engineers implement controls — logging, human oversight, access tiers, eval evidence — while legal counsel interprets obligations. This lesson builds case memo habit: scenario → risks → engineering mitigations → open legal questions.

Not legal advice. Engineering practice with explicit boundaries.

Pick a realistic scenario

Choose one aligned with your capstone or interest:

• Customer support agent with access to refunds (financial harm)
• Internal code agent with production repo write (security dual-use)
• Health-adjacent triage bot (high-risk classification in some jurisdictions)
• Synthetic media feature (fraud/deepfake concerns)
• Cross-border data routing (privacy regulation)

One page scenario narrative with actor, data, model, tools, harm scenario.

Callout — Dual-use: Capability legitimate for user A (security audit) enables harm for user B (unauthorized intrusion). Policy must bind identity + intent + environment, not feature name.

Case memo structure

safety/case_memo.md:

1. Scenario summary
2. Stakeholders & harm model — who gets hurt, how, likelihood
3. Regulatory touchpoints (non-authoritative) — which regimes might apply; flag legal review
4. Engineering controls — concrete, mapped to your stack
5. Eval & monitoring evidence — what artifacts prove diligence
6. Residual risk & open questions
7. Disclaimer — engineering memo, not legal opinion

Engineering controls catalog

Map controls to lessons you built:

Control	Implementation
Access tiering	Uber mock authz
Human-in-loop	approval on mutating tools
Audit logs	tool allow/deny JSONL
Red-team	injection suite ASR
Guardrails	PII/policy middleware
Retention limits	privacy sketch TTLs
Model routing	gateway restrict frontier for risky routes
Transparency	uncertainty UX citations

Pick ≥6 applicable to scenario with priority (P0/P1).

Dual-use specific patterns

• Capability tokens — feature enabled per contract, not globally
• Environment separation — prod write tools unavailable in untrusted workspaces
• Rate limits on sensitive exports
• Watermarking on generated media (if relevant)

Document abuse monitoring — spike in blocked guardrail events.

Separating legal from engineering

Engineering delivers: data flow diagram, control checklist, eval pass rates, incident runbook.

Legal delivers: classification, contractual language, regulatory filings.

Memo ends with questions for counsel bulleted — shows professional boundary.

EU AI Act literacy (non-legal)

High-risk categories (biometrics, critical infra, etc.) may impose logging and human oversight — case memo maps scenario to questions for counsel, not self-classification. Engineering prepares evidence: eval logs, audit trails, HITL records.

Incident response for AI harm

If agent outputs harmful instruction in prod, runbook: disable feature flag, preserve trace, notify legal/comms — case memo references runbook stub even if capstone never launches publicly.

Export controls awareness

Dual-use models and weights may face export restrictions — one bullet in memo if scenario crosses borders; flag legal review.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Safety is engineering + policy. Map requirements to controls and evidence.

Diagram — Control mapping

flowchart LR
  Reg[Requirement] --> Ctrl[Control] --> Ev[Evidence/logs]

Precise definitions & mental model

Threat models, prohibited uses, audit trails.

Tradeoffs — when to use what

Over-block vs under-protect.

Failure modes (interview + on-call)

Paper policies without technical enforcement.

Production & OSS practices

Safety case docs; red-team cadence.

Micro-project: Case memo

Deliver complete case_memo.md for chosen scenario plus:

1. Architecture diagram with trust boundaries marked.
2. Control table ≥6 items with P0/P1.
3. Link to existing portfolio artifacts (guardrails, privacy sketch).
4. Red-team or eval metric proposed as compliance evidence.
5. Disclaimer section verbatim at top or bottom.

Acceptance: engineer reader knows what to build; legal reader knows what to review.

Checklist

• Scenario narrative committed
• Harm model and stakeholders documented
• ≥6 engineering controls with priorities
• Legal review questions listed
• Disclaimer present; not presented as legal advice