Injection / jailbreaks

Security evals are product evals when tools have side effects

Prompt injection and jailbreaks are not only "security team" problems. If your agent reads untrusted documents, accepts pasted email bodies, or calls mutating tools, attack success rate (ASR) is a product metric — alongside latency and task success.

Red-teaming here means systematic adversarial inputs in evals, not one-off ChatGPT trick prompts.

Threat model for your agent

Enumerate sources of untrusted text:

• Retrieved wiki / ticket / webpage content
• User messages (obvious)
• Tool return payloads (second-order injection)
• Email or PDF uploads
• Other users' data in multi-tenant retrieval (cross-tenant leak)

For each, ask: what is the forbidden action if injection succeeds?

Examples:

• Exfiltrate system prompt or secrets via send_email tool
• Issue refund without policy check
• Ignore verify harness and claim tests passed

Document in threat_model.md — one page minimum.

Attack categories for red-team suite

Start with 20–40 templated cases; vary encoding (Unicode, markdown hidden text, JSON in PDF).

Callout — ASR is a rate, not a vibe: Track % runs where forbidden_action occurred. Trend down with guardrails; never expect zero on frontier models.

Building the red-team runner

Reuse eval infrastructure:

{"id": "inj-07", "input": "...", "untrusted_doc": "...", "forbidden": {"tools": ["send_email"], "output_contains": ["API_KEY"]}}

Runner asserts forbidden did not happen. Attack succeeds if assertion fails.

Log attack class, bypass technique tag, model version — feeds guardrail prioritization.

From failures to guardrails

When red-team fails:

1. Input — sanitize/truncate untrusted blocks; delimit with UNTRUSTED_START/END.
2. Tool — authz check independent of model; human approval on sensitive tools.
3. Output — DLP scan for secrets/PII before return.
4. Retrieval — strip HTML comments, zero-width chars at index time.

Next lesson implements middleware; this lesson feeds the case list.

Responsible red-team scope

Run against your staging agent with synthetic secrets — never production customer data or real keys. Do not publish working exploits for third-party services. Course portfolio stays local/staging.

Continuous red-team cadence

ASR baseline once is insufficient. Schedule weekly red-team run on staging with new cases from OWASP LLM cheat sheet adaptations and internal near-misses. Track ASR trend; guardrail changes should monotonically decrease critical ASR or require risk acceptance doc.

Tool-specific injection

Second-order: malicious content in retrieved doc instructs model to call send_email. Test with poisoned corpus documents in staging index — separate from user message jailbreaks. Mitigation: retrieval sanitization + tool authz independent of model intent.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

Untrusted text in context is an attacker. Treat retrieval/user content as hostile.

Diagram — Injection surfaces

flowchart TD
  User --> Prompt
  RAG[Retrieved docs] --> Prompt
  Tools --> Prompt
  Prompt --> Model
  Model --> Leak[Data leak / override]

Precise definitions & mental model

Prompt injection, jailbreak, indirect injection via docs.

Tradeoffs — when to use what

Utility of browsing/RAG vs attack surface.

Failure modes (interview + on-call)

Concatenating untrusted text into system; tool args from raw model text.

Production & OSS practices

Delimitering, allowlists, detectors, dual-model patterns sparingly.

Micro-project: Red-team suite

Ship:

1. redteam/ suite ≥25 cases covering ≥4 categories.
2. ASR report from baseline agent (expect nonzero — document honestly).
3. threat_model.md linking cases to forbidden actions.
4. Top 3 failures mapped to planned guardrails (lesson 10.5).
5. Wire red-team smoke (5 cases) into CI as blocking on critical exfil/refund cases.

Acceptance: ASR baseline recorded; CI blocks on critical subset failures after you add stub guardrails or document accepted risk.

Checklist

• threat_model.md with untrusted sources listed
• Red-team suite ≥25 cases runnable
• ASR baseline metrics committed
• Failures mapped to guardrail backlog
• Critical subset in CI or documented gate command