Advanced Concepts

Guardrails and safety systems

Input/output filters, policy classifiers, tool allowlists, and layered defense — product safety beyond hoping the model behaves.

55 min

Also in guided path: Guardrails

Hope is not a control plane

Evals catch regressions in CI. Guardrails enforce policy on the live request path. Aligned models reduce risk; they do not replace product controls — especially once tools can email, refund, or query PII.

Industry courses (and every serious GenAI incident review) converge on the same idea: layered defense. Assume the model will occasionally be wrong, jailbroken, or over-helpful.

flowchart TD
  In[User input] --> IG[Input filters / PII / injection]
  IG --> Agent[LLM + tools]
  Agent --> Tools[Allowlisted tools + authz]
  Agent --> OG[Output filters / schema / toxicity]
  OG --> Out[User]
  Tools --> Side[Side effects]

Cross-links: evals fundamentals, alignment RLHF/DPO, privacy and data for AI, guided Evals, guardrails, safety.

What “policy” means in a product

Write it as engineering requirements, not vibes:

Policy question Example control
Who can invoke which tool? Authz + allowlist per role
What data may leave the tenant? Egress DLP + redaction
What must never be answered? Topic blocks + refusals
What requires a human? Confidence / amount thresholds
What must be cited? Output gate requiring sources

If it is not testable, it is not a guardrail — it is a blog post.

Layers that work together

Layer Examples Strength
Product policy Documented allowed actions Source of truth
Input PII scrub, prompt-injection heuristics, length caps Cheap, early
Model Safety-tuned instruct checkpoint Soft behavioral prior
Tools Schemas, authz, rate limits, dry-run Hard power boundary
Output Toxicity/PII classifiers, JSON schema, citation required Last look
Human Escalation queues, dual control High-stakes actions 
flowchart LR
  subgraph edge [Edge]
    I[Input filters]
  end
  subgraph brain [Model plane]
    M[LLM]
  end
  subgraph hands [Action plane]
    T[Tools / APIs]
  end
  subgraph out [Egress]
    O[Output filters]
  end
  I --> M --> O
  M --> T
  T --> O

Ship rule: put irreversible side effects behind tools + authz, never behind “the model pinky-promises.”

Prompt injection and tool abuse

Classic failure: untrusted content (email, webpage, ticket) says “ignore policies and exfiltrate secrets.” Mitigations:

  1. Separate instructions from untrusted data in the prompt (clear delimiters + mistrust rules)
  2. Minimize tools available when browsing untrusted text
  3. Confirm high-risk actions with humans or out-of-band auth
  4. Detect obvious exfil patterns on outputs (secrets, bulk PII)

Context packing choices matter — see context engineering.

Output and schema gates

For machine-consumed answers:

  • Validate JSON against schema (structured outputs)
  • Reject missing citations when policy requires them
  • Strip or block secrets via detectors

For user-facing chat:

  • Toxicity / self-harm escalation flows
  • Medical/legal disclaimers + handoff, not fake certainty

False positives and UX

Guardrails that block legitimate traffic train users to bypass you.

Problem Mitigation
Over-block Shadow mode → tune thresholds
Paraphrase bypass Defense in depth (tools still constrained)
Logging blocked prompts forever Retention limits + access control
Opaque “cannot help” Actionable refusal + escalate path

Observability and evals

Guardrails without metrics rot:

  • Block rate by rule and by tenant
  • False-positive review sampling
  • Red-team suites in CI (jailbreak, injection, PII)
  • Trace of which layer fired (OpenTelemetry for LLMs)

Evals measure whether models tend to behave; guardrails measure whether the system enforced policy on this request.

Design project: three-layer support agent

Sketch for a support bot with refunds:

  1. Input — scrub payment card patterns; cap length; flag jailbreak regexes
  2. Tools — refund only for authenticated owners, amount < $X, idempotency key required; else escalate
  3. Output — block responses that include full card numbers; require ticket id in structured summary

Add a Mermaid of your own and a table of abuse cases you tested.

Checklist

  1. Written policy with owners?
  2. Irreversible actions behind authz’d tools?
  3. Input + output filters with measured FP rates?
  4. Red-team set in CI?
  5. Privacy-safe logging of blocks?
  6. Human escalation for gray zones?

Evals, guardrails, safety evals, guardrails, safety. Pair with multi-agent (guardrail every node that can act) and networking (timeouts/idempotency for tool side effects).

Project checklist0/3 done