Tools (schema, validation, side effects)

Tools are APIs with blast radius

Agent tools are functions the model invokes — search, send email, update database, create ticket. Unlike chat text, tools change state and touch external systems. Engineering order: JSON Schema first, validate arguments in code, classify side-effect level, then expose to the model.

This lesson extends your ReAct loop with ≥3 real tools, strict validation, and separation between read-only and mutating operations.

Callout — validate before execute: The model hallucinates argument shapes. Pydantic/jsonschema rejection is a normal loop outcome, not an crash.

Tool definition anatomy

Each tool needs:

{
  "name": "create_support_ticket",
  "description": "Create a ticket when user confirms issue details.",
  "parameters": {
    "type": "object",
    "properties": {
      "title": {"type": "string", "maxLength": 120},
      "priority": {"enum": ["low", "medium", "high"]},
      "body": {"type": "string"}
    },
    "required": ["title", "body"]
  }
}

Descriptions steer model behavior — write them for the model, not humans skimming OpenAPI.

Implement:

class CreateTicketArgs(BaseModel):
    title: str = Field(max_length=120)
    priority: Literal["low", "medium", "high"] = "medium"
    body: str

def create_support_ticket(args: CreateTicketArgs) -> dict:
    ...

Map validation errors to tool result messages the model can read and retry.

Three tool categories

Tag tools in registry:

@tool(side_effect="mutating", requires_confirmation=True)
def send_email(...): ...

Loop driver checks tags before execute().

Side-effect safety patterns

Idempotency keys — pass request_id to prevent duplicate charges on retry.

Allowlists — email tool only sends to domains in session allowlist during dev.

Sandbox backends — SQLite instead of prod Postgres for course projects.

Rate limits — per-session caps on mutating calls.

Return structured results:

{"ok": true, "ticket_id": "T-9921"}
{"ok": false, "error": "validation", "details": "priority must be low|medium|high"}

Never return stack traces to the model in production — log internally, summarize externally.

Real tools for the micro-project

Pick a coherent mini-domain — support ops, dev productivity, personal finance (fake ledger):

Example set:

1. search_kb(query) — read-only, hits local JSON index
2. draft_reply(ticket_id, tone) — read-only generation helper
3. create_ticket(title, body, priority) — mutating, writes SQLite
4. (Optional) assign_ticket(ticket_id, agent) — mutating

Three minimum; four if you want dry-run preview_ticket.

Callout — descriptions are prompts: If create_ticket fires too eagerly, tighten description: "Only after user explicitly confirms creation."

Tool description engineering

Descriptions are mini-prompts. Improve weak tools by revising:

Before: "Searches the knowledge base."

After: "Search internal KB for policy answers. Use when user asks about refunds, shipping, or account settings. Input: short keyword query, not full sentences. Returns top 3 snippets with ids."

Add negative guidance: "Do not use for creating tickets — use create_ticket instead."

Review tool confusion matrix from logs: which tools get substituted for which — merge or clarify boundaries.

Timeouts and cancellation

Wrap each tool execution:

result = run_with_timeout(lambda: tool_fn(args), seconds=30)

Long-running search tools need cancellation tokens when user aborts session — stub with KeyboardInterrupt handling in CLI for course scope.

Tool versioning

When tool schema changes, version the name:

• create_ticket_v2 with new required field
• Keep create_ticket_v1 read-only during migration

Log tool_schema_version in trajectory — old runs replay against wrong schema if unversioned.

Models cache old argument habits — update descriptions highlighting new required fields.

Simulating side effects in tests

Use in-memory fake backends:

class FakeTicketStore:
    def __init__(self): self.tickets = []
    def create(self, title, body, priority):
        tid = f"T-{len(self.tickets)+1}"
        self.tickets.append({"id": tid, "title": title})
        return {"ok": True, "ticket_id": tid}

Swap real SQLite for fake in unit tests — agent integration tests use real SQLite in temp file.

Rate limiting tool calls

Per-session caps prevent runaway loops calling mutating tools:

if session.tool_calls.get(name, 0) >= LIMITS[name]:
    return {"ok": False, "error": "rate_limit", "retry": False}

Log rate-limit events in trajectory — distinct from validation failures.

Tool result size limits

Truncate large tool outputs before appending to messages:

def clamp(text: str, max_chars: int = 4000) -> str:
    return text if len(text) <= max_chars else text[:max_chars] + "\n...[truncated]"

Search tools returning 100k JSON blow context windows — agent loops fail mysteriously on step 2. Log original size when truncating.

Testing tools without the model

Unit test each tool:

• Valid args → expected side effect
• Invalid enum → validation error
• Mutating tool blocked when dry_run=True global flag set

Agent integration tests come later; tool tests catch most bugs cheaply.

OpenAI vs Anthropic tool formats

Abstract behind ToolRegistry.execute(name, raw_args) -> str so loop driver stays provider-agnostic. Conversion layer maps to each API's tool schema export.

Engineering problem (staff framing)

Tools are RPCs. Schema + validation + side-effect policy prevent agents from becoming exploit chains.

Diagram — Tool invocation path

sequenceDiagram
  participant M as Model
  participant D as Driver
  participant V as Validator
  participant T as Tool
  M->>D: tool_call JSON
  D->>V: schema check
  V-->>D: ok/fail
  D->>T: execute
  T-->>D: result
  D->>M: tool message

Precise definitions & mental model

JSON Schema, allowlists, authz scopes, dry-run vs mutating tools.

Tradeoffs — when to use what

Many fine tools vs few coarse — discoverability vs safety.

Failure modes (interview + on-call)

Trusting model JSON; path traversal args; unbounded shell.

Production & OSS practices

Pydantic validation; timeouts; audit log; human gate for mutate.

Deep dive (FAANG / OSS bar)

Side-effect classes

Classify every tool: read, write, money, egress, code_exec. Default deny for money/egress/code_exec without HITL. Validation is necessary but not sufficient — authz scopes matter.

Argument validation pattern

1. Parse JSON.
2. Validate against JSON Schema / Pydantic.
3. Canonicalize paths (resolve, then ensure under sandbox root).
4. Enforce size limits on strings/arrays.
5. Execute with timeout + CPU/memory limits if applicable.

Micro-project: ≥3 real tools

In m7/tools/:

1. Registry with ≥3 tools, JSON schemas, Pydantic validation.
2. Integrate with ReAct loop from 7.1.
3. Mark mutating vs read-only; mutating requires dry_run flag OR stub confirmation hook for 7.5.
4. Unit tests for each tool + one agent run log in README.

Checklist

• All tool args validated before side effects
• Mutating tools identifiable in registry
• Validation failures return model-readable errors
• Unit tests pass without LLM