Chat APIs and message roles

The real-world interface

Most production AI features talk to models through chat APIs: ordered lists of messages, each tagged with a role. This is not a cosmetic wrapper around a single prompt string. Providers (OpenAI, Anthropic, Google, local servers via Ollama) all converged on the same pattern because multi-turn products need structured conversation state, and because separating standing instructions from user input improves both safety and debuggability.

When you call a chat endpoint, you typically send:

{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "You classify support tickets..."},
    {"role": "user", "content": "My refund never arrived"},
    {"role": "assistant", "content": "billing"},
    {"role": "user", "content": "Actually it was a duplicate charge"}
  ]
}

The model reads the full transcript and generates the next assistant message. Your application appends that output and continues the thread. This is the foundation for everything later in the course — structured outputs, prompt versioning, agents with tool roles.

Callout — roles are not suggestions: Treat system as policy and format contract, user as untrusted input, and assistant as model history you may replay. Mixing these up causes subtle bugs — for example, putting user text in system can weaken injection defenses.

Message roles in practice

Provider details differ — some APIs allow multiple system blocks, some collapse developer messages — but the mental model holds. Your job as an engineer is to construct the message list deliberately, not to dump strings into one blob.

System prompts: narrow and durable

A good system prompt answers three questions:

1. What is the task? One sentence scope, not a product manifesto.
2. What must never happen? Safety, privacy, refusal boundaries.
3. What shape should output take? Even before formal JSON schema (next lesson), specify labels, bullet limits, or "reply in one word."

Example for ticket classification:

You classify customer support messages into exactly one label:
billing, shipping, account, product_bug, other.
Reply with the label only, lowercase, no punctuation.
If uncertain, reply other.

Short system prompts are easier to version, diff in code review, and A/B test.

User and assistant turns

Multi-turn chat is powerful when the task genuinely requires clarification or follow-up. It is expensive when you could have solved the task in one shot. A useful heuristic: if your average conversation exceeds three user turns for a batch job, consider restructuring into a single user message with structured context.

When replaying assistant history, log whether the content came from the live model or from a cached response. Debugging "the model changed its mind" often traces to silently swapping cached assistant text from an older prompt version.

Prompt packs beat one-off strings

Shipping AI features from scattered string literals in application code fails quickly. Prompts become untraceable, untestable, and unreviewable. A prompt pack is a versioned directory that treats prompts like configuration artifacts:

m4/prompt_pack/
  TASK.md           # what success means, acceptance criteria
  system.md         # system prompt
  few_shots.jsonl   # input/output examples
  examples/         # edge-case notes, screenshots
  run.py            # sends pack to API, writes outputs/
  cases.jsonl       # eval inputs
  NOTES.md          # manual spot-check observations

Benefits stack up fast:

• Reproducibility: run.py reads the same files every time.
• Reviewability: Prompt changes appear in git diffs.
• Handoff: A teammate runs one command without hunting strings in Flask routes.
• Bridge to evals: cases.jsonl becomes your golden set in later modules.

Your run.py should log the full request payload (redacting secrets) and raw response to outputs/<timestamp>/. That log is the seed for prompt versioning in lesson 4.3.

Callout — separate task definition from system prompt: TASK.md is for humans — success metrics, label definitions, known ambiguities. system.md is for the model — imperative instructions only. Keeping them separate prevents bloated system prompts.

Calling the API responsibly

Before you scale calls, nail the basics:

Secrets: Load API keys from environment variables or a secrets manager. Never commit .env. Add .env to .gitignore on day one.

Errors: Handle rate limits (429), timeouts, and empty responses. Retry with exponential backoff for transient failures; fail loudly for auth errors.

Token budget: Count or estimate input tokens. A prompt pack with twenty few-shots can silently exceed context limits on smaller models.

Determinism vs creativity: For classification and extraction, use low temperature (0–0.3). For brainstorming assistants, higher temperature is fine — but that is a different product surface.

A minimal Python caller pattern:

import json, os, pathlib
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
pack = pathlib.Path("m4/prompt_pack")
system = (pack / "system.md").read_text()
messages = [{"role": "system", "content": system}]
for row in (pack / "few_shots.jsonl").read_text().splitlines():
    ex = json.loads(row)
    messages += [
        {"role": "user", "content": ex["input"]},
        {"role": "assistant", "content": ex["output"]},
    ]
# append live user message, call client.chat.completions.create(...)

Adapt for Anthropic or local OpenAI-compatible servers by swapping the client — keep the pack layout stable.

Logging for what comes next

Even before formal versioning, log these fields on every call:

• prompt_version (start with v0.1.0 or git SHA of the pack folder)
• model name and provider
• full messages array (post-redaction)
• raw completion
• latency_ms and token counts if available

When something breaks in production, "which prompt was live?" is the first question. If you cannot answer it, you cannot fix it. Lesson 4.3 turns this habit into a registry; start collecting data now.

Engineering problem (staff framing)

Chat APIs are stateful message arrays with roles. Wrong role discipline breaks tools and caching.

Diagram — Message roles

sequenceDiagram
  participant U as User
  participant S as System
  participant A as Assistant
  participant T as Tool
  S->>A: policy
  U->>A: request
  A->>T: tool_call
  T->>A: tool result
  A->>U: answer

Precise definitions & mental model

system/user/assistant/tool roles; multi-turn state; provider schema differences.

Tradeoffs — when to use what

Stateless request rebuild vs server-side session — control vs convenience.

Failure modes (interview + on-call)

Putting secrets in system prompts logged everywhere; role desync after tool errors.

Production & OSS practices

Normalize providers behind one Message type; redact logs.

Micro-project: Prompt pack

Pick a narrow task — classify support tickets into five labels, extract meeting action items, or tag internal doc sections. Avoid open-ended chat; you want crisp pass/fail cases.

Ship in m4/prompt_pack/:

1. system.md plus 3–5 few-shots in few_shots.jsonl covering easy and ambiguous inputs.
2. run.py that reads inputs from cases.jsonl and writes model outputs to outputs/.
3. Manual spot-check notes in NOTES.md — which cases passed, which failed, hypotheses why.

Run with one command documented in a short README (e.g. uv run python m4/prompt_pack/run.py).

Checklist

• Pack runs with one command
• Secrets not committed
• Cases cover at least one hard/ambiguous example
• Every run logs prompt version and raw responses