Rubric: tests, eval JSON, and README

What "good" looks like in ShipAI

Demos without evals do not pass milestones. This is intentional. The AI industry spent years shipping chatbots that impressed in live demos and failed on edge cases in production. ShipAI trains the opposite reflex: measure first, demo second.

From the setup module onward, every non-trivial build aims at a triple of artifacts:

You will refine eval methodology deeply in the evals and guardrails module. The habit starts now with a trivial golden fixture — the complexity scales later; the pattern does not.

Why tests still matter in the LLM era

Large language models are nondeterministic at non-zero temperature, so you cannot pytest an LLM's exact wording. You can pytest:

• JSON schema validation on tool outputs
• Parser behavior on fixed strings
• Retrieval ranking on a frozen corpus
• Prompt template rendering
• Cost/token logging wrappers

Golden fixtures live in the test layer. LLM quality lives in the eval JSON layer. Both are required for production-shaped work.

What belongs in a README

A ShipAI README is not marketing copy. Minimum sections:

1. What this is — one paragraph.
2. How to run — copy-paste commands, env vars required.
3. Results — link to metrics or eval JSON, summarize key numbers.
4. Failures and limits — what broke, what you skipped, known bugs.
5. Next steps — what you would do with another day or a bigger GPU.

Reviewers trust engineers who document failure modes. "Accuracy is 94% on digits but confuses 8 and 3" beats "model works great."

Callout — evals ≠ unit tests: Unit tests assert exact behavior on fixed inputs. Evals sample model behavior over a dataset and aggregate scores. You need both; conflating them leads to either brittle LLM tests or untested parsers.

Golden fixtures

A golden fixture is a checked-in input/expected pair used for regression testing. When you change a parser, prompt template, or post-processing function, golden tests tell you immediately if behavior drifted.

Anatomy of a fixture

Store fixtures as JSON (or YAML) under a predictable path:

m0/fixtures/golden/
  hello.json
  expense-001.json   # example for a future extractor lesson

Example fixture for a future JSON extractor:

{
  "id": "expense-001",
  "input": {"text": "Coffee 4.50 USD at Cafe Roastery"},
  "expected": {"amount": 4.5, "currency": "USD", "merchant": "Cafe Roastery"}
}

Example fixture for this lesson (minimal):

{
  "id": "hello-001",
  "input": {"message": "  ShipAI Ready  "},
  "expected": {"normalized": "shipai ready"}
}

Loading fixtures in pytest

# tests/test_golden_hello.py
import json
from pathlib import Path

from m0.fixtures.normalize import normalize_message  # your trivial function

FIXTURE = Path("m0/fixtures/golden/hello.json")

def test_golden_hello():
    data = json.loads(FIXTURE.read_text())
    result = normalize_message(data["input"]["message"])
    assert result == data["expected"]["normalized"]

The function under test can be trivial — lowercase and strip whitespace. The point is the pattern: fixture on disk, pure function, deterministic assert, CI-friendly.

Later modules reuse this layout for RAG answer checks (expected citations), tool-call parsing, and agent trajectory validation.

Eval JSON shape (preview)

Your first LLM call already produced JSONL logs. Eval artifacts extend that idea:

{
  "run_id": "m0-smoke-2026-08-11",
  "model": "gpt-4.1-mini",
  "cases": [
    {"id": "exact-reply", "pass": true, "notes": "matched shipai-ready"}
  ],
  "aggregate": {"pass_rate": 1.0}
}

Full eval harnesses arrive in later modules. For Milestone 0, the golden test plus call log satisfy the spirit: structured, diffable output.

Engineering problem (staff framing)

Probabilistic systems need a definition of done: unit tests where deterministic, golden eval JSON where not, READMEs that admit fails.

Diagram — Done criteria

flowchart TD
  Run[Code runs] --> Unit[Schema / unit tests]
  Unit --> Gold[Golden eval JSON]
  Gold --> README[README + limits]
  README --> Pass[Milestone]

Precise definitions & mental model

Rubric, golden fixtures, machine-diffable eval JSON, honest failure notes.

Tradeoffs — when to use what

Failure modes (interview + on-call)

Screenshot-only proof; exact string match on LLM text; README claims contradict eval files.

Production & OSS practices

Version prompts with evals; nightly model jobs OK; CPU schema tests in PR CI.

Deep dive (FAANG / OSS bar)

Push «rubric-tests-eval-readme» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: First golden fixture

1. Add m0/fixtures/golden/hello.json with a simple input/expected object (see example above).
2. Implement a trivial pure function (e.g., normalize_message) in m0/fixtures/normalize.py.
3. Add tests/test_golden_hello.py that loads the fixture and asserts the function output matches expected.
4. Ensure make test or pytest passes from repo root (add a Makefile target if helpful).
5. Update root README with Milestone 0: Lab ready and links to make doctor, first call, and fixture test.

Commit message suggestion: feat(m0): golden fixture and milestone 0 README.

Milestone 0 — Lab ready

You are done with the setup module when all of the following hold:

• course-portfolio exists with init README and module checklist
• make doctor passes and report is committed
• First LLM call logged to JSONL with token and cost fields
• Golden fixture + pytest pass
• m0/README.md summarizes the lab (doctor, first call, fixture, budget)

Ship it. The next module maps the AI field from symbolic systems through transformers so you know why you are building what you will build.

Callout — milestone before perfection: If your cost estimate is off by 20%, ship anyway and note the approximation. If Docker is missing, ship anyway with docker: false in the doctor report. Milestones reward complete loops, not flawless loops.

Checklist

• Golden fixture committed under m0/fixtures/golden/
• pytest passes for golden test
• Root README marks Milestone 0 complete with links
• m0/README.md summarizes all setup artifacts