Code assist loops

Coding agents start as disciplined loops

Before frameworks and autonomous IDEs, coding assistance is a bounded loop around verifiable feedback: read failing tests, propose a patch, run the real test runner, repeat or stop. This pattern previews agent harness thinking you will formalize in later modules — stop conditions, tool sandboxes, regression fixtures — but stays small enough to ship in an afternoon.

The LLM is not the source of truth. Tests are. The model suggests edits; pytest (or your runner) decides success. That inversion keeps you honest when the model confidently breaks types or deletes assertions.

Callout — never trust green without running tests: Models hallucinate success messages. Your loop must shell out to the actual runner and parse exit codes.

Loop architecture

┌─────────────┐
│ failing tests│
└──────┬──────┘
       ▼
┌─────────────┐     ┌──────────────┐
│ gather context│──▶│ LLM proposes │
│ (trace, files)│     │ patch/diff   │
└─────────────┘     └──────┬───────┘
                           ▼
                    ┌──────────────┐
                    │ apply patch  │
                    └──────┬───────┘
                           ▼
                    ┌──────────────┐
                    │ run tests    │──fail──┐
                    └──────┬───────┘        │
                      pass │                │
                           ▼                │
                         STOP          increment step
                                        (max_steps?)

Minimum components:

1. Context builder — failing test names, stderr snippet, relevant source files (not entire repo).
2. Prompt pack — system instructions: minimal diff, preserve public API, no deleting tests.
3. Patch applier — apply unified diff or write files in a git worktree.
4. Test runner subprocess — pytest -x or targeted path.
5. Stop policy — max iterations (e.g. 5), no progress detection, cost cap.

Context budget matters

Dumping 40 files into the prompt wastes tokens and confuses the model. Heuristics:

• Include the failing test file and the module under test.
• Add stack trace frames up to depth 3.
• If import errors, include pyproject.toml or deps snippet.

Use ripgrep or AST helpers to pull symbols referenced in the traceback. Lesson 8.5 goes deeper on context budgets; apply the same discipline here.

Prompt design for code repair

Strong system prompt constraints:

You fix failing tests with the smallest change possible.
Output a unified diff only — no prose unless asked.
Never remove or weaken assertions.
Never add network calls or sleep in tests.
If you cannot fix in one diff, explain blockers in CHANGELOG line.

Few-shots with tiny before/after diffs teach format better than paragraphs of rules.

Version this prompt in your registry (lesson 4.3). Code-assist prompts drift fast when you change output format expectations.

Safety rails

Run inside a dedicated branch or git worktree add ../fix-loop so main stays clean.

Closing Milestone 4

This lesson completes the prompt engineering milestone. Your portfolio m4/ folder should contain:

• Versioned prompt pack(s) with registry
• Structured extractor or classifier with validation
• Eval set (cases.jsonl) with scored results
• This code-assist loop with logged trajectories

Write m4/README.md summarizing: tasks covered, best prompt version, eval accuracy, what broke on code-assist runs.

The milestone bar: a reviewer runs documented commands and reproduces eval scores within tolerance.

When not to use code-assist loops

Not every failing test should go to an LLM. Skip or block the loop when:

• Failures require architectural redesign, not local patches.
• Tests are flaky (timing, network) — fix determinism first.
• Security-sensitive code (crypto, auth) needs human review regardless.
• The diff touches more than N files — escalate to human with summary.

Encode these as guardrails in loop.py before calling the model. A pre-flight git diff --stat check that aborts when changed lines exceed 80 prevents runaway refactors.

Pair the loop with your prompt registry: tag code-fix prompts separately from classification prompts. They evolve at different cadences and should not share version ids.

Reviewing LLM-generated patches

Before applying any model-produced diff, enforce a human or CI review checklist even in automated loops:

• Does the change address the stated test failure or unrelated refactors creep in?
• Are new imports safe and licensed?
• Did the model weaken assertions (assert True) or skip tests?

Optional review_mode flag prints diff and waits for [y/n] before apply — mirrors HITL patterns from the agents module applied to code.

Measuring the loop

Log each iteration:

{
  "step": 2,
  "prompt_version": "code_fix/v1.0.0",
  "tests_failed": ["test_parser.py::test_null_date"],
  "patch_lines": 14,
  "pytest_exit": 1,
  "duration_s": 23.4
}

Analyze loops that exceed three steps — often the test is ambiguous or context was incomplete, not that the model is "bad at coding."

Engineering problem (staff framing)

Code assistants are agentic loops over repo tools with tests as oracle.

Diagram — Code assist loop

flowchart TD
  Goal --> Edit[Edit files]
  Edit --> Test[Run tests]
  Test -->|fail| Goal
  Test -->|pass| PR[Diff / PR]

Precise definitions & mental model

Repo grounding, test oracles, patch discipline, context budgets.

Tradeoffs — when to use what

Whole-file rewrite vs unified diff — simplicity vs reviewability.

Failure modes (interview + on-call)

Unbounded edits; ignoring failing tests; leaking secrets in context.

Production & OSS practices

Sandbox, allowlisted commands, CI as source of truth.

Micro-project: Fix failing tests with LLM

In m4/code_assist_loop/:

1. Start from a tiny Python package with intentionally failing tests (2–3 failures, one subtle).
2. Implement loop.py that calls your chat API, applies diffs, runs pytest, stops on pass or max_steps=5.
3. Save iteration logs to runs/*.jsonl.
4. Document one run that succeeded and one that hit max_steps in README.

Integrate prompt versioning and reference your eval habits from earlier lessons.

Checklist

• Tests invoked via subprocess, not model self-report
• max_steps enforced with clear log on exhaustion
• Prompt version logged each iteration
• Milestone 4 README ties prompt pack + eval + loop together