Local Practice Checklist for Code Generation and Long-Task Iteration with Large Language Models
This article provides a practical, executable checklist for developers using large language models (LLMs) to generate code and automatically iterate on long tasks. It covers defining explicit acceptance criteria, setting up a closed-loop execution environment with rollback and command whitelists, designing a multi-agent role structure, establishing consistent cost and selection metrics, and running local test checks. The goal is to help developers build reproducible workflows where models can generate runnable projects and self-correct errors—without relying on unverified benchmark numbers.
Introduction
When choosing a large language model for code generation, developers typically care about two things: whether the model can generate a runnable project in one pass, and whether it can fix its own errors iteratively. This article breaks down these two concerns into actionable steps. The workflow and sample code provided here are reproducible templates. Performance metrics need to be measured in your own project; we do not present unverified numbers.
1. Define Scope and Acceptance Criteria
Before starting, write down three things: the task type, the executable environment, and the criteria for completion. Without acceptance criteria, a model may treat “the page opens” as success while actual functionality is missing.
Take a “financial report season data monitoring page” as an example:
- Project structure:
src/for components,data/sample.jsonfor sample data,tests/for test cases. - Runtime environment: Node.js 20 or above, Python 3.11 or above. Dependency versions must be pinned in
package.jsonandrequirements.txtto prevent the model from arbitrarily upgrading. - Startup command: run
npm install, thennpm run dev, and access the local port in a browser. - Acceptance criteria: the page loads sample data; the table can filter by quarter; the console shows no errors; replacing
sample.jsondoes not change the page structure.
Once acceptance criteria are written as testable statements, each iteration can be judged against a concrete baseline.
2. Closed-Loop Long-Task Execution: Execute, Feedback, Modify
The core of a long task is the loop: the model proposes an action, the local environment executes it, the result is fed back to the model, and the next iteration begins. Without boundaries, the loop can lead to accidental file deletion, infinite loops, or cost overruns. The following prerequisites are required:
- Run inside a container or virtual machine with a non-privileged user. Do not put production credentials in environment variables.
- Run
git commitbefore each execution so every change can be rolled back. - Use a command whitelist—only pre-approved commands are allowed.
- Set a timeout for each command (120 seconds in the example).
- Write logs from each iteration to
./agent_logs/for later review.
Below is a pseudocode template. llm_propose and apply_patch must be replaced with your SDK’s actual implementations.
import subprocessimport pathlibALLOWED_CMDS = {"npm test", "npm run build", "pytest -q"}LOG_DIR = pathlib.Path("./agent_logs")MAX_ITER = 5TIMEOUT_S = 120def run_cmd(cmd: str, cwd: str) -> tuple[bool, str]:if cmd not in ALLOWED_CMDS:raise PermissionError(f"Command not whitelisted: {cmd}")try:p = subprocess.run(cmd.split(), cwd=cwd,capture_output=True, text=True,timeout=TIMEOUT_S)except subprocess.TimeoutExpired:return False, f"Timeout after {TIMEOUT_S} seconds"return p.returncode == 0, p.stdout + p.stderrdef agent_loop(task: str, project_dir: str) -> str:LOG_DIR.mkdir(exist_ok=True)history = [f"Task: {task}"]for step in range(1, MAX_ITER + 1):plan = llm_propose(task, history) # returns a patch or a command to runif plan.kind == "patch":apply_patch(plan.diff, project_dir) # only allow modifications within project_dirok, output = run_cmd(plan.command or "npm test", project_dir)(LOG_DIR / f"step_{step}.log").write_text(output, encoding="utf-8")if ok:return "Success"history.append(f"Step {step} failed:\n{output[-2000:]}")return "Max iterations reached, not passed"
Key points in this code: only the last 2000 characters of the failure log are sent back to the model to keep the context length manageable; each round’s result is persisted to disk, making it easy to trace which step failed.
3. Multi-Agent Division of Labor: Roles and Message Formats
There is no standard for multi-agent collaboration. A practical structure uses three roles: Planner (breaks down the task), Coder (modifies code), and Tester (runs tests and provides judgment).
| Role | Input | Output | Prohibited Actions |
|---|---|---|---|
| Planner | Requirement document, current file tree | Subtask list, each with target file and acceptance condition | Does not directly modify code |
| Coder | Single subtask, relevant file content | Patch (diff) | Does not modify test cases |
| Tester | Patch, test command | Pass or fail, with failure summary | Does not modify business code |
Messages between roles are passed as JSON objects for easy validation:
{"from": "Coder","to": "Tester","task_id": "T-03","patch": "diff content","expected": "Row count after quarter filter matches sample"}
Every message carries a task_id so failures can be traced to a specific subtask. When the context becomes too long, only summaries and relevant files are passed, not the entire project.
4. Cost and Model Selection: Unified Measurement Criteria
Model prices and quotas change frequently; we do not provide specific numbers here. Use the following table to record your own measurements and compare models on identical tasks:
| Metric | Definition | Recording Method |
|---|---|---|
| Context length | Maximum token count per request | Read from SDK usage fields |
| Task cost | Total token cost to meet acceptance criteria | Calculate from billing unit price |
| Task duration | Time from first request to passing tests | Script captures timestamps |
| Failure retry count | Number of iterations before acceptance | Count files in agent_logs |
| First-pass rate | Percentage of tasks passing without manual correction | Run the same task at least 5 times |
Looking only at a single benchmark score or unit price is insufficient. Cost, duration, and retry count must be compared on the same task to reflect real productivity differences.
5. Test Checklist
Local tests should cover three dimensions. For each dimension, write down the input, expected behavior, and how to judge the result.
| Dimension | Test Input | Expected Behavior | Verification Method |
|---|---|---|---|
| Instruction following | Complex instructions requiring fixed JSON output | Format remains stable across rounds; all fields present | Validate against JSON Schema |
| Environment awareness | Project where dependency versions differ from actual package.json |
Reads actual versions; does not call non-existent APIs | Check against lock file |
| Error handling | Deliberately delete a dependency or make a test timeout | Catches the error, gives a fix direction, does not repeat the same action | Observe logs for repeated commands |
Design at least two test cases per dimension—one normal and one abnormal—to avoid testing only the happy path.
6. Risks and Boundaries
Local execution carries risks that must be addressed beforehand:
- Accidental file deletion or modification: use
gitto roll back and restrict write directories. - Sensitive information leakage: never place keys in code or prompts; filter tokens and passwords from logs.
- Dependency conflicts: install in a virtual environment and pin versions.
- Generated code quality: always manually review the diff before merging, especially code involving permissions, input validation, and database access.
A model passing tests does not guarantee security or business fit. Tests only cover the explicitly written acceptance criteria; uncovered edge cases still require human judgment.
7. Conclusion
Large language models are well-suited for development tasks that are decomposable and verifiable. To make them work reliably, you need clear acceptance criteria, a restricted execution environment, consistent measurement metrics, and a human review step. Run this checklist on your own project, and the data you collect will support model selection decisions better than any external benchmark.
