Local Practice Checklist for Code Generation and Long-Task Iteration with Large Language Models

    This article provides a practical, executable checklist for developers using large language models (LLMs) to generate code and automatically iterate on long tasks. It covers defining explicit acceptance criteria, setting up a closed-loop execution environment with rollback and command whitelists, designing a multi-agent role structure, establishing consistent cost and selection metrics, and running local test checks. The goal is to help developers build reproducible workflows where models can generate runnable projects and self-correct errors—without relying on unverified benchmark numbers.

    Introduction

    When choosing a large language model for code generation, developers typically care about two things: whether the model can generate a runnable project in one pass, and whether it can fix its own errors iteratively. This article breaks down these two concerns into actionable steps. The workflow and sample code provided here are reproducible templates. Performance metrics need to be measured in your own project; we do not present unverified numbers.

    1. Define Scope and Acceptance Criteria

    Before starting, write down three things: the task type, the executable environment, and the criteria for completion. Without acceptance criteria, a model may treat “the page opens” as success while actual functionality is missing.

    Take a “financial report season data monitoring page” as an example:

    • Project structure: src/ for components, data/sample.json for sample data, tests/ for test cases.
    • Runtime environment: Node.js 20 or above, Python 3.11 or above. Dependency versions must be pinned in package.json and requirements.txt to prevent the model from arbitrarily upgrading.
    • Startup command: run npm install, then npm run dev, and access the local port in a browser.
    • Acceptance criteria: the page loads sample data; the table can filter by quarter; the console shows no errors; replacing sample.json does not change the page structure.

    Once acceptance criteria are written as testable statements, each iteration can be judged against a concrete baseline.

    2. Closed-Loop Long-Task Execution: Execute, Feedback, Modify

    The core of a long task is the loop: the model proposes an action, the local environment executes it, the result is fed back to the model, and the next iteration begins. Without boundaries, the loop can lead to accidental file deletion, infinite loops, or cost overruns. The following prerequisites are required:

    1. Run inside a container or virtual machine with a non-privileged user. Do not put production credentials in environment variables.
    2. Run git commit before each execution so every change can be rolled back.
    3. Use a command whitelist—only pre-approved commands are allowed.
    4. Set a timeout for each command (120 seconds in the example).
    5. Write logs from each iteration to ./agent_logs/ for later review.

    Below is a pseudocode template. llm_propose and apply_patch must be replaced with your SDK’s actual implementations.

    1. import subprocess
    2. import pathlib
    3. ALLOWED_CMDS = {"npm test", "npm run build", "pytest -q"}
    4. LOG_DIR = pathlib.Path("./agent_logs")
    5. MAX_ITER = 5
    6. TIMEOUT_S = 120
    7. def run_cmd(cmd: str, cwd: str) -> tuple[bool, str]:
    8. if cmd not in ALLOWED_CMDS:
    9. raise PermissionError(f"Command not whitelisted: {cmd}")
    10. try:
    11. p = subprocess.run(cmd.split(), cwd=cwd,
    12. capture_output=True, text=True,
    13. timeout=TIMEOUT_S)
    14. except subprocess.TimeoutExpired:
    15. return False, f"Timeout after {TIMEOUT_S} seconds"
    16. return p.returncode == 0, p.stdout + p.stderr
    17. def agent_loop(task: str, project_dir: str) -> str:
    18. LOG_DIR.mkdir(exist_ok=True)
    19. history = [f"Task: {task}"]
    20. for step in range(1, MAX_ITER + 1):
    21. plan = llm_propose(task, history) # returns a patch or a command to run
    22. if plan.kind == "patch":
    23. apply_patch(plan.diff, project_dir) # only allow modifications within project_dir
    24. ok, output = run_cmd(plan.command or "npm test", project_dir)
    25. (LOG_DIR / f"step_{step}.log").write_text(output, encoding="utf-8")
    26. if ok:
    27. return "Success"
    28. history.append(f"Step {step} failed:\n{output[-2000:]}")
    29. return "Max iterations reached, not passed"

    Key points in this code: only the last 2000 characters of the failure log are sent back to the model to keep the context length manageable; each round’s result is persisted to disk, making it easy to trace which step failed.

    3. Multi-Agent Division of Labor: Roles and Message Formats

    There is no standard for multi-agent collaboration. A practical structure uses three roles: Planner (breaks down the task), Coder (modifies code), and Tester (runs tests and provides judgment).

    Role Input Output Prohibited Actions
    Planner Requirement document, current file tree Subtask list, each with target file and acceptance condition Does not directly modify code
    Coder Single subtask, relevant file content Patch (diff) Does not modify test cases
    Tester Patch, test command Pass or fail, with failure summary Does not modify business code

    Messages between roles are passed as JSON objects for easy validation:

    1. {
    2. "from": "Coder",
    3. "to": "Tester",
    4. "task_id": "T-03",
    5. "patch": "diff content",
    6. "expected": "Row count after quarter filter matches sample"
    7. }

    Every message carries a task_id so failures can be traced to a specific subtask. When the context becomes too long, only summaries and relevant files are passed, not the entire project.

    4. Cost and Model Selection: Unified Measurement Criteria

    Model prices and quotas change frequently; we do not provide specific numbers here. Use the following table to record your own measurements and compare models on identical tasks:

    Metric Definition Recording Method
    Context length Maximum token count per request Read from SDK usage fields
    Task cost Total token cost to meet acceptance criteria Calculate from billing unit price
    Task duration Time from first request to passing tests Script captures timestamps
    Failure retry count Number of iterations before acceptance Count files in agent_logs
    First-pass rate Percentage of tasks passing without manual correction Run the same task at least 5 times

    Looking only at a single benchmark score or unit price is insufficient. Cost, duration, and retry count must be compared on the same task to reflect real productivity differences.

    5. Test Checklist

    Local tests should cover three dimensions. For each dimension, write down the input, expected behavior, and how to judge the result.

    Dimension Test Input Expected Behavior Verification Method
    Instruction following Complex instructions requiring fixed JSON output Format remains stable across rounds; all fields present Validate against JSON Schema
    Environment awareness Project where dependency versions differ from actual package.json Reads actual versions; does not call non-existent APIs Check against lock file
    Error handling Deliberately delete a dependency or make a test timeout Catches the error, gives a fix direction, does not repeat the same action Observe logs for repeated commands

    Design at least two test cases per dimension—one normal and one abnormal—to avoid testing only the happy path.

    6. Risks and Boundaries

    Local execution carries risks that must be addressed beforehand:

    • Accidental file deletion or modification: use git to roll back and restrict write directories.
    • Sensitive information leakage: never place keys in code or prompts; filter tokens and passwords from logs.
    • Dependency conflicts: install in a virtual environment and pin versions.
    • Generated code quality: always manually review the diff before merging, especially code involving permissions, input validation, and database access.

    A model passing tests does not guarantee security or business fit. Tests only cover the explicitly written acceptance criteria; uncovered edge cases still require human judgment.

    7. Conclusion

    Large language models are well-suited for development tasks that are decomposable and verifiable. To make them work reliably, you need clear acceptance criteria, a restricted execution environment, consistent measurement metrics, and a human review step. Run this checklist on your own project, and the data you collect will support model selection decisions better than any external benchmark.