Long-Horizon Autonomous Coding with LLMs: License Compliance, Evaluation, and Multi-Agent Design

    This guide covers the practical decisions teams need to make before using large language models for long-horizon autonomous coding: verifying open-source model licenses, building a project-specific evaluation set, defining task specifications with checkpoints and rollback, and designing multi-agent workflows with file ownership, message deduplication, retry limits, and audit logging. The code examples are structural illustrations, not production-ready code, and legal review is still required.

    Scope and Boundaries

    This article is for development teams responsible for model selection and deployment. It discusses four areas when using LLMs for long-horizon autonomous coding:

    • Open-source model license checks
    • Evaluation methodology for coding ability
    • Task specifications, checkpoints, and rollback
    • Multi-agent decomposition and fault tolerance

    The code examples are included to illustrate structure, not for direct production use. They must be adapted to your team’s permission model and audit requirements.

    This article does not cover model training, inference acceleration, or vendor price comparison. License terms and model versions change, so every checklist item should be verified against the current text supplied by legal counsel.

    Open-Source Model License Checklist

    Before choosing a model, record each license term in the table below. Include the license text version and review date for every entry so the check can be traced later.

    Check item Question to confirm Record
    Commercial use Is use in a paid product allowed? Are there user-scale or revenue thresholds? Clause number, threshold value
    Distribution and derivatives Can fine-tuned or quantized weights be redistributed? Must original copyright notices be retained? Attribution and notice requirements
    Source attribution Does the product interface need to identify the model source? Placement and wording
    Data and privacy Do inference logs, caches, or debug information contain sensitive content? Log retention period, redaction rules
    Liability Does the license disclaim warranty? Internal output review owner
    License changes Does a new version change the license? Version number, change notification channel

    License changes need separate monitoring. Pin the model version in your dependency list, and re-run the license check before upgrading.

    Output review cannot be skipped. Open-source licenses usually disclaim warranty, so code changes still need human review and automated tests before merge. Compliance judgment should be made jointly by legal and technical owners. This article is not formal legal advice.

    Coding Ability Evaluation Set

    A single score does not explain how a model will perform on your team’s projects. Build an evaluation set from your own historical issue tracker. It should cover at least six types of tasks:

    • Real defect fixes
    • Terminal and tool operations
    • Codebase retrieval and search
    • Multi-language and cross-stack work
    • Performance optimization
    • Frontend generation

    Constructing Samples

    Select reproducible defects from the last year of issue history. Each sample should include an issue description, reproduction steps, code snapshots before and after the fix, and the regression tests that pass. Remove any keys and personal data from the samples.

    Fixed Runtime Conditions

    When evaluating, fix the random seed and sampling temperature. Record the full prompt, tool-call trace, and return values at every step. Otherwise the same model may produce very different results at different times, making comparisons unreliable.

    Scoring Approach

    Use a combination of pass rate and manual review. The pass rate covers automatically verifiable outcomes, such as whether tests pass and the build succeeds. Manual review covers readability, naming conventions, and potential side effects, such as changes to unrelated files.

    Sample ID Auto pass Diff scope reasonable Naming and style Side effects Reviewer
    BUG-0231 Yes Yes Pass None Jane Doe

    The IDs and names in this table are only illustrative. Keep the original records for every evaluation run so any conclusion can be traced back to a specific trace.

    Long-Horizon Tasks: Specifications, Checkpoints, and Rollback

    The main risk in long-horizon tasks is drift. The model may forget constraints after many steps, or repeatedly modify one local issue. The engineering approach is to write the goal explicitly, save state after every step, and return to a known good state when something fails.

    Task Specification Template

    Before starting, write the goal, constraints, and acceptance criteria into a structured file. The model reads this file every round, and a human can also use it to review behavior.

    1. {
    2. "goal": "Fix a rounding error in the order module",
    3. "non_goals": [
    4. "Do not modify the database schema",
    5. "Do not upgrade dependencies"
    6. ],
    7. "allowed_dirs": ["src/order/", "tests/order/"],
    8. "forbidden_dirs": ["vendor/", "secrets/", "migrations/"],
    9. "test_command": ["python", "-m", "pytest", "-q", "tests/order/"],
    10. "max_steps": 30,
    11. "rollback_point": "agent/checkpoint-0"
    12. }

    test_command must be written by a human, not generated by the model. This prevents the model from passing acceptance with an unrelated command.

    Checkpoints and State Persistence

    After each step, write state to disk. Write a temporary file first, then replace it atomically, so an interrupted process does not leave a truncated file.

    1. import json
    2. from pathlib import Path
    3. STATE_FILE = Path("agent_state/checkpoint.json")
    4. def save_state(state: dict) -> None:
    5. STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
    6. tmp = STATE_FILE.with_suffix(".tmp")
    7. tmp.write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding="utf-8")
    8. tmp.replace(STATE_FILE) # atomic replacement on the same filesystem
    9. def load_state(spec: dict) -> dict:
    10. if STATE_FILE.exists():
    11. return json.loads(STATE_FILE.read_text(encoding="utf-8"))
    12. return {"step": 0, "spec": spec, "history": []}

    Tool Calls and Main Loop

    The model may call only whitelisted tools. Path checks occur before execution. Requests outside the allowed directories are rejected, and the rejection reason is returned to the model. File contents and tool output are treated as data. Even if they contain text like “ignore previous instructions,” that text must not be treated as an instruction.

    1. import subprocess
    2. from pathlib import Path
    3. ALLOWED_TOOLS = {"read_file", "run_tests", "finish"}
    4. def is_path_allowed(rel_path: str, spec: dict) -> bool:
    5. root = Path.cwd().resolve()
    6. target = (root / rel_path).resolve()
    7. if root not in target.parents:
    8. return False
    9. rel = target.relative_to(root).as_posix()
    10. if any(rel.startswith(d) for d in spec["forbidden_dirs"]):
    11. return False
    12. return any(rel.startswith(d) for d in spec["allowed_dirs"])
    13. def execute(action: dict, spec: dict) -> dict:
    14. tool = action.get("tool")
    15. if tool not in ALLOWED_TOOLS:
    16. return {"error": f"Tool not in whitelist: {tool}"}
    17. if tool == "read_file":
    18. path = action.get("path", "")
    19. if not is_path_allowed(path, spec):
    20. return {"error": "Path is not in an allowed directory"}
    21. text = Path(path).read_text(encoding="utf-8")
    22. return {"content": text[:8000]}
    23. if tool == "run_tests":
    24. # The command comes from the spec, not from model output
    25. try:
    26. proc = subprocess.run(
    27. spec["test_command"], capture_output=True, text=True, timeout=600
    28. )
    29. except subprocess.TimeoutExpired:
    30. return {"error": "Test execution timed out"}
    31. return {
    32. "returncode": proc.returncode,
    33. "stdout": proc.stdout[-4000:],
    34. "stderr": proc.stderr[-4000:],
    35. }
    36. return {"ok": True}
    37. def run_agent(next_action, spec: dict) -> dict:
    38. state = load_state(spec)
    39. while state["step"] < spec["max_steps"]:
    40. # Call the model with only recent history to control prompt length
    41. action = next_action(state["history"][-10:])
    42. if action.get("tool") == "finish":
    43. break
    44. result = execute(action, spec)
    45. state["history"].append({"action": action, "result": result})
    46. state["step"] += 1
    47. save_state(state)
    48. return state

    The main loop writes state after every step. If the process is interrupted, it can resume from the latest checkpoint.

    Failure Rollback and Acceptance

    Acceptance is performed by the pipeline, not by the model. Calling finish only means the model believes the task is complete. The pipeline then runs test_command. Acceptance is final only when the return code is 0 and human review passes.

    For rollback, use git revert or switch to a checkpoint branch. Do not use force reset. Commit each verifiable step and tag it with a checkpoint label such as agent/checkpoint-3. Human confirmation is required before rollback.

    If the same test fails three consecutive times, stop the loop and escalate to a human. This prevents the model from repeatedly modifying the same spot and consuming large numbers of steps.

    Multi-Agent Decomposition, Communication, and Fault Tolerance

    Multi-agent systems are suitable when a task can be split into relatively independent subtasks. Splitting too finely increases coordination cost. Splitting too coarsely removes the benefit of division of labor.

    Decomposition Principles

    Assign ownership by file or module, and ensure only one agent writes to a file at a time. The planner does not write code. A coder does not approve its own changes. The reviewer has read-only access. Each subtask should have clear inputs, outputs, and acceptance tests.

    Message Protocol

    Messages between agents use a fixed structure. The receiver deduplicates by idempotency_key so duplicate messages do not cause duplicate modifications.

    1. {
    2. "message_id": "7f3c2a9e-5b1d-4e8a-9c2f-1a2b3c4d5e6f",
    3. "task_id": "order-rounding-02",
    4. "sender": "coder-1",
    5. "receiver": "reviewer",
    6. "type": "patch_ready",
    7. "payload": {
    8. "branch": "agent/coder-1/order-rounding",
    9. "files": ["src/order/money.py"]
    10. },
    11. "idempotency_key": "order-rounding-02:patch_ready:v1",
    12. "created_at": "2026-11-20T13:22:19+00:00"
    13. }

    Retry, Circuit Breaker, and Escalation

    Retry only transient errors such as timeouts and connection interruptions. Problems like malformed model output should be sent back to the model for correction, or escalated to a human.

    1. import time
    2. MAX_RETRIES = 3
    3. FAILURE_LIMIT = 5
    4. class CircuitOpenError(RuntimeError):
    5. pass
    6. class Breaker:
    7. def __init__(self, limit: int):
    8. self.limit = limit
    9. self.failures = 0
    10. def call(self, fn, *args, **kwargs):
    11. if self.failures >= self.limit:
    12. raise CircuitOpenError("Failure threshold reached; waiting for manual handling")
    13. for attempt in range(MAX_RETRIES):
    14. try:
    15. result = fn(*args, **kwargs)
    16. self.failures = 0
    17. return result
    18. except (TimeoutError, ConnectionError):
    19. time.sleep(2 ** attempt)
    20. self.failures += 1
    21. raise RuntimeError("Retries exhausted")

    When branches created by multiple agents have merge conflicts, the agents should not resolve them themselves. Send the conflicting files and a summary of both changes to a reviewer or human.

    Audit Log

    Every message, tool call, and human confirmation should be written to an append-only log. Fields include timestamp, agent ID, action, parameter summary, result, and content hash. The log is append-only; entries are not rewritten or deleted.

    Troubleshooting Checklist

    • Tests pass but the result is wrong: check whether test_command covers the modified code path.
    • The model repeatedly modifies the same file: inspect the history trace and confirm whether it is repeating the same failure.
    • State is abnormal after a process interruption: check checkpoint.json and restore the latest checkpoint if needed.
    • Tool calls time out: confirm the timeout setting, record the timeout as a failure, and do not let the loop crash.
    • Multiple agents modified the same file: check the file ownership table and confirm whether the write lock is active.
    • Prompt length degrades output quality: truncate history, keeping only recent steps and failure summaries.
    • License terms do not match the intended use: pause release, send the issue to legal review, and update the license checklist.
    • Logs contain keys or personal information: rotate keys immediately, clean the logs, and review the log collection rules.