Multi-Agent Collaboration for Long-Running Engineering Tasks: Practices and Lessons Learned

1. From Single-Turn Q&A to Long-Running Autonomous Tasks

In recent years, large language models have primarily been used for code generation through single-turn interactions. A developer provides a requirement, the model returns a function, and a human reviews the output line by line. This approach works well for localized tasks, but it becomes far less efficient and reliable when the objective spans several days, thousands of files, and numerous test cases.

A long-running autonomous task begins with a high-level goal—for example, implementing a small operating system kernel that can boot—and expects the system to complete requirement decomposition, code generation, testing, defect correction, and code review with limited human intervention. The main challenge is not the quality of any single generation, but state management, error accumulation, and quality control over the entire process.

2. A Basic Architecture for Multi-Agent Collaboration

The core principle of a multi-agent system is division of labor. An orchestrator maintains the overall objective, breaks it into subtasks, assigns those tasks to agents with different roles, and consolidates their results. Common roles include:

• Planning agent: Breaks the objective into modules, interfaces, and milestones.
• Implementation agent: Writes code by module. Each execution unit is responsible for one file or a closely related group of functions.
• Testing agent: Generates unit and integration tests based on interface contracts.
• Review agent: Checks coding standards, edge cases, and security issues, then provides modification recommendations.
• Record-keeping agent: Maintains decision logs and change summaries so that later agents can understand the relevant context.

Agents typically do not share the entire context. Instead, they exchange information through shared state, such as task boards, interface documentation, and test results. This design has two advantages. First, each execution unit has a limited context window and only needs to load materials related to its task. Second, centralized state makes checkpoint recovery possible, allowing the system to resume from a recent stable point rather than restarting from the beginning.

The following is illustrative orchestration pseudocode:

state = load_checkpoint_or_init(goal)
while not state.acceptance_passed():
    tasks = planner.decompose(state.open_issues())
    for task in schedule(tasks, max_parallel=N):
        patch = worker[task.role].execute(task, state.context_for(task))
        result = test_runner.run(patch)
        review = reviewer.check(patch, result)
        if result.ok and review.approved:
            state.merge(patch)
        else:
            state.record_failure(task, result, review)
    state.save_checkpoint()

This logic contains three important design decisions:

1. Concurrency is controlled through a parameter to prevent resource exhaustion.
2. Every change must pass testing and review before it is merged.
3. Failure information is written back to the shared state rather than discarded, allowing later tasks to adapt their approach.

3. Engineering Challenges in Long-Running Tasks

3.1 Context Management

The code, logs, and discussion records generated by a long-running task can quickly exceed the capacity of a single context window. A common solution is hierarchical summarization. After a module is completed, the system generates an interface description. Later tasks reference the interface instead of loading the full implementation.

These summaries must include key design decisions and unresolved issues. Otherwise, later agents may repeat the same mistakes.

3.2 State Persistence and Recovery

Tasks that run for several days will inevitably encounter process interruptions, resource timeouts, or failures in external services. A checkpoint should contain at least:

• The current code snapshot
• The open task list
• Test results
• The decision log

During recovery, the system should first validate the integrity of the snapshot instead of automatically trusting the most recent write.

3.3 Error Accumulation

Even when the error rate for an individual step is only a few percentage points, the overall success rate can decline substantially after hundreds of steps. The main countermeasure is to shorten the feedback loop. Relevant tests should run immediately after each change, keeping problems localized and preventing them from propagating into later steps.

3.4 Quality Gates

Test pass rates, static analysis results, and code coverage thresholds should be defined as merge conditions. Changes that fail these gates must not enter the main branch.

The thresholds should be set by humans rather than relaxed automatically by the execution system.

3.5 Versioning and Rollback

Every merge should be traceable and reversible. A separate change record should be generated for each merge, including the triggering task, test report, and review comments. If a problem occurs, the team should be able to identify the specific change responsible instead of inspecting the entire repository line by line.

4. How to Interpret Evaluation Data

Public benchmark scores are often used to compare different models. When reading this type of data, several factors deserve attention.

• The gap between benchmarks and real-world tasks: Standardized tests usually cover a limited range of problem types and may differ significantly from the distribution of code in an enterprise repository.
• The relationship between reasoning effort and cost: Many systems allow the amount of reasoning effort to be adjusted. Higher reasoning effort often improves results, but it also increases per-task cost and waiting time. Comparisons should consider both quality and cost rather than focusing only on the highest score.
• Self-reported results and independent evaluations: Vendor-published results should be independently reproduced where possible. Third-party evaluations are useful because they apply a consistent process, but the test version, scoring rules, and number of runs still need to be examined.
• The scope of each metric: A high programming benchmark score does not automatically indicate equivalent reliability in security auditing, requirements communication, or operations.

A simple evaluation framework is shown below:

Dimension	Key question	Suggested metrics
Quality	Does the output meet acceptance criteria?	Acceptance rate, defect density
Cost	How many resources does each task consume?	Cost per task, total duration
Stability	Can the system operate reliably over a long period?	Recovery success rate after interruption
Controllability	How easily can humans intervene?	Checkpoint granularity, number of approval gates

5. Balancing Cost and Performance

The cost of a multi-agent system generally increases with both concurrency and reasoning depth. Assume a task contains N subtasks and each subtask consumes an average cost of c units. The total cost can be approximated as:

N × c

This estimate must also include orchestration and review overhead.

Parallel scheduling can reduce elapsed time, but it generally cannot reduce total cost. Optimization should therefore focus on three areas:

• Assign reasoning effort according to task difficulty: Use lower reasoning effort for simple boilerplate code and higher effort for critical modules and security-sensitive code.
• Reduce repeated context: Use interface summaries instead of full source code as input whenever possible.
• Stop promptly: When all tests associated with the requirement have passed, terminate unnecessary iterations.

Organizations should define spending limits within their budgets. When a limit is reached, the system should pause automatically instead of running indefinitely.

6. Security and Permission Levels

A system that can autonomously write, compile, and run code also amplifies the risk of misuse. Deployment should follow these principles:

1. Least privilege: Each execution unit should access only the directories and interfaces required for its current task.
2. Sandbox isolation: Code should run in an isolated environment, with network access and system calls restricted.
3. Risk-based access levels for sensitive domains: Capabilities involving dual-use areas such as cybersecurity or biological research should require additional review and detailed operation logs.
4. Human approval gates: Actions such as external publication, permission changes, or data deletion must require human confirmation.

The same underlying capabilities can also be exposed at different levels according to their intended use. A general-purpose version can support routine development needs, while a specialized version can be made available only to organizations that have passed appropriate qualification checks. This layered approach is also applicable to internal enterprise systems.

7. A Phased Approach to Enterprise Adoption

Phase 1: Pilot small modules. Select modules with clear boundaries and comprehensive tests, such as data-format conversion components or basic utility libraries. Use the pilot to verify whether quality gates and rollback procedures work as intended.

Phase 2: Expand to cross-module tasks. Introduce interface contracts and integration tests. Observe how errors accumulate and adjust checkpoint frequency accordingly.

Phase 3: Scale operations. Establish cost dashboards, audit logs, and alerting rules. Concentrate human review on high-risk changes.

An acceptance checklist can include the following questions:

• Is the requirement described in a testable way?
• Do the tests cover critical paths?
• Is every merge documented?
• Are cost and execution time within budget?
• Can the system recover within the required time when an exception occurs?

8. Conclusion and Limitations

Multi-agent collaboration creates new possibilities for long-running engineering tasks, especially for projects with a high degree of modularity and strong test coverage. However, it does not eliminate the need for human involvement.

Humans remain responsible for defining objectives, setting acceptance criteria, establishing security boundaries, and assuming final accountability. When evaluating these systems, organizations should rely on data from local pilot projects rather than accepting promotional figures at face value.