Multi-Agent LLM Collaboration: Reshaping Prototyping and Engineering Workflows

1. Background: From Single-Turn Q&A to Long-Horizon Tasks

Over the past few years, the focus of large model applications has shifted from “answering questions” to “completing tasks.” Early usage was one question and one answer, with requirement decomposition and result checking handled by people. When a model can continuously call tools, read and write files, and run tests, it begins to take on a complete workflow. This shift creates new questions for engineering teams: how tasks should be split, which steps should be handed to the model, how to roll back after failures, and how to control cost and latency.

Public reports have mentioned that some frontier models completed long-horizon tasks such as hardware design, system prototype construction, and formal mathematical proofs during pre-release testing. These cases show that the boundary of model capability is still moving. However, external teams find it difficult to reproduce these results independently, and the reported data lacks unified evaluation criteria. A more effective approach is therefore to break such cases down into general capabilities and then validate their performance in specific business contexts using internal benchmarks.

2. Basic Architecture of Multi-Agent Collaboration

Multi-agent collaboration refers to a division-of-labor system composed of multiple model instances or “model plus tools” members, with each member taking on a clear role. Common roles can be divided into four categories:

• Planner: breaks the overall goal into subtasks with dependencies and outputs a task graph.
• Executor: completes a single subtask within limited tools and context, for example by writing a module or generating a set of test cases.
• Reviewer: independently checks the execution result and gives a conclusion of pass, return, or escalation to human review. The reviewer should ideally use a different context from the executor to avoid self-confirmation.
• State management: records task progress, artifact versions, and failure reasons so that a long process can recover after interruption.

The key to task splitting is dependency control. Subtasks without dependencies can run in parallel, while dependent subtasks must run serially, and input and output formats must be agreed in advance. The following pseudocode shows a scheduling skeleton based on a dependency graph. The execution function must be replaced with actual model calls and review logic:

import asyncio
async def execute(name: str) -> bool:
    """Call the executor to complete a subtask, have the reviewer check the result, and return whether it passes."""
    await asyncio.sleep(0)  # Placeholder: actual model, tool, and review calls go here
    return True
async def orchestrate(graph: dict[str, list[str]], limit: int = 8) -> dict[str, bool]:
    sem = asyncio.Semaphore(limit)   # Limit the number of concurrently running executors
    tasks: dict[str, asyncio.Task] = {}
    async def run(name: str) -> bool:
        upstream = [tasks[d] for d in graph[name]]
        results = await asyncio.gather(*upstream)
        if not all(results):         # If an upstream task fails, skip this task
            return False
        async with sem:
            return await execute(name)
    # Create handles for all tasks first, then await them together; dependencies are handled inside run
    for name in graph:
        tasks[name] = asyncio.create_task(run(name))
    return {name: await t for name, t in tasks.items()}
if __name__ == "__main__":
    graph = {
        "design": [],
        "driver": ["design"],
        "test": ["design"],
        "integration": ["driver", "test"],
    }
    print(asyncio.run(orchestrate(graph)))

This code reflects three engineering principles. First, dependencies are explicitly declared, which makes them easier to review and reuse. Second, concurrency is limited by a semaphore to avoid excessive instantaneous requests that could trigger quota limits or uncontrolled cost. Third, when an upstream task fails, downstream tasks are skipped automatically to prevent erroneous artifacts from spreading further.

3. Typical Scenarios

3.1 Hardware and Embedded Prototyping

Agents can draft schematics, bills of materials, firmware framework code, and simulation scripts. Hardware errors are expensive to fix, so model outputs can only serve as drafts. Teams should set external validation gates in the process, such as electrical rule checks, component library comparison, power simulation, and physical testing. Only versions that pass these checks can move to the next stage.

In practice, the most common problem is a model’s misunderstanding of component parameters, such as incorrect pin definitions or mismatched voltage ratings. The solution is to provide component datasheets as structured input, require the executor to cite the parameter source for each item in its output, and have the reviewer check whether the citations are accurate.

3.2 Prototyping Complex Software

Building an operating system prototype from scratch involves multiple layers, including the bootloader, memory management, process scheduling, device drivers, and file systems. These layers have a clear dependency order; for example, the scheduler depends on memory management, and the file system depends on block device drivers. A dependency graph can fix this order, allowing multiple executors to work in parallel within the same layer while connecting layers through interface tests.

Acceptance criteria need to be defined before the project starts, not added after it ends. Common metrics include bootability, unit test pass rate, the number of system calls covered by integration tests, and reproducibility in the target environment. Metrics should be determined by the team according to project goals and should not directly copy numbers from external reports.

3.3 Formal Proofs and Mathematical Verification

A proof assistant can check step by step whether each step of a proof is valid. A model can propose proof ideas, lemmas, and intermediate steps, which are then verified by the proof assistant. The advantage of this division of labor is that model output does not need to be fully trusted, because incorrect steps are rejected directly by the machine.

For large-scale parallel search, two things need attention. The first is deduplication, to avoid multiple executors repeatedly exploring the same path and wasting compute resources. The second is progress management, storing branches that have not yet succeeded in state so they can be continued when resources are sufficient.

3.4 Research Assistance

In research scenarios, agents are suitable for literature review, hypothesis organization, parameter sweeps, initial screening for structure prediction tasks, and result summarization. Steps involving wet-lab experiments, clinical decisions, or real biological samples must be reviewed by domain experts, and agents should not directly provide actionable operational conclusions. For tasks such as biomolecular structure prediction, model results should be compared with known experimental data, and confidence and scope of applicability should be labeled.

4. Cost, Speed, and Mode Selection

Large model services are usually billed by the number of input and output tokens, and output tokens often have a higher unit price. At a high level, the cost of a single task can be roughly estimated from its input and output token usage.