Multi-Agent Collaboration: Capability Boundaries and Costs of Frontier LLMs
This article examines what recent demonstrations of long-horizon multi-agent work do and do not prove, then outlines a practical framework for evaluating multi-agent systems. It covers system roles, validation and retry costs, an API cost estimation formula with a worked example, common failure modes, and a step-by-step implementation checklist. The takeaway is that public case studies are directional signals; teams need reproducible internal tasks, clear validation criteria, and complete cost logs before moving frontier models into production.
1. Start by Understanding the Premises of the Cases
Recent reports say that frontier models have completed formal verification of a mathematical theorem and have built a bootable minimal operating system from an empty directory. These cases suggest that the direction of model work on long-horizon tasks is changing, but the reports alone cannot establish generality.
Take formal verification as an example. It requires translating a natural-language proof into machine-checkable code, where errors cannot slip through. The reported task used an intermediate theorem list as the scoring basis, and the agents completed derivations without external reference materials. This design makes the result automatically checkable, but the report did not disclose the model version, number of agents, total time, or failure rate, nor did it state how many times humans intervened.
The operating system case is similar. The report says over a thousand sub-agents were orchestrated to run continuously for several days and delivered a bootable system. Code size, hardware environment, module failure counts, and human correction points were not made public. Readers should treat such cases as signals of capability direction, not as benchmark data that can be applied directly.
2. Basic Structure of Multi-Agent Systems
A multi-agent system usually contains three types of roles. A controller node breaks the goal into subtasks, sub-agents execute them, and a validation module checks whether outputs meet the criteria. On failure, the system retries or rolls back according to predefined rules.
The key to this structure is validation. If validation criteria are vague, erroneous results will be treated as passing and inherited by later steps, so errors will accumulate. Therefore, each subtask should have explicit input/output contracts and automated checks, such as compilation, unit tests, formal verification, or data format validation.
Retries also have a cost. Each failure consumes tokens again, so the system must set retry caps and record the failure reason for each subtask.
3. How to Calculate Speed and Cost
Whether a new model is cost-effective cannot be judged only by unit price or output speed. An output speed improvement of tens of percentage points does not mean end-to-end task time will fall by the same amount, because retrieval, validation, and waiting for tool responses also consume total time.
API cost can be estimated as follows:
Cost per task ≈ number of calls × (input tokens × input unit price + output tokens × output unit price)
The number of calls includes retries. The table below lists the key parameters to record.
| Parameter | Meaning | How to record |
|---|---|---|
| Input tokens | Prompt and context for each call | Average from logs |
| Output tokens | Content generated by the model | Average from logs |
| Concurrency | Number of sub-agents running at the same time | Record peak and average |
| Retry rate | Proportion of re-calls after validation failure | Track separately by subtask type |
| Cache hit rate | Proportion of repeated context served from cache | Check vendor billing details |
| Unit price | Prices for input, output, and cache respectively | Use the vendor’s current price list |
Here is a hypothetical example. A task is split into 1,000 subtasks. Each subtask averages 8,000 input tokens and 2,000 output tokens, with a 20% retry rate, so the number of calls is 1,200. If the assumed input price is RMB 10 per million tokens and the output price is RMB 40 per million tokens, the input cost is RMB 96, the output cost is RMB 96, and the total is RMB 192. Actual numbers need to be replaced with your own price list and logs.
Compute quotas and reset rules in subscription plans also change. Before purchasing, rely on the terms of service and confirm whether the quota covers production concurrency requirements.
4. Frequently Asked Questions
Which tasks are suitable for multi-agent collaboration?
Tasks that can be decomposed and automatically validated, such as code generation plus unit tests, batch data cleaning, and document format conversion. Tasks that lack validation criteria and require strong subjective judgment, such as strategic decisions or content aesthetics, are not suitable for fully automated operation.
How should API cost be calculated?
Use the formula above and include retries in the number of calls. First run a small batch of tasks for a week to obtain the real token distribution and retry rate, then extrapolate the full cost.
What are common mistakes?
There are four main types: no retry cap, leading to runaway costs; validation criteria that are too weak, so erroneous results are treated as passing; state that is not persisted, so recovery after failure is impossible; and treating the number of sub-agents as a capability metric while ignoring output quality.
5. Implementation Steps
- Define automatically checkable success criteria for each task type.
- Design a task tree, with each leaf node specifying its input, output, and validation method.
- Choose state storage, messaging, and container orchestration. You can compare platforms such as Alibaba Cloud, Tencent Cloud, and AWS, or use a self-hosted option; focus on price, latency, data location, and compliance requirements.
- Build an evaluation sandbox and set retry caps and timeouts.
- Monitor token consumption, elapsed time, and failure rate, and set alert thresholds.
- Keep rollback mechanisms and human review checkpoints, especially for steps involving code merge or data writes.
- Run a small-scale pilot with real business data, compare the quality, time, and cost of the new and old models, and then decide whether to scale up.
6. Conclusion
The capabilities of frontier models on long-horizon tasks and multi-agent collaboration are worth watching, but public case studies are not a substitute for an organization’s own evaluation. Teams should use reproducible internal tasks, clear validation criteria, and complete cost records to judge whether a model is suitable for production.
