Multi-Agent Orchestration Benchmarks: From Fixed Topologies to Autonomous Scheduling
Multi-agent systems are moving from engineer-designed, fixed topologies toward autonomous orchestration, where a model decomposes tasks, spawns sub-agents, and decides how to collaborate at runtime. This shift exposes the limits of traditional benchmarks, which evaluate a single agent and see only the final answer. This article explains why conventional evaluation falls short for orchestration, argues for a delivery-focused principle that prescribes goals and acceptance criteria rather than the division of labor, and lays out how to build an orchestration benchmark: a tiered task set that scales from single-agent problems to open-ended ones, a metrics system spanning quality, efficiency, and stability (including the orchestration overhead ratio), and engineering essentials such as isolated sandboxes, DAG-based execution, deterministic verification, and observability. It closes with common pitfalls and practical guidance for developers and enterprises.
In traditional multi-agent systems, engineers usually decide up front how the team divides the work. An architect sketches the flow first: who plans, who writes code, who reviews, where results are consolidated, and how to retry when something goes wrong. This approach is controllable and predictable, but the ceiling on what the system can achieve is limited by the designer’s experience. Once a task falls outside the predefined boundaries, a rigid structure struggles to adjust in time.
That static design is now loosening. An emerging paradigm lets the model decompose tasks at runtime, create sub-agents dynamically, choose how to collaborate, and decide when to consolidate results. Orchestration is no longer a hard-coded configuration but a behavior the model generates in real time. This shift gives systems stronger adaptability and forces a corresponding change in how we evaluate them.
1. Why Traditional Benchmarks Miss Orchestration
To measure orchestration, first understand how it differs from single-agent capability. A single agent faces a linear problem-solving process: understand the requirement, retrieve information, call tools, and produce an answer. Multi-agent orchestration deals with a nonlinear collaboration problem, mainly in three ways:
- The combinatorial space explodes. How to split a task, who takes each part, whether to run serially or in parallel, and how to merge results — the number of possible plans grows exponentially with the number of agents. A single task often has several valid paths but many more invalid ones.
- It is highly global. Once the upstream decomposition is off, the harder the downstream executes, the farther it drifts from the goal. A locally optimal choice does not guarantee a globally optimal one.
- Resource consumption is uncertain. The number of agents, communication rounds, and context duplication all add overhead, and quality gains do not always scale proportionally with cost.
Traditional evaluation methods mainly cover the first two capabilities: single-turn QA benchmarks assess knowledge and reasoning, while agent benchmarks assess tool use and task execution — but both treat a single agent as the unit of evaluation. Applying them directly to a multi-agent system has at least four shortcomings:
- Only the answer, not the division of labor. Even when the final answer is correct, it may come from several agents duplicating effort and happening to get it right, which says nothing about whether the collaboration structure is sound.
- Only pass or fail, not the cost. A classic multi-agent problem is “more expensive but not better,” and pass rate alone cannot expose it.
- No localization of failures. A single agent usually fails at a single point, whereas orchestration can fail at any stage — decomposition, assignment, communication, aggregation, or termination — and a single metric cannot pinpoint where the problem lies.
- Insensitive to dynamic orchestration. Under a fixed topology, the team structure is decided before evaluation, so the result represents only one configuration. With runtime autonomous orchestration, the team structure itself is the object under test.
2. The Core Principle: Judge the Delivery, Not the Division of Labor
To address these problems, the design of a multi-agent orchestration benchmark should follow one basic principle: specify only the task goal and acceptance criteria, not the way the task is completed. A model is free to answer on its own or to organize multiple agents to collaborate; the task prompt describes only “what must be delivered” and says nothing about “how many agents to use or in what order.”
There are three reasons to hold to this principle:
- The quality of orchestration lies in the choice itself. If a benchmark prescribes the optimal topology, it measures only execution ability, not orchestration ability.
- Real scenarios rarely have a standard topology. Business requirements vary enormously, and prescribing the division of labor in advance turns the benchmark back into a test of configuration compliance.
- It avoids an implicit bias toward a particular architecture. A task that is forced into a “planner–executor” pattern may be better served by a “parallel decompose–aggregate” strategy, and a fixed pipeline would wrongly fail some reasonable solutions.
“Judging only the results” does not mean the process is irrelevant; it means shifting the emphasis onto deliverables that can be objectively accepted:
- Code tasks are judged by whether they compile, pass all unit tests, and satisfy code-quality rules.
- Data-processing tasks are judged by whether the output matches the expected result.
- Document-writing tasks are judged by whether they contain the required points and meet the formatting requirements.
- Information-retrieval tasks are judged by whether key facts are complete and free of factual errors.
Process information is still valuable, but its purpose is post-hoc attribution and cost analysis, not deciding correctness. This also aligns with the evaluation ethics of “a fair process and comparable results”: as long as the deliverable meets the bar, different collaboration approaches used by different models should all be recognized. For “results only” to truly hold up, however, two supporting conditions are needed. First, results must be verifiable, not dependent on subjective impressions that cannot be judged objectively. Second, a reasonable resource budget must be set to prevent buying pass rate through “cost-no-object piling on of agents.” Only under these two conditions does “not prescribing the division of labor” become a fair, open competition.
3. Designing the Task Set
A convincing orchestration benchmark cannot consist of homogeneous tasks. Build a graded task set along the axis of “collaboration payoff,” so it can distinguish both single-agent from multi-agent systems and one orchestration strategy from another:
| Task Tier | Task Characteristics | Focus | Example (illustrative) |
|---|---|---|---|
| L1 Solvable by a single agent | Clear goal, limited steps | Whether basic execution is reliable | Complete a function to spec and pass tests |
| L2 Decomposable into independent subtasks | Parts are independent and can run in parallel | Decomposition granularity, parallel scheduling, result merging | Implement features for multiple modules and integrate them |
| L3 Strong dependencies | Later steps depend on earlier results | Dependency management, upstream/downstream handoff, error-propagation control | Clean, model, and report from raw data in sequence |
| L4 Requires multi-perspective trade-offs | A single perspective cannot yield a complete decision | Complementary roles, conflict resolution | Implement, test, and review a solution and fix defects |
| L5 Open-ended and complex | Vague goals, open boundaries, unknown steps | Autonomous exploration, dynamic planning, termination judgment | Locate and fix a defect in an unfamiliar codebase |
At the same time, the task set needs to do three things:
- Keep the scale moderate. Too few tasks are vulnerable to chance, while too many drive up evaluation cost. Follow the logic of statistical sampling and use variance estimates to determine the sample size.
- Balance the difficulty. Allocate the tiers in sensible proportions so the score curve reflects how a system performs across different levels of collaboration complexity.
- Prevent contamination. Reserve a portion of private tasks for periodic regression so that public problems cannot be optimized through rote “memorize-the-answer” tactics.
4. Metrics: Balancing Quality, Efficiency, and Stability
Orchestration evaluation cannot rest on a single pass rate. Three categories of metrics are recommended:
- Quality metrics. Delivery success rate — the proportion of tasks that meet the acceptance criteria — and key-metric attainment rate, used for fine-grained evaluation of open-ended tasks.
- Efficiency metrics. Average time per task, average token consumption, average number of tool calls, and the actual number of agents enabled. Some of these require runtime instrumentation, and the agent count can only be measured, never specified by the party under test.
- Stability metrics. The variance in success rate across repeated runs of the same task, and the degree to which the orchestration structure stays consistent from one run to another.
Statistically, the same task should be sampled multiple times. When reporting results, give the sample size and confidence interval to avoid mistaking noise for a real difference.
One metric deserves special attention: the orchestration overhead ratio — the ratio of quality change to cost change for a multi-agent system relative to a single-agent baseline of comparable model scale. The reasonable goal of orchestration is “a significant quality gain at controllable cost,” not merely a higher score. The overhead ratio immediately exposes strategies that “pile on agents to buy points.”
5. Engineering Implementation
1. Isolated Sandbox and Reproducible Environment
Tasks run inside isolated containers with independent file systems, controlled resource usage (CPU, memory, and runtime), and network access restricted as needed. Environment dependencies are pinned so that the same task yields similar results across different times and different machines. For tasks with side effects, a snapshot-and-rollback mechanism is also required.
2. Execution Orchestration and Timeout Governance
The evaluation platform itself must have orchestration capability and can borrow proven ideas from mature task-scheduling frameworks:
- Describe the flow of “prepare environment → run system → verify and score → clean up environment” as a directed acyclic graph (DAG);
- Set a timeout, retry count, and failure-degradation policy for each node;
- Apply rate limiting by model, by task, and by concurrency to avoid overwhelming the backend inference services.
3. Structured Verification and Scorers
Prefer deterministic checks in verifiers, such as compilation, running tests, comparing output files, and checking coding conventions. For open-ended tasks that genuinely require model-based scoring, use a judge model that is independent of the model under test, and provide clear scoring rubrics and reference points to reduce subjective variance.
A task specification can be described like this (illustrative):
from dataclasses import dataclassfrom typing import Callable@dataclass(frozen=True)class Task:task_id: strinstruction: str # describes only the goal and constraints, not the division of laborsetup: list[str] # commands that prepare the sandbox environmentverify: Callable[[str], dict] # runs inside the sandbox and returns the verification resultbudget: dict # resource budget: tokens, call count, wall-clock time@dataclassclass RunResult:task_id: strpassed: boolmetrics: dict # latency, tokens, tool calls, number of agents, etc.artifacts: list[str] # deliverables, run logs, and orchestration traces
The main flow on the execution side stays just as concise:
def evaluate(task: Task, system) -> RunResult:with Sandbox(task.setup) as box:# run the system under test; the budget is enforced by the platform,# and the party under test cannot specify the topologyrun = system.run(task.instruction, box, budget=task.budget)# judge only by the deliverables, not by how the work is dividedverdict = task.verify(box)return RunResult(task_id=task.task_id,passed=verdict["passed"],metrics={**run.metrics, "score": verdict.get("score", 0)},artifacts=run.artifacts,)
4. Observability and Cost Control
At runtime, record message passing between agents, the creation and completion of subtasks, and the input, output, and latency of every model call. This information serves both failure attribution and cost analysis as raw data. In the cloud, the orchestration execution layer can achieve elastic scaling on top of general-purpose container services, use message queues for task distribution and result collection, and store deliverables and run records in object storage. Full-pipeline runtime data can feed into logging services and monitoring/alerting to support long-term comparison and alerts. For example, general-purpose cloud products such as those from Baidu AI Cloud can be combined on demand to support the engineering implementation of an evaluation platform.
6. Common Pitfalls and How to Avoid Them
- Treating agent count as capability. More agents does not mean stronger; measure input versus output with the overhead ratio.
- Drawing conclusions from a single run. Orchestration behavior is stochastic; sample multiple times and report variance.
- Ignoring cost and concurrency. Evaluating only at low concurrency does not reflect performance under real load, and a success rate that ignores cost has little practical meaning.
- Using a scorer from the same source as the model under test. This easily creates “self-preference”; introduce an independent judge or cross-review.
- Overlooking the distribution of failure modes. Tally failure types such as “decomposition failure, communication failure, aggregation failure, termination failure” to produce a kind of “health report” for orchestration capability.
- Equating success with reproducibility. The structure and results of the same system across multiple runs should be roughly consistent; poor stability is itself a defect.
7. Practical Implications for Developers and Enterprises
For developers, the evaluation mindset is shifting from “optimizing prompts” to “optimizing orchestration strategy.” An actionable suggestion is to build a small-scale regression evaluation on your own business data and rerun it after each orchestration change, using the overhead ratio rather than pass rate as the primary basis for decisions.
For enterprises, orchestration capability determines whether agents can carry end-to-end business processes. When selecting solutions, do not look only at single-point demo results; also pay attention to three things: adaptability when facing unfamiliar tasks, the ability to self-correct after failures, and the ability to make trade-offs under resource constraints. An orchestration benchmark can give quantitative answers to all three.
Multi-agent orchestration is moving from “preset flows” toward “autonomous organization.” The value of an orchestration benchmark is to establish a fair, reproducible, and sustainable yardstick for this new paradigm: don’t prescribe the division of labor, judge only the final delivery. For developers, this is both an upgrade in evaluation methods and an entirely new test of engineering capability.
