画板 1

    LLM Multi-Agent Capability Evaluation and Cost-Effectiveness Practice Guide

    6 hr ago0 views

    A new generation of frontier models has drawn attention for multi-agent capabilities, including large-scale parallel proof formalization, progress in robot manipulation without task-specific training, and lower inference prices with higher throughput. This article explains the engineering mechanisms behind these signals and provides practical methods for evaluating multi-agent systems, verifiable rewards, embodied AI risks, effective task cost, tiered routing, and safe deployment.

    1. Introduction: Engineering problems exposed by a parallel experiment

    A recent release in a frontier large-model series has sparked discussion. In evaluations, it showed three notable capabilities: a team of thousands of agents used about 30 hours to fully formalize Fermat’s Last Theorem into a formal language that can be verified step by step; it made progress on robot manipulation tasks without task-specific training; and inference prices were reduced while throughput improved further.

    Community discussion quickly split into two views. One view held that frontier capabilities had been substantially raised. The other cautioned that a single result does not represent general capability. Both views captured part of the picture, but neither answered the most important engineering questions: how the result was produced, how cost should be calculated, and how it can be implemented in practice. For developers and enterprises, the questions go further: how should multi-agent systems be selected? What are common mistakes? How can the effective cost of a task be measured?

    This article starts from the technical mechanism behind these signals and provides actionable evaluation and implementation methods. It should be noted that these evaluations are often single results published by vendors or evaluation organizations. They are relatively convincing on tasks where a verifier provides an objective judgment, but they are still far from proving that general capabilities have improved across the board without more independent replication.

    2. From single-point capability to large-scale parallel inference

    2.1 Mathematical proof is an extremely large search problem

    Fermat’s Last Theorem states that for integer n > 2, the equation xⁿ + yⁿ = zⁿ has no positive integer solutions. Andrew Wiles completed the proof in 1994. Its core, the Taniyama–Shimura conjecture in the rational case, was itself a massive, multi-year undertaking. For a machine to complete the proof, it must first rewrite the entire proof chain into a formal language that a theorem prover can check line by line. If any reasoning step is wrong, the verifier returns a failure. The amount of work and engineering complexity is comparable to translating and reviewing a large technical manual.

    Abstracting the formalization process, the core loop is as follows:

    1. Candidate proof (a single reasoning step)
    2.     | Check / compile (cheap, fast, deterministic judgment)
    3.     +-- Pass --> Submit and extend the proof state
    4.     +-- Fail --> Locate the error and enter the next repair round

    This is a typical “verifiable search” problem: the answer has an objective judgment criterion, but the search space is enormous. When a single agent advances serially, it faces three bottlenecks at the same time:

    1. Context-window limits: The longer the proof, the more intermediate conclusions must be maintained. When the context window is exceeded or long-range attention decays, the agent may lose key constraints or repeat paths that have already been ruled out.
    2. High variance in a single line of reasoning: A single agent follows one line of reasoning at each step. If it enters a wrong direction and there is no independent checker, errors accumulate gradually.
    3. High time cost: When search depth is large, total serial time grows approximately linearly with depth.

    2.2 Multi-agent parallelism changes the search structure

    A team composed of thousands of agents essentially expands a serial search chain into large-scale parallel search. Its basic flow includes decomposition, parallel exploration, aggregation and verification, and a correction loop. Upper-level agents split the main goal into independent sub-lemmas, such as modular form lifting, Galois representations, and modularity and level lowering. Sub-agents try different proof strategies for the same subgoal. Independent checkers compile and verify each candidate proof. Accepted conclusions enter a “proved theorem library” and can be referenced by other agents. Outputs that fail verification are returned with error-localization information, and agents fix them and retry.

    A typical agent orchestration pattern can be illustrated as follows. The key point is that frontier must be a list of task or lemma objects, rather than directly treating raw model output strings as task objects; model output needs to be parsed into Lemma objects with fields such as lemma_id, deps, and status; verifier.check needs to return an explicit structure; and cases such as an empty agent list, all attempts failing, and parallel execution must also be handled explicitly.

    1. from dataclasses import dataclass, field
    2. from typing import List
    3. from concurrent.futures import ThreadPoolExecutor, as_completed
    5. @dataclass
    6. class Lemma:
    7.     lemma_id: str
    8.     statement: str
    9.     deps: List[str] = field(default_factory=list)
    10.     status: str = "pending"
    11.     errors: str = ""
    13. @dataclass
    14. class VerifyResult:
    15.     status: str      # accepted / rejected / error
    16.     message: str = ""
    17.     logs: str = ""
    19. @dataclass
    20. class ProofCandidate:
    21.     lemma_id: str
    22.     agent_id: str
    23.     script: str
    24.     status: str = "pending"
    25.     errors: str = ""
    27. class FormalizationTeam:
    28.     """An agent team composed of planner, executor, verifier, and fixer roles."""
    30.     def __init__(self, model_client, verifier, max_rounds=20, max_workers=8):
    31.         self.model = model_client
    32.         self.verifier = verifier
    33.         self.max_rounds = max_rounds
    34.         self.max_workers = max_workers
    35.         self.proved = set()
    36.         self.library = {}
    37.         self.rounds = 0
    39.     def plan(self, root: Lemma) -> List[Lemma]:
    40.         """Parse the root statement into lemma objects with lemma_id, deps, and status."""
    41.         raw = self.model.call(
    42.             role="planner",
    43.             prompt=f"Decompose the statement: {root.statement}"
    44.         )
    45.         items = self.model.parse_json(raw)
    46.         return [Lemma(**item) for item in items]
    48.     def prove(self, lemma: Lemma, agent_ids: List[str]) -> ProofCandidate:
    49.         """Multiple executor agents try in parallel; the first result that passes verification is accepted."""
    50.         if not agent_ids:
    51.             return ProofCandidate(
    52.                 lemma.lemma_id,
    53.                 "",
    54.                 "",
    55.                 "rejected",
    56.                 "Empty agent list"
    57.             )
    59.         last = None
    60.         with ThreadPoolExecutor(max_workers=min(self.max_workers, len(agent_ids))) as pool:
    61.             futures = {
    62.                 pool.submit(self._prove_one, lemma, aid): aid
    63.                 for aid in agent_ids
    64.             }
    65.             for fut in as_completed(futures):
    66.                 cand = fut.result()
    67.                 last = cand
    68.                 if cand.status == "accepted":
    69.                     self.library[lemma.lemma_id] = cand
    70.                     return cand
    72.         return ProofCandidate(
    73.             lemma.lemma_id,
    74.             last.agent_id,
    75.             last.script,
    76.             "rejected",
    77.             f"All attempts failed: {last.errors}"
    78.         )
    80.     def _prove_one(self, lemma: Lemma, agent_id: str) -> ProofCandidate:
    81.         script = self.model.call(
    82.             role="prover",
    83.             agent_id=agent_id,
    84.             prompt=f"Dependencies: {lemma.deps}; statement: {lemma.statement}",
    85.             context=self.library
    86.         )
    87.         result: VerifyResult = self.verifier.check(lemma.lemma_id, script)
    88.         return ProofCandidate(
    89.             lemma.lemma_id,
    90.             agent_id,
    91.             script,
    92.             result.status,
    93.             (result.message + "\n" + result.logs).strip()
    94.         )
    96.     def run(self, root: Lemma, agent_ids: List[str]):
    97.         frontier: List[Lemma] = self.plan(root)
    98.         for _ in range(self.max_rounds):
    99.             self.rounds += 1
    100.             lemma = next(
    101.                 (x for x in frontier if x.lemma_id not in self.proved and x.status != "proved"),
    102.                 None
    103.             )
    104.             if lemma is None:
    105.                 break
    107.             candidate = self.prove(lemma, agent_ids)
    108.             if candidate.status == "accepted":
    109.                 lemma.status = "proved"
    110.                 self.proved.add(lemma.lemma_id)
    111.             else:
    112.                 lemma.status = "failed"
    113.                 lemma.errors = candidate.errors
    114.                 fixed_raw = self.model.call(
    115.                     role="fixer",
    116.                     prompt=f"Fix lemma {lemma.lemma_id}; errors: {lemma.errors[-500:]}"
    117.                 )
    118.                 fixed_items = self.model.parse_json(fixed_raw)
    119.                 for item in fixed_items:
    120.                     new_lemma = Lemma(**item)
    121.                     new_lemma.deps = list(set(new_lemma.deps + lemma.deps))
    122.                     frontier.append(new_lemma)
    124.         return {"proved": self.proved, "rounds": self.rounds}

    This pseudocode expresses a scheduling skeleton. In production, an asynchronous task queue can be used, for example by creating multiple proof tasks with asyncio and canceling the remaining tasks after the first accepted result is returned. Whether using a thread pool or asynchronous tasks, concurrency limits, task timeouts, maximum retry counts, and cost circuit breakers should be configured to prevent parallel scale from getting out of control.

    Compared with “a single agent working continuously for 30 hours,” the value of a multi-agent approach is not only “more people.” It lies in three aspects:

    • Fault tolerance: The failure of individual agents does not affect the whole; the team always has tasks in flight.
    • Knowledge reuse: Proven conclusions accumulate in the theorem library, so later reasoning does not need to start from zero.
    • Error isolation: Verification is completed locally, so errors do not propagate backward as they would in a serial chain.

    Therefore, the gap between the previous-generation model and the new-generation model cannot be simply reduced to “higher single-point intelligence.” A more reasonable explanation is that the new-generation model is better at following strict format constraints, locating compilation errors, and converging stably over multiple rounds of feedback. These are precisely the prerequisites for an efficient parallel pipeline.

    2.3 Cost and benefit change together

    Parallelization is not without cost. The main costs include three categories:

    • Communication and scheduling cost: When subtask dependencies are complex, agents need to frequently pass intermediate conclusions to one another, and scheduling and waiting can reduce effective utilization.
    • Redundant exploration cost: When multiple agents explore the same line of reasoning, failed attempts also consume compute resources. If decomposition granularity is too fine, the number of subtasks increases and coordination cost rises accordingly.
    • Error-propagation risk: A verifier usually provides a “pass or fail” signal rather than a complete explanation of “why it failed.” Errors at the planning level may be difficult to detect automatically.

    A practice that can be directly borrowed is “budget grading”:

    1. Task A: formalize several basic lemmas       budget coefficient 1x
    2. Task B: integrate lemmas and complete the main theorem       budget coefficient 4x

    These coefficients are not isolated empirical values. They should be related to task difficulty, number of verification rounds, failure rate, and parallel scale. An actionable parameterization idea is as follows:

    1. Budget coefficient ≈ base coefficient
    2.         + verification-round coefficient × verification rounds
    3.         + parallel-redundancy coefficient × number of parallel agents
    4.         + failure-compensation coefficient × failure rate

    For example, a basic task can use a base coefficient of 1; each additional verification round can add 0.2; each additional parallel agent can add 0.05; and every 10 percentage-point increase in failure rate can add 0.3. For critical tasks such as main-theorem integration, complex code repair, and long-chain planning, the result can be multiplied by an additional 2x to 4x redundancy factor.

    In practice, most subtasks can be completed at low cost, and only a few key steps require high-redundancy exploration. Allocating compute according to difficulty is often more economical than allocating it evenly throughout the process.

    3. Why verifiable rewards matter

    A parallel team can run stably only if every step has a cheap and deterministic objective judgment. This matters beyond mathematics itself:

    • Reliable signals: Verifier output does not depend on subjective judgment and can be used as a reward signal for processes such as reinforcement learning, rejection sampling, and automated evaluation.
    • Reproducible results: Any third party can use the same checker to reproduce the conclusion, which is far more credible than subjective evaluation.
    • Engineering feasibility: Failure localization is clear, making it easier to build automatic retry, rollback, and audit mechanisms.

    In contrast, open-domain tasks lack a unified “compiler.” For tasks such as copywriting, product design, and customer-service responses, approximate verifiers can be constructed using engineering methods such as rule checks, unit tests, business-process instrumentation, human spot checks, and offline replay. The more parts that can be automatically judged, the easier it is for model output to enter a stable production process.

    4. Embodied intelligence: general models are becoming “priors for the physical world”

    4.1 From “training for robots” to “helping machines understand the world”

    Traditional robot control is often trained separately for specific hardware and specific tasks. Once the task or embodiment changes, data, models, and tuning almost have to start over. A new generation of vision-language-action models tries to change this paradigm. Its core ideas include:

    • Learning common sense about objects, scenes, and operations from large amounts of internet image-text and video data;
    • Performing joint training on cross-embodiment robot manipulation data;
    • Mapping “visual input + language instruction” into executable action sequences.

    The statement that tasks can be completed “without task-specific training” indicates that general models already have some degree of physical common sense and task understanding. This differs from the previous empirical judgment that “robot capability ≈ task data volume.” If this can be independently reproduced, it would be an important turning point for embodied intelligence.

    4.2 Simulation-to-real: a key part of the data loop

    A general prior does not mean out-of-the-box usability. Real-robot deployment usually requires the following steps:

    • Domain randomization: Randomize lighting, materials, friction coefficients, and object poses in simulation to improve policy generalization.
    • Real-robot fine-tuning: Use a small amount of real data for adaptation to narrow the gap between simulation and reality.
    • Action abstraction: Break high-level skills such as “grasp-move-place” into sub-actions to reduce the difficulty of end-to-end learning.

    This can be illustrated as follows:

    1. Text instruction --> Task decomposer --> Skill library (grasp / place / push-pull / open-close)
    2.                                 |
    3.               +-----------------+------------------+
    4.         Simulation environment   Real-robot execution   Data return
    5.               |                  |                  |
    6.    Policy training/verification   Execution and exception detection   Failure-case labeling
    7.               +------------------+------------------+
    8.                         Continuous iteration

    4.3 Risks and fields that must be evaluated before deployment

    When enterprises evaluate embodied-intelligence solutions, they should not look only at demonstration videos. A unified evaluation form should be established. At least the following fields are recommended:

    Evaluation dimension Field Description
    Task success rate First-pass success rate, success rate including recovery Distinguish cases that need no human intervention from cases that need human fallback
    Completion time Average completion time, P95 completion time Pay attention to long-tail latency
    Recovery capability Number of failure retries, number of recoveries Evaluate exception-handling strategy
    Human intervention rate Number of human takeovers, reasons for takeover Measure real deployment cost
    Long-tail scenario list Transparent objects, deformable objects, occlusion, abnormal placement, etc. These need separate testing
    Safety events Number of collisions, emergency-stop triggers, dangerous actions Used for safety and compliance evaluation

    In addition, reliability, safety and compliance, real-time performance, and evaluation standards need to be considered. General models often perform well in typical scenarios, but success rates can drop significantly in long-tail scenarios such as abnormal placement, transparent objects, and deformable objects. When human-robot collaboration is involved, force control, collision detection, and emergency-stop mechanisms are required; a model cannot replace safety design. On-device inference latency requirements are usually stricter than cloud-side requirements, so techniques such as model quantization and operator optimization are needed.

    5. Cost-effectiveness: from unit price to effective task cost

    5.1 Complete cost formula

    The “unit-price comparison” often cited during model selection cannot directly represent business cost. A more practical metric is “effective task cost.” The complete formula should include input, output, retry, parallel redundancy, verification, and scheduling costs:

    1. Effective task cost = (input tokens × input unit price + output tokens × output unit price + retry / parallel redundancy / verification / scheduling cost) ÷ successful tasks

    The definition of success rate must be clarified. At least two definitions are recommended:

    1. First-pass success rate = tasks passing first-round verification ÷ total tasks
    2. Success rate with retries = final successful tasks ÷ total tasks

    If “success rate with retries” is used in the formula, the numerator should also include retry, parallel redundancy, and verification costs. If “first-pass success rate” is used, the subsequent retry cost of failed tasks should be counted separately in total cost. When comparing different models, the definition must be consistent; otherwise, the conclusion can easily be distorted.

    5.2 Understanding a 20% price reduction and 30% throughput improvement

    A 20% reduction in unit price does not automatically mean a 25% increase in task volume. The budget can complete about 25% more tasks only if the following conditions are met:

    • Input tokens per task remain unchanged;
    • Output tokens per task remain unchanged;
    • Task success rate remains unchanged;
    • Retry rate remains unchanged;
    • Parallel redundancy and scheduling overhead remain unchanged;
    • The model capability is sufficient to complete the original task.

    Under these unchanged conditions, a 20% price reduction means the same budget can purchase more service:

    1. 1 ÷ (1 - 20%) = 1 ÷ 0.8 = 1.25

    That is, about a 25% increase in task volume. If the new model has a lower success rate, a higher retry rate, longer reasoning chains, or higher verification costs, the actual benefit will be offset.

    A 30% throughput improvement also cannot be directly equated with a 30% improvement in user experience. Throughput mainly affects the number of requests that can be processed per unit time, but real-time interactive applications also need to consider:

    • First-token latency: the time users wait for the first output;
    • Per-request SLA: whether a single request meets the maximum response-time requirement;
    • Concurrent queuing: whether high throughput comes from queuing rather than true parallelism;
    • Single-token generation speed: whether the experience of generating long text is smooth;
    • Peak stability: whether rate limiting, degradation, or retries affect availability.

    Therefore, a throughput improvement should be interpreted together with first-token latency, concurrent queuing, and per-request SLA.

    5.3 A reproducible calculation example

    Assume a batch document-structuring task:

    • Average input tokens: 8,000;
    • Average output tokens: 1,000;
    • Input unit price: 10 CNY per million tokens;
    • Output unit price: 30 CNY per million tokens;
    • First-pass success rate: 90%;
    • Success rate with retries: 95%.

    First, calculate the basic invocation cost:

    1. Basic cost = (8,000 × 10 + 1,000 × 30) ÷ 1,000,000
    2.           = (80,000 + 30,000) ÷ 1,000,000
    3.           = 0.11 CNY

    If the success rate with retries, 95%, is used as the definition, the effective task cost is:

    1. Effective task cost = 0.11 ÷ 0.95 ≈ 0.116 CNY per task

    If the first-pass success rate, 90%, is used, the extra cost from failed-task retries must be included in the numerator. For example, if retry costs increase total cost to 0.125 CNY, then:

    1. Effective task cost = 0.125 ÷ 0.90 ≈ 0.139 CNY per task

    This example shows that with the same basic unit price, the final effective task cost can differ significantly under different success-rate definitions and retry strategies.

    5.4 Cost and bottleneck comparison for typical task types

    The table below is based on general experience. Specific values vary by business, and it is recommended to replace them with your own data:

    Task type Main bottleneck Factors most sensitive to cost Optimization focus
    Batch document structuring Throughput Input token volume, batch size Batching, long-document chunking, caching
    Real-time dialogue / customer service Latency First-token latency, queuing Small-model routing, speculative decoding, caching
    Complex reasoning / code Quality Reasoning token consumption, retry count Tiered routing, verifier loops
    Long multi-agent tasks Total cost Parallel scale and redundancy Budget grading, early stopping, task merging

    5.5 Controlling cost with tiered routing

    Most businesses do not need to use the most advanced model for all requests. A reasonable approach is to route requests by task difficulty. The following is an illustrative routing function:

    1. def estimate_cost(model_price, input_tokens, output_tokens, success_rate):
    2.     """Estimate the effective cost of a single task. Unit price is per million tokens."""
    3.     total = (
    4.         input_tokens * model_price["in"]
    5.         + output_tokens * model_price["out"]
    6.     ) / 1_000_000
    7.     return total / max(success_rate, 1e-6)
    9. def route(task, policies):
    10.     """
    11.     policies: [
    12.         {"name": "lite", "price": {"in": 2, "out": 8}, "match": lambda t: t["type"] in ("extract", "classify")},
    13.         {"name": "std", "price": {"in": 10, "out": 30}, "match": lambda t: t["complexity"] <= 2},
    14.         {"name": "frontier", "price": {"in": 30, "out": 90}, "match": lambda t: True},
    15.     ]
    16.     """
    17.     for p in policies:
    18.         if p["match"](task):
    19.             cost = estimate_cost(
    20.                 p["price"],
    21.                 task["in_tokens"],
    22.                 task["out_tokens"],
    23.                 task["success_rate"]
    24.             )
    25.             return {"model": p["name"], "est_cost": round(cost, 5)}
    26.     return {"model": "frontier"}

    The prices and success rates in this example are illustrative values. In actual use, they should be replaced with data obtained from your own load testing. Evaluation conclusions should also be reviewed regularly as models iterate.

    6. Release strategy and capability boundaries: interpret cautiously

    Some evaluations or community feedback indicate that different model versions may perform differently on certain R&D assistance tasks. For example, on difficult R&D tasks such as accelerator kernel optimization and large-model architecture improvement, different models may differ in stability, reproducibility, and engineering usability.

    Because such judgments often lack verifiable public details, this article does not make factual claims about vendor-internal capability limits. Enterprise users should pay more attention to the following:

    • Use official release information as the standard: The scope of capability availability, version limits, and terms of use should follow the vendor’s official statement.
    • Use your own evaluation set as the standard: Do not replace business-scenario validation with a single leaderboard.
    • Pay attention to long-term iteration cadence: Model versions, prices, throughput, context length, and compliance policies may all change.
    • Keep human review: In high-risk scenarios involving kernel optimization, framework development, financial risk control, or medical assistance, it is not advisable to fully outsource tasks to models automatically.
    • Build degradation and fallback mechanisms: When new-model quality fluctuates, teams should be able to quickly switch back to an older model or a backup model.

    7. Developer action checklist

    1. Define the task and judgment criteria first
      Whether a task can be automatically verified determines how much automation resource can be invested. It is recommended to prioritize tasks with compilers, unit tests, rule engines, business-process instrumentation, or human spot-check standards.

    2. Validate multi-agent systems in small steps
      Start with a pilot of 3 to 5 agents and gradually expand scale. Metrics should include at least task success rate, cost per task, average rounds, verification failure rate, human takeover rate, average duration, and redundant token consumption.

    3. Use budget grading instead of blindly adding compute
      Allocate exploration budgets according to task difficulty. Use low-cost models for basic extraction, classification, and format conversion tasks; use higher-redundancy budgets for complex reasoning, code repair, and long-chain planning.

    4. Build a continuous evaluation mechanism
      Rerun the evaluation set after each model iteration and update the routing strategy and cost model. The evaluation set should cover typical and long-tail scenarios, and failed samples should be retained for regression analysis.

    5. Design for safety and compliance
      In scenarios such as embodied intelligence, automated R&D, financial risk control, and customer-service outbound calls, retain human review, access control, log auditing, and rollback capabilities. The stronger the model, the more engineering boundaries are needed.

    8. Conclusion

    The most valuable aspect of this new-model release is not that it “proved a theorem,” but that it demonstrated the potential of the engineering paradigm “verifiable tasks + large-scale parallelism + feedback correction.” For developers and enterprises, the key is not to chase a single leaderboard, but to:

    • Identify which parts of your business can be verified;
    • Design reasonable agent-collaboration and budget-allocation mechanisms;
    • Continuously measure effective task cost with real data;
    • Keep human fallback in high-risk scenarios such as embodied intelligence and automated R&D;
    • Build a reproducible evaluation set, cost model, and canary-release process.

    In Baidu AI Cloud practice scenarios, developers can apply these patterns to concrete capabilities: unified access to models of different sizes, on-demand selection of compute specifications, and combining logging and monitoring, object storage, message queues, and other infrastructure into a complete pipeline, thereby validating the value of new paradigms such as multi-agent systems and embodied intelligence in their own businesses at a lower trial-and-error cost.

    09040904

    Top articles