Evaluating Long-Horizon Multi-Agent Tasks: Delivery and Cost-Effectiveness

When the industry discusses whether a model can take over a project, the real question is not whether a single Q&A response is impressive, but whether it can continuously decompose, execute, review, and integrate across cross-day, multi-stage, decomposable tasks to ultimately return an inspectable deliverable. Public experiments have featured two representative cases: one building a bootable prototype operating system from an empty directory over several days, and another completing a formal reproduction of an existing mathematical proof in about thirty hours. While these cases are noteworthy, they cannot be directly equated to production readiness. Rather than reviewing specific products, this article addresses general technical issues, discussing the delivery capabilities, acceptance boundaries, and cost-effectiveness evaluation methods for long-horizon multi-agent tasks.

1. The Core of Long-Horizon Tasks: Shifting from “Generating Content” to “Delivering Artifacts”

Short tasks typically end with a single output, whereas long-horizon tasks culminate in inspectable artifacts. The differences lie in state persistence, context management, error recovery, role collaboration, and auditability. A cross-day task often involves stages such as requirement clarification, solution design, task decomposition, parallel execution, test review, conflict resolution, integration verification, and delivery documentation. If a system can only continuously generate text but cannot maintain task states and artifact versions, it can hardly be considered deliverable.

During evaluation, focus on whether the following five closed loops are established:

1. Planning closed loop: Can it break down high-level goals into verifiable subtasks, specifying dependencies, inputs/outputs, and completion criteria?
2. Execution closed loop: Can sub-agents complete code, documentation, testing, or analysis tasks within limited tools and permissions?
3. Review closed loop: Is there an independent review role to check artifacts, rather than having the executor self-prove correctness?
4. Integration closed loop: Can it merge scattered artifacts into a holistically runnable deliverable and provide a change summary?
5. Recovery closed loop: Upon failure, timeout, or conflict, can it roll back, retry, or re-plan instead of simply aborting?

The prototype OS case reveals that starting from an empty folder, a model coordinating thousands of sub-agents to continuously write code, test, and review—and eventually booting on a laptop—demonstrates the demonstration value of “multi-role orchestration + artifact integration.” However, “being able to boot” only verifies runnability on a specific path and does not cover long-term stability, hardware compatibility, security boundaries, or operational costs. The formal reproduction case demonstrates writing an existing proof into a form where every step can be checked by a computer. This verifies formal expression and proof-checking success, which is not equivalent to proposing a new theorem or achieving mathematical innovation. Both cases answer specific questions and cannot be extrapolated to mean that all projects are deliverable.

2. Selection Tables Must Record Capabilities, Safety States, and Access Qualifications

The same underlying capability can present different service scopes under varying safety measures and access rules. For developers, the most dangerous misjudgment is directly transferring “high scores in public benchmarks” to “the service available in their own account.” An evaluation table should at least include the following columns:

• Capability items: Code generation, test repair, formal expression, long-context maintenance, tool calling, artifact integration, etc.
• Safety state: Whether high-risk domains like cybersecurity and biology are restricted; whether there is a degradation path after interception.
• Access qualification: Whether the organization has the corresponding verification or authorization; whether different product forms open the same scope.
• Observability: Whether it provides task logs, subtask states, artifact versions, and review records.
• Cost items: Call volume, concurrency, retries, storage, compute, and manual review costs.
• Acceptance items: Stage gates, final artifact inspection, and failure handling rules.

A common statement in public materials is “interception volume decreased,” but reduced interception does not mean high-risk tasks can be executed. During selection, you cannot just look at the interception ratio; you must itemize the safety scope, degradation strategy, and compliance requirements. Some high-risk benchmark scores only correspond to product forms with specific access qualifications and cannot be borrowed as proof of capability for restricted forms. In other words, if the selection table lacks the “Safety state” and “Access qualification” columns, a seemingly leading score might correspond to a practically unusable service.

3. Acceptance Criteria: Decomposing “Demo Success” into Inspectable Conditions

Acceptance criteria for long-horizon tasks should be defined upfront rather than judged by feeling after the fact. It is recommended to design them across three layers: “artifacts, process, and boundaries”:

• Artifact layer: Is the prototype system bootable, are core functions demonstrable, and are the installation and login flows complete? For formal tasks, does it pass the checker, are the proof scripts reproducible, and are dependencies locked?
• Process layer: Do subtasks have state logs, are failure retries traceable, are review comments closed-loop, and are merge conflicts handled with records?
• Boundary layer: Explicitly state the unverified scopes, such as long-term stability, compatibility, performance limits, security penetration, and large-scale concurrency.

An actionable acceptance checklist can be written as a structured configuration:

task: cross_day_delivery
acceptance:
  artifacts:
    - runnable_prototype: true
    - reproducible_script: true
    - change_summary: required
  process:
    - subtask_state_log: required
    - independent_review: required
    - rollback_plan: required
  boundaries:
    - long_term_stability: not_verified
    - hardware_compatibility: limited
    - security_assessment: separate_gate

The value of such a configuration lies in transforming “looks completed” into “verified as completed by conditions.” For the prototype OS, acceptance should specify the boot environment, must-test features, and out-of-scope areas; for formal reproduction, acceptance should specify the checker version, dependency locks, and reproduction steps. The clearer the acceptance criteria, the more reliable the cost-effectiveness evaluation.

4. Collaboration Structure and Complete Billing Determine True Costs

Long-horizon multi-agent tasks are not “fully automated outsourcing” but a design problem of human-machine collaboration. Humans need to handle goal setting, acceptance criteria formulation, critical gate reviews, and risk decisions; agents can handle planning drafts, parallel execution, test generation, documentation, and cross-review. If the collaboration structure is missing, common consequences include duplicated subtasks, superficial reviews, and rework during integration.

Costs also cannot be judged solely by single-call pricing. A complete bill includes at least: primary and auxiliary model calls, sub-agent concurrency, failure retries, long-context maintenance, artifact storage, runtime compute, logging and monitoring, and manual review time. In cross-day tasks, retries and reviews often account for a significant long-tail cost. Cost-effectiveness should be calculated based on the “cost per artifact that passes acceptance,” not the “number of generated tokens.”

In general cloud environments, object storage can be used to save artifacts and versions, container platforms can host runnable prototypes, and logging services with monitoring alerts can record subtask states and exceptions. Taking general cloud environments like Baidu AI Cloud as an example, these capabilities can serve as a bearing layer for delivery auditing and cost allocation, but the specific solution must still be designed according to organizational compliance and access requirements.

5. Deployment Recommendations: Verify on a Small Scale First, Then Expand the Task Radius

A progressive deployment approach is recommended: start with medium-sized tasks with clear boundaries that can be automatically checked, such as module refactoring with tests, documentation and code consistency checks, or formal script reproduction; then gradually expand to cross-day tasks. Retain manual gates and rollback plans at each stage. The orchestration logic can be abstracted into the following pseudocode:

plan = planner.decompose(goal, acceptance_criteria)
for stage in plan.stages:
    artifacts = executor.run(stage, context=artifact_store)
    review = reviewer.check(artifacts, stage.criteria)
    if review.failed:
        artifacts = executor.repair(review.issues)
    artifact_store.commit(stage, artifacts, review.log)
final = integrator.merge(artifact_store, plan.integration_spec)
assert acceptance_check(final, goal.criteria)

In summary, the value of long-horizon multi-agent tasks lies not in how impressive the demo looks, but in whether capabilities, safety, access, collaboration, acceptance, and billing can be placed into the same evaluation table. Only by reading the conditions for establishment together can we judge whether it can truly take over the project and whether the cost-effectiveness holds up.