Evaluating Long-Task Capabilities in Next-Generation Large Models: From Prototype Systems to Formal Proofs

1. What Two Landmark Experiments Reveal

Two recently disclosed experiments have provided the industry with new reference points for evaluating how large models handle extremely long tasks.

The first involved building a prototype operating system from scratch. The model coordinated more than 1,000 sub-agents through ongoing code generation, testing, and review. The resulting system could boot on a standard laptop and included a kernel, disk and keyboard drivers, network protocols, persistent file storage, a window system, a login interface, and an installer.

The second involved reproducing a formalized mathematical proof. One thousand agents running a newer model took approximately 30 hours to rewrite an existing major proof into a form that could be verified step by step by a computer. Comparable work had previously taken 11 days, representing a significant improvement in efficiency.

The verification boundaries of both results are important. A system that can boot has not necessarily achieved long-term stability or broad hardware compatibility. A proof accepted by a checker only demonstrates successful formalization; it does not mean the model independently discovered a new proof. These experiments answer specific questions. They do not show that a model can independently take over any project.

2. Three Conditions for Deliverable Long-Running Tasks

Before applying these findings to real-world business workflows, teams should answer three questions.

1. Can the acceptance criteria be evaluated by machines?

Booting an operating system and passing a proof checker are both clearly defined acceptance conditions. Enterprise projects, by contrast, often involve changing requirements, implicit constraints, and human review. The less precise the acceptance criteria, the greater the gap between model autonomy and actual project delivery.

In practice, teams should first break tasks into subgoals that can be verified automatically, such as unit test coverage, successful static analysis, and consistency with interface contracts.

2. Does the collaboration structure support multi-agent specialization?

Coordinating thousands of sub-agents depends on clear task decomposition, code-merging strategies, and conflict-resolution mechanisms. A simplified orchestration pattern might look like this:

# Simplified orchestration pattern for a long-running multi-agent task
plan = planner.decompose(requirement)        # Task decomposition
for subtask in plan:
    result = worker.run(subtask)             # Generation
    verdict = reviewer.check(result)         # Review
    if not verdict.passed:
        worker.revise(subtask, verdict.issues)
integrator.merge_all()                       # Integration and regression testing

Adding more sub-agents is not automatically better. The key factors are the independence of the review process and how quickly the feedback loop converges.

3. Can the full cost ledger support the workload?

The 30-hour proof-reproduction result represents sustained, large-scale inference. Cost evaluations should include more than the price of individual requests. Failed attempts, retries, review overhead, and human intervention also need to be accounted for.

A more useful cost metric is the cost per unit of verifiable output, rather than token consumption alone.

3. The Same Underlying Model Can Have Different Service Boundaries

Another easily overlooked selection risk is that the same underlying model may be offered through different product forms. The main differences can involve safety protections and access requirements.

Based on publicly available information, public-facing versions apply stronger protections in areas such as cybersecurity and biological safety. Requests involving penetration testing, exploit development, and many research-level biological tasks may be restricted or handled with reduced capabilities. Versions available to verified organizations may provide a broader capability range, and some high scores on higher-risk benchmarks have been reported for those versions.

This means evaluations should follow three rules:

• Do not transfer scores across product versions. A seemingly leading benchmark result may correspond to a product form that your account cannot access.
• Confirm protections on a task-by-task basis. A reduction in the overall number of blocked requests does not mean that your specific workflow will be permitted. Real business use cases should be tested individually.
• Track three dimensions in parallel. Selection tables should record capability scores, protection status, and organizational access eligibility. Omitting any one of these can lead to an incorrect assessment.

4. Recommendations for Engineering Teams

1. Pilot with slices of real projects, and define automatically verifiable acceptance conditions before evaluating model performance.
2. When building multi-agent pipelines, prioritize review and rollback mechanisms instead of simply increasing the number of agents.
3. Calculate costs based on deliverable output, including retries and human review.
4. For tasks in high-risk domains, verify actual availability using the target account in advance to avoid being misled by public benchmark results.

Overall, progress in long-task capabilities is real. However, the distance between a successful demonstration and reliable project delivery still includes three major barriers: acceptance criteria, collaboration architecture, and total cost. Sound model selection requires evaluating all three alongside the model’s raw capabilities.