Evaluating Long-Task Capability and Cost-Effectiveness in Next-Generation LLMs: From Prototype Systems to Formal Proofs

1. Why Long-Task Capability Suddenly Matters

For the past two years, mainstream LLM evaluation has focused on short-horizon work: single-turn Q&A, single-file code generation, and one-step reasoning. These benchmarks describe a model’s immediate ceiling, but they do not answer the question enterprise users care about most: can a model independently take on a real engineering task that spans days, hundreds of iterations, and multiple collaborating modules?

Recently published experiments with a new generation of models offer a notable signal. Models are beginning to sustain a continuous cycle of “decompose, execute, and integrate” across multi-day timescales. Two landmark experiments stand out:

• Building a prototype operating system. Starting from an empty folder, a model coordinated more than 1,000 sub-agents through repeated coding, testing, and review. The result could boot on laptop hardware and included a kernel, disk and keyboard drivers, a network protocol stack, persistent file storage, a window system, a login interface, and an installer.
• Formalizing an existing mathematical proof. One thousand agents running an early model version spent about 30 hours rewriting a well-known proof—previously formalized by hand over 11 days—into a form in which every step can be checked automatically by a computer.

The boundaries of both “successes” are explicit. The operating-system test did not cover long-term stability or hardware compatibility. The formalization made an existing proof machine-checkable; it was not an original mathematical discovery. Understanding those boundaries is the starting point for any later evaluation.

2. A Technical Breakdown: The Three-Layer Stack Behind Long Tasks

Why these results are possible can be analyzed at three levels.

2.1 Task decomposition and planning

The first step in a long task is turning a vague high-level goal—“build an operating system that boots”—into an executable graph of subtasks. That requires the model to:

• Reason about dependencies. For example, driver development depends on stable kernel interfaces, and testing depends on a minimal bootable kernel.
• Split work into milestones. A goal measured in weeks must become reviewable units measured in hours, each with a clear completion criterion.
• Recover from failure. When a subtask fails repeatedly, the system should revise the upstream design instead of looping locally.

2.2 Multi-agent collaboration

The “more than 1,000 sub-agents” in these experiments is not marketing language. It describes an engineered parallel architecture. A typical pattern is:

Coordinator (Planner)
 ├── Generator (Writer): produces code or proof fragments
 ├── Reviewer: performs static checks and logical review
 ├── Tester: compiles, runs, and formally verifies
 └── Integrator: merges branches, resolves conflicts, and runs regression checks

Sub-agents pass intermediate artifacts through structured messages. Review and testing create a closed loop similar to continuous integration in software engineering. The point of the architecture is to dilute single-inference errors through repeated cross-checking. A model that is 95% accurate on one step retains only about 36% success across 20 unchecked steps in sequence. Independent review at each step can catch and repair errors locally.

2.3 State and memory

The largest engineering challenge in multi-day work is the limited context window. Practical approaches include persisting key decisions and interface contracts to external storage, giving each subtask a self-contained task card with background, constraints, and acceptance criteria, and taking periodic snapshots of global state. This is also why long-task experiments often end with a deliverable that can be inspected as a whole: it is the final check that externalized state remains consistent.

3. A Clear View of Capability Limits

These results are exciting, but their conditions of applicability matter:

1. “It boots” is not “it is production-ready.” A prototype that passes a boot test is still orders of magnitude away from long-term stability, security hardening, and ecosystem compatibility. Formal reproduction also depends on an existing proof as a reference answer. That is not comparable to the difficulty of an open research problem.
2. The cost structure is still opaque. Running thousands of agents continuously for dozens of hours consumes inference compute that, once converted into a bill, may far exceed the labor cost of an engineer doing the same work. There is a full economic calculation between “it can be done” and “it is worth doing.”
3. Acceptance criteria decide the value. Model output must be strictly verifiable—successful boot, or a proof that passes checking. Many real business tasks have ambiguous acceptance criteria. The vaguer the criteria, the larger the discount on the value of long-task automation.

4. A Practical Framework for Model Selection

Enterprises that want to use this capability in real projects should build a four-part evaluation matrix:

Dimension	Key question	Recommended practice
Capability ceiling	Do benchmark scores cover my task types?	Build a small evaluation set from real business samples and score the model on it
Safeguards and compliance	Are high-risk domains, such as security and biology, blocked?	Confirm the protection scope item by item and test fallback behavior after a request is blocked
Access eligibility	Does the advanced version require organizational verification?	Verify your own eligibility so the selection sheet does not cite results from an unavailable version
Cost accounting	Token usage × unit price × retry rate	Run an end-to-end trial and record the full bill instead of estimating it

The third point deserves special attention. The same underlying model may be offered as different products because of safeguard policies and access rules. Scores reported for a high-threshold version in public benchmarks cannot be applied directly to the version available to an ordinary account. A seemingly leading score may correspond to a product configuration you cannot use at all. That is one of the most common traps in current model selection.

5. A Practical Adoption Path: From Assistance to Ownership

Based on the analysis above, a pragmatic adoption path is:

1. Stage 1: Assisted generation. Break long tasks into small units driven by people. The model implements individual pieces, and people accept the results. The main goal is to accumulate a library of acceptance criteria.
2. Stage 2: Controlled agents. Let the model complete subtasks with explicit acceptance criteria inside a sandbox, such as developing a single module or organizing documentation into a formal structure. Build automated test guardrails.
3. Stage 3: Project-level delegation. Only after automated acceptance coverage is high enough, and failure-recovery mechanisms have been validated, consider handing a complete project to an agent cluster—while keeping a final human review.

At every stage, connect the work to a unified observability stack: logging, monitoring and alerts, and a task-tracking board. Every agent decision should be traceable and reversible. Mainstream cloud platforms already offer product categories for this infrastructure. The setup cost is modest, but it is a prerequisite for reliable long-task operation.

Conclusion

The jump in long-task capability marks a key step from LLMs as “Q&A tools” toward “digital labor.” For technical decision-makers, however, judgment is more valuable than excitement. Understand the conditions under which an experiment holds, calculate the full cost, build automated acceptance, and then let the model take on more work inside boundaries you control. Capability will keep improving. A sound method for evaluation and adoption is the reusable asset that lasts across technology cycles.