Frontier LLM Long-Horizon Task Capabilities: From Answering Questions to Delivering Work

1. Introduction: Long-Horizon Tasks as a New Capability Benchmark

In recent years, the focus of foundation-model evaluation has been shifting. Earlier benchmarks mainly measured single-turn question-answering accuracy, code-completion pass rates, or exam-style scores. These evaluations answered questions such as “What does the model know?” and “Can it solve a single problem correctly?”

In real-world engineering, research, and operations settings, however, tasks often unfold over hours, days, or even weeks. Requirements must be decomposed. Solutions must be iterated. Results must be verified. The process must be traceable. Failures must be diagnosable. As a result, the industry has begun to focus more on a different question: can a long task be completed independently and then effectively verified?

This is why long-horizon tasks have become an important window into the capabilities of frontier models.

Based on public information, a new generation of flagship model series generally shares the same underlying model base. Different versions are then provided according to use case and compliance requirements: a general-purpose version with stricter safeguards for a broad user base, and a targeted version with certain restrictions relaxed for compliant organizations.

The main capability breakthroughs appear to be concentrated in four categories:

• High-difficulty mathematical and logical reasoning: completing a full formal proof of a classical number-theory theorem.
• Large-scale autonomous development: fixing stubborn defects, migrating architectures across languages, and building a standalone operating system from scratch.
• Basic scientific research: addressing difficult problems in black-hole physics and identifying viral capsid structures based on structural prediction.
• Embodied manipulation and hardware development: completing fine-grained manipulation tasks in simulation benchmarks and independently completing the development loop for electronic devices.

These four categories share a common feature: the correctness of the result cannot be judged reliably from a single output. They require independent verification mechanisms, and the process itself forms a long decision chain.

This article uses these scenarios as entry points to analyze the underlying technical mechanisms and to outline reusable engineering methods and selection guidance.

2. Capability Framework: From “Answering” to “Delivering”

Understanding long-horizon tasks requires looking beyond the idea that the model has simply “become stronger.” The more important change is structural: the capability stack has shifted from producing an answer to executing and delivering a task.

Autonomous execution capability can be decomposed into four connected stages:

1. Reasoning and planning: decomposing the overall goal into subgoals, identifying dependencies, and adjusting the plan when failures occur.
2. Tool calling and execution: retrieving code repositories and documentation, and invoking compilers, test frameworks, simulation environments, and analysis software.
3. Multi-agent collaboration: using a controller agent to coordinate multiple sub-agents that work on subtasks in parallel, after which results are aggregated and passed into the next round.
4. Memory and verification: maintaining task state and accumulated knowledge, checking each intermediate result with objective signals, and ensuring traceability through logs and artifacts.

Together, these four stages form a closed loop of perception, decision-making, execution, and verification. The longer this loop becomes, the more errors accumulate over time, and the more dependent the system becomes on verification mechanisms and process management.

3. Formal Mathematical Proof: A Testbed for Multi-Agent Collaboration

Fermat’s Last Theorem states that no positive integer solutions exist for the equation:

x^n + y^n = z^n

when the integer exponent n is greater than 2.

The theorem remained unproven for more than 300 years. Its eventual proof relied on connections between elliptic curves and modular forms, completed in the 1990s. A formal version of the proof involves many prerequisite results from number theory, algebraic geometry, and representation theory. The proof consists of numerous steps that are tightly connected.

A formal proof requires each reasoning step to be expressed in a machine-checkable form, using an interactive theorem prover to verify it one step at a time. This type of task therefore tests both the model’s mathematical ability and its ability to manage long-horizon work.

Public information indicates that one flagship model used roughly a thousand collaborating agents to complete a full formal proof of the theorem in about 30 hours. By comparison, an earlier human-assisted version took about 11 days. The reported token consumption was about one-third of the original approach, and the proof length was compressed to about one-seventh.

Another flagship model with the same configuration completed only about 26% of the task within 30 hours.

The technical lessons behind these results can be summarized as follows:

• The verifier is the sole judge of correctness: whether a proof holds is determined by the proof checker, not by the model’s self-assessment. This is the fundamental advantage of formal methods over natural-language reasoning, and it makes large-scale parallel exploration feasible.
• Task-graph decomposition and parallel search: the main proof is decomposed into lemmas and intermediate theorems represented as a dependency graph. Different agents attempt different lemmas or proof strategies. Successful paths are reused immediately, while unsuccessful paths are pruned. Overall throughput depends on task partitioning granularity and scheduling quality.
• Knowledge reuse and context compression: existing results from mathematical libraries can be automatically retrieved and reused, avoiding duplicated effort. A significant reduction in token consumption and proof length usually indicates more efficient proof strategies, such as reusing existing lemmas or removing redundant steps, rather than simply writing fewer steps.
• Failure-driven iteration: errors such as “goal not narrowed,” “type mismatch,” or related proof-state issues are converted into concrete constraints for the next attempt, forming a search-like iterative process.

An important caveat is that formal proof guarantees correctness of the derivation process. If the formalization itself deviates from the semantics of the original problem, the machine may not detect the issue. Therefore, the choice of axioms and the statement of propositions still require expert review.

For enterprises, this pattern of machine-verifiable long-chain tasks can be transferred to scenarios such as compliance review, financial-rule validation, and configuration-consistency checking. In such systems, rule engines or domain-specific validators remove correctness judgment from the model, allowing the model to focus on generation and repair.

4. Autonomous Software Engineering: From Code Completion to Large-Scale Projects

4.1 The Agentic Development Loop

Traditional code completion operates at the level of individual lines written by developers. It is a local, human-in-the-loop form of assistance. Long-horizon autonomous development goes further: the development process itself is delegated to agents, while humans move toward goal definition and result acceptance.

A typical loop can be represented as follows:

1. Read the task and codebase context.
2. Create a plan and decompose it into subtasks.
3. Modify code, then run compilation, static checks, and tests.
4. Read errors and logs, identify causes, and apply fixes.
5. If checks fail, return to step 3; if checks pass, submit changes and update task state.
6. After all checks pass, generate change descriptions for human review.

This loop works only when each step has objective and automated feedback. Without automated tests and compilation checks, long loops quickly degrade into repeated edits that fail to converge.

4.2 Typical Task Types

Public examples mainly cover three types of tasks:

• Stubborn defect repair: long-standing defects often come from hidden state pollution, concurrency timing issues, or edge conditions. Agents can narrow down the problem by combining runtime logs, reproduction cases, and hypothesis validation, avoiding some of the blind spots of manual investigation.
• Cross-language architecture migration: for example, migrating a high-level scripting-language service to a system-level strongly typed language. This is a systems-engineering task that requires dependency analysis, module partitioning, incremental rewriting, and behavior-consistency validation.
• Building an operating system from scratch: in one public example, a model scheduled thousands of sub-agents to work continuously for about 8 days, producing a complete system that included a kernel, drivers, a file system, and a graphical interface. The resulting system was able to boot and run independently in a virtual machine.

Additional public information indicates that one vendor reported that about 90% of internal code iteration was model-assisted, and that the startup time of a certain service was reduced to about 1/26 of its original value.

4.3 A General Approach to Migration

In cross-language migration, performance improvement is not the only goal. Behavioral equivalence is equally important. A staged strategy is recommended:

Stage 0: Establish a baseline
       Map dependencies, set up continuous integration, and record interface contracts and performance metrics.
Stage 1: Migrate boundary layers (gateway, serialization, configuration parsing)
       Run both implementations in parallel and compare output differences under the same inputs.
Stage 2: Migrate core business logic
       Use automated rewriting for mechanical parts and manual review for complex branches.
Stage 3: Gradual traffic cutover
       Shift traffic incrementally and monitor latency, error rate, and resource usage.
Stage 4: Decommission the old implementation
       Keep a rollback path and archive test cases accumulated during migration.

Cases involving large startup-time reductions often benefit from compile-time checks introduced by static typing, multithreading or asynchronous runtimes, and optimizations to cold-start paths. These benefits depend on sufficient validation of behavioral equivalence.

Therefore, the following practices are necessary:

• Differential testing: compare outputs of the old and new implementations under the same inputs.
• Fuzz testing: use random or malformed inputs to probe boundary behavior.
• Golden-case replay: use real production requests for regression testing.

4.4 Building an Operating System from Scratch: Verification Closure Is the Key

Building an operating system from scratch is extremely difficult. It involves low-level hardware abstraction, concurrent scheduling, memory management, drivers, and user-space interfaces. Errors may only appear after the system has run for some time.

The key to agent success in this setting is an automatically triggered verification loop:

1. Code is written into a virtual disk image.
2. The image is booted in a virtual machine or emulator.
3. Self-check scripts are executed.
4. Boot logs and test results are read through serial output or logging.
5. Failure information is fed back to the agent, which then re-diagnoses and fixes the issue.

In essence, this automates the human cycle of compile, run, observe, and fix. Without such a loop, a model may write code but cannot determine whether the system can actually run.

5. Scientific Research and Embodied Intelligence: Extending Capability Boundaries

5.1 From Problem Solving to Participating in Discovery

Basic scientific research scenarios illustrate one path by which models can participate in scientific discovery.

According to public information, one flagship model independently addressed two previously unsolved black-hole physics problems. It also filled gaps in calculations related to holographic duality theory and clarified under what conditions certain quantum-gravity-related techniques are applicable.

Such results should be interpreted cautiously.

First, many problems in theoretical physics have multiple candidate models and formulations, so judging whether a problem has been “solved” can involve subjective judgment.

Second, regardless of how much derivation the model contributes, conclusions still require peer review.

A reasonable approach is to position the model as a high-efficiency computational assistant and hypothesis generator, while human researchers define the problem, set constraints, and review conclusions.

5.2 Multi-Model Fusion for Viral Capsid Structure Prediction

Another public case used a multi-model fusion approach. The model combined sequence information, evolutionary conservation, and results from different structure-prediction methods to model viral capsids as a whole. In 34 tests, 28 achieved accurate modeling.

Multi-model fusion can be effective because different models or methods rely on different data and inductive biases, and their error patterns are often complementary. Through ensemble voting, weighted averaging, or physical-constraint validation, overall accuracy can often exceed that of a single model.

When interpreting such results, three issues should be considered:

• Sample size: 34 tests produce a relatively wide confidence interval. A result of 28 out of 34 is about 82%, and statistical fluctuation cannot be ignored.
• Evaluation criteria: the definition of “accurate modeling,” such as interface-residue match rate or global folding-error threshold, can significantly affect conclusions.
• Data leakage: if test samples are highly similar to training data, results may be overestimated.

These are necessary premises for evaluating predictive tasks in any domain.

5.3 Embodied Manipulation and Hardware Development

In embodied intelligence, one flagship model achieved a 64.9% success rate on a robot-simulation manipulation benchmark. This was 37.3 percentage points higher than the previous generation and clearly above another flagship model, which scored 39.4%, and an overseas model, which scored 29.8%.

Demonstrated tasks included fine-motion-control operations such as opening bottle caps, solving a 3D puzzle toy, and drawing by hand.

The same model also completed the circuit design, CAD modeling, software development, and physical debugging of three electronic devices with only limited human intervention.

There is still a sim-to-real gap between simulation benchmark scores and real-world performance. This gap mainly appears in three areas:

• Differences in physical parameters, such as friction, mass, deformation, and lighting.
• Sensor noise and latency.
• Long-tail situations not covered by training scenarios.

Common engineering strategies include:

• Domain randomization: randomize physical parameters and visual appearance during training to improve policy robustness.
• Real-data calibration: fine-tune simulation models with a small amount of real-device data.
• Safety constraints and human takeover: apply force control and emergency-stop mechanisms for high-risk actions.

Hardware development demonstrates a complete loop from digital design to physical validation: circuit design and simulation, structural modeling, embedded software development, and prototype debugging form one continuous chain. The model handles generation and iteration, while engineers review safety and reliability. For products involving electrical safety or compliance certification, human review cannot be omitted.

6. Efficiency and Cost: From Peak Metrics to Cost per Effective Task

Public information indicates that the new generation of flagship model series achieves generation speeds up to about 30% higher than the previous generation. Basic pricing was reported at around 8 US dollars per million input tokens and around 40 US dollars per million output tokens.

The series also introduced a fast mode with a speed of about 2.5 times the normal mode, and some subscription-based usage-reset benefits were mentioned.

When evaluating model cost-effectiveness, the following metrics are more useful than peak capability alone:

• Cost per effective task = total token cost ÷ number of successfully completed tasks. Long-horizon tasks often involve many retries, and the cost of failed tasks should also be included.
• End-to-end latency: the full time from task submission to obtaining usable results, not just the generation speed of a single output.
• Agent overhead: intermediate token consumption for planning, memory, and tool calls. Under high reasoning intensity, this can be several times larger than the final output.
• Human intervention cost: the human time required for review and correction, which is often underestimated.

Public information also indicates that under high reasoning intensity, one flagship model achieved results comparable to another flagship model’s highest reasoning mode at about half the cost. This suggests that model selection should be based on actual business workload rather than unit price alone.

A common strategy is to assign models by task tier:

Simple queries, formatting, classification          → lightweight model / standard mode
Medium-complexity code generation and analysis      → flagship model / standard mode
High-difficulty reasoning and long-horizon tasks    → flagship model / high reasoning intensity or fast mode

At the same time, token usage should be monitored with budget alerts to prevent runaway retries or loops from causing unexpected costs.

7. Engineering Deployment: How Enterprises Can Use Long-Horizon Agents Effectively

For long-horizon agents to operate stably in production, platform-level capabilities are required:

1. Environment isolation: each agent task should run in an isolated environment, with restricted network access and file-system permissions, to prevent accidental operations from affecting production systems.
2. State and artifact management: task plans, intermediate results, and build artifacts should be stored in object storage to support resume-after-failure and replay. This is especially important for long tasks measured in days.
3. Asynchronous task scheduling: use message queues to distribute tasks and results, and use workflow engines to orchestrate DAG dependencies, avoiding single points of blocking.
4. Observability: collect agent decision chains, tool calls, token consumption, and task success rates in a unified way. Log services and monitoring alerts can help detect abnormal loops and cost spikes.
5. Human review gates: code merges, production releases, and outputs involving security or compliance should require human approval.

On Baidu AI Cloud, these capabilities can be implemented by combining common cloud products such as container platforms, object storage, message queues, log services, and monitoring and alerting. Foundation model capabilities can be accessed on demand through the Baidu AI Cloud Qianfan platform, after which model selection and tuning can be completed using enterprise-specific evaluation sets.

For example, the orchestration logic of a code-migration task can be simplified as follows:

def run_migration_task(repo, target_lang, config):
    """Example orchestration for a long-horizon code migration task."""
    state = load_or_init_state(repo)              # Restore task state from object storage
    plan = planner.decompose(repo, target_lang)   # Generate a DAG of subtasks
    while not plan.is_finished() and budget.remaining():
        tasks = plan.ready_tasks()                # Collect subtasks whose dependencies are satisfied
        results = parallel_map(
            [spawn_worker(repo, t, target_lang) for t in tasks],
            max_workers=config.concurrency,       # Control concurrency to avoid resource contention
        )
        state = verifier.merge(state, results)    # Use compilation, tests, and static checks as the arbiter
        state.log.append(results)                 # Record the process for replay after failure
        save_state(state)                         # Periodically persist state to support resumption
        plan = planner.replan(plan, state)        # Replan based on failure information
    return final_report(state, require_human_review=True)

This example reflects three principles:

• Separate planning from execution: the planner decides what should be done, while the executor decides how to do it.
• Separate verification from generation: success is determined by objective signals such as compilation and tests.
• Externalize state: task progress is stored externally, making long-running tasks resumable and auditable.

8. Limitations and Risks

A rational view of these breakthroughs requires clear boundaries:

• Gap between benchmarks and real-world scenarios: benchmark tasks often have clear definitions and explicit verification signals. Real business environments may include ambiguous requirements, legacy systems, and multi-party collaboration, making them significantly more difficult.
• Limited reference value of vendor disclosures: many of the figures above come from vendor statements, and evaluation criteria or prompting strategies may not be public. Enterprises should build evaluation sets from their own business data, independently re-test, and then make decisions.
• Security and compliance must come first: high-capability versions in sensitive areas such as network security and biological safety should be made available only to compliant organizations under the principle of least privilege, with access auditing and operation logging.
• Runaway risk in long tasks: long-running autonomous execution can lead to excessive resource consumption, goal drift, or generation of unsafe content. Mechanisms such as budget limits, timeout circuit breakers, and human confirmation for sensitive operations are necessary.
• Responsibility remains with the user: code, designs, and research conclusions generated by models remain the responsibility of the user. License compliance, intellectual-property review, and security review cannot be omitted.

9. Conclusion

The significance of the new generation of flagship models lies not in a single benchmark number, but in the fact that long-horizon tasks are beginning to meet the engineering conditions of being plannable, executable, and verifiable.

Multi-agent collaboration decomposes complex problems into parallel subtasks. Formal verification provides an objective standard for correctness. Simulation and automated testing allow results from both physical systems and software systems to be checked.

Together, these capabilities move models from being tools that answer questions toward becoming executors that can take on tasks.

For developers and enterprises, real competitiveness comes from embedding model capabilities into a reliable engineering system: constraining outputs with explicit verification signals, controlling costs with strong observability, and protecting risk boundaries with tiered human review. On this foundation, cloud platforms such as Baidu AI Cloud can provide common infrastructure and model services, allowing teams to focus more on the business itself and deploy long-horizon agents more smoothly in real production environments.