The LLM Race Shifts from Question Answering to Closed-Loop Task Execution

Over the past year, the pace of LLM releases has clearly accelerated, with new versions arriving in quick succession. What deserves more attention from developers than the growing number of model variants is a migration in evaluation focus: from “can a model get a single-turn question right?” to “can it reliably finish a multi-step task?” A recent flagship release illustrates this divergence—it showed limited progress on some comprehensive intelligence leaderboards, but made large gains in automation, agent orchestration, scientific tool use, and system-level tasks. That contrast is a key entry point for understanding the capability boundaries of the next generation of models.

1. Benchmark Divergence: Not a Regression, but a Different Ruler

Start with the phenomenon. A new-generation model was roughly on par with the previous generation in comprehensive intelligence evaluations, and in some cases slightly below contemporaneous competitors. Yet on several system-level task benchmarks, its scores rose sharply:

• Automation task benchmark: from about 18% to about 41%;
• Terminal scientific-task benchmark: from about 22% to about 65%;
• Vulnerability-exploitation benchmark: close to a perfect score;
• Abstract-reasoning benchmark: close to 99% under a high-compute configuration.

Looking only at the first table, it is easy to conclude that “the new generation is not much better.” Putting the two tables together leads to a very different conclusion: the center of gravity in model upgrades has moved from knowledge retrieval and single-step reasoning to multi-step planning, tool invocation, and closed-loop execution.

This is not an isolated case. Comprehensive intelligence benchmarks mostly use single-turn Q&A, limited context, and static answers; they measure the coverage of a model’s internal knowledge and the stability of its reasoning chain. Automation and agent benchmarks, by contrast, measure a model’s ability to break down goals, call tools, handle intermediate states, and recover from errors in a real environment. These are almost different dimensions of evaluation. Asynchronous scores are normal; synchronized scores are the exception.

For developers, this raises a practical issue: using old leaderboards for technology selection may identify an “answer specialist” rather than a “work specialist.”

2. Why Comprehensive Intelligence Scores Have Stopped Rising

The marginal returns from comprehensive intelligence benchmarks are diminishing for three reasons.

First, task difficulty is approaching the upper bound of the data distribution. These benchmarks are mostly written by experts, with well-formed questions and unique answers. Models have already encountered large amounts of similar distributions during training. Further improvement requires higher-quality long-tail data, not simply larger parameter counts, and the return on investment declines noticeably.

Second, single-turn evaluation cannot reflect engineering capability. Answering a question depends on knowledge matching and a reasoning chain; completing a task depends on goal decomposition, state management, tool selection, and exception handling. The latter cannot be measured by static Q&A, so the scores naturally do not show up on old leaderboards.

Third, the evaluations themselves have a saturation effect. When the gap between leading models on comprehensive intelligence leaderboards narrows to a few percentage points, the leaderboard’s discriminative power is no longer sufficient to support model-selection decisions. The industry generally agrees that evaluations closer to real workflows are needed—for example, multi-turn tasks with tool environments, long-horizon tasks with file systems, and automation flows that require failure recovery.

3. Four Capabilities That Are Actually Creating Separation

Public data shows that improvements in the new generation of models are concentrated in four directions. These four capabilities also form the technical backbone of current agent deployment.

3.1 Automated Task Execution

Automation benchmarks typically require a model to complete a series of operations in a given environment: read input, plan steps, execute actions, and validate results. The jump from 18% to 41% means the model has made substantive improvements in step-planning success rate and correct maintenance of intermediate state.

From an engineering perspective, the value of this capability lies in reducing the frequency of human-in-the-loop intervention. In traditional automation flows, every step may need a human fallback. When a model can reliably complete multi-step operations, humans only need to review key checkpoints, and overall throughput improves significantly.

3.2 Agent Orchestration

The core of agent orchestration is not a single call, but coordination across multiple roles, multiple tools, and multiple turns. A typical orchestration chain includes a task decomposer, tool selector, executor, result validator, and failure retrier. The model must understand the responsibility boundaries of each role and pass context correctly between them.

The key metric here is not “can it call a tool?” but “can it switch to another path after a failed call?” The score improvement shows that the model is more robust in error recovery and path replanning, which is the dividing line between an agent demo and a production system.

3.3 Scientific Tasks and Tool Use

The sharp rise in the terminal scientific-task benchmark reflects the model’s ability to operate within structured toolchains: reading and writing files, running scripts, parsing output, and iteratively correcting errors. These tasks impose extremely high requirements on format correctness; a single misplaced field can cause the entire chain to fail.

The lesson for developers is that the reliability of tool invocation depends on how well the model understands the tool contract. When designing tool interfaces, define parameter types, return structures, and error-code semantics clearly, and include failure examples in prompts. The stronger the model, the greater the benefit from standardized interface design.

3.4 Long-Horizon Task Execution

Long-horizon task execution tests context management and goal retention. The longer the task, the more intermediate states there are, and the more likely the model is to drift from the initial goal or lose key constraints. Improvements in the new generation of models are mainly reflected in the retention rate of key information in long contexts and convergence toward stage goals.

In engineering practice, it is advisable to split long tasks into stages with clear acceptance criteria. After each stage, take a state snapshot and verify the goal to prevent error accumulation.

4. What High Abstract-Reasoning Scores Mean

The near-perfect abstract-reasoning score deserves separate discussion. This type of benchmark tests the ability to induce rules from a small number of examples and transfer them to new problems. Unlike traditional knowledge Q&A, the answer is not in the training data; it must be induced on the spot.

A high score means the model has strong pattern recognition and rule generalization capabilities, which is especially important in agent scenarios—many problems in real environments have no precedent, and the model must infer operating rules on the spot. However, a high abstract-reasoning score does not equal engineering reliability. Cases in which the rule is induced correctly but an execution step goes wrong are still common, so engineering-level validation and fallback mechanisms remain necessary.

5. Selection and Deployment Recommendations for Developers

Based on the analysis above, here are several actionable recommendations.

First, build scenario-specific evaluation sets. Do not make decisions based only on public leaderboards. Sample 50–100 real tasks from your own business, covering four categories: single-step Q&A, multi-step automation, tool invocation, and long-horizon execution. Record success rates and average completed steps for each. Example evaluation dimensions are shown below:

Scenario type           Metric                          Sample size
Single-step Q&A         Answer accuracy                 30
Multi-step automation   Task completion rate/steps      30
Tool invocation         Invocation success/retries      20
Long-horizon execution  Goal retention/stage convergence 20

Second, prioritize validating error-recovery capability. Actively inject failures during evaluation—tool timeouts, malformed returns, missing intermediate results—and observe whether the model can recognize the failure and switch paths. This reflects production readiness more effectively than success rate alone.

Third, design tool interfaces by contract. Define parameter schemas, return structures, and error codes, and provide positive and negative examples in the system prompt. The more standardized the interface, the more stable the model’s calls will be.

Fourth, decompose long tasks into stages. Set verifiable completion criteria for each stage, take state snapshots between stages, and avoid goal drift caused by context expansion.

Fifth, retain human review checkpoints. Set confirmation steps before high-risk operations such as write operations, external calls, and resource changes. Improved model capability does not replace risk-control design.

6. Conclusion

Stagnating comprehensive intelligence scores alongside surging system-level task scores are not a contradiction; they are a normal manifestation of a switch in evaluation dimensions. The message is that the focus of competition for the next generation of models is shifting from “how much they know” to “how much they can do.” For developers, selection criteria should change accordingly—from looking at leaderboard rankings to looking at closed-loop task capability, and from evaluating the quality of a single answer to evaluating the stability and recoverability of multi-step execution.

When building agent applications on the cloud, common cloud primitives—such as object storage for intermediate artifacts, message queues to decouple task stages, logging to record execution paths, and monitoring and alerting to catch exceptions—can be combined to create an observable execution framework. This places model capabilities inside an engineering system that is verifiable and rollback-capable. Models will continue to iterate, but evaluation and engineering methods centered on closing the task loop will remain effective for a long time.