From Answering to Executing: The Technical Logic Behind Agent Capability Leap

1. A Phenomenon Worth Examining: Stagnant Scores, Breakthrough Capability Elsewhere

A next-generation foundation model recently released by a major cloud provider sparked considerable discussion across the industry. The focus of the conversation was not how many leaderboard records it broke, but a striking paradox: on traditional general-intelligence benchmarks, the new model scored nearly the same as its predecessor — and in certain dimensions was even surpassed by competing models released around the same time. Yet on automation, agent-based tasks, scientific computing, and tool invocation workloads, its performance showed a dramatic, discontinuous lead.

This disconnect — flat scores alongside sharply improved real-world capability — left many developers puzzled at first glance. If general intelligence did not improve, what justifies calling it a new generation?

To answer that question, we need to examine a more fundamental issue: Is the yardstick we use to measure model capability still fit for purpose?

2. The Structural Mismatch in Evaluation Systems

2.1 What Traditional Benchmarks Actually Measure

Over the past few years, mainstream LLM evaluation suites — whether knowledge Q&A, mathematical reasoning, code completion, or multi-turn dialogue — have essentially measured one thing: given a defined context and a well-specified question, can the model produce the correct answer?

These evaluations rest on several implicit assumptions:

• The problem is closed-ended, with a standard answer.
• The task boundary is clear, requiring no decomposition by the model.
• A single turn or a small number of turns is sufficient; long-horizon planning is not involved.
• Tool use and environment interaction are outside the scope.

Under these assumptions, benchmark scores do reflect a model’s knowledge base and reasoning ability reasonably well. The problem is that real-world tasks rarely satisfy these assumptions.

2.2 The Complexity of Real Tasks Is Severely Underestimated

The tasks developers face in practice look more like this:

• Requirements are ambiguous and must first be clarified and decomposed.
• Multiple steps are involved, with dependencies between them.
• External tools need to be called, databases queried, and files read and written.
• Execution may fail midway, requiring retries and error recovery.
• The final deliverable is a working system, not a block of text.

The quality of completing such tasks depends on the model’s ability to perform long-horizon planning, state tracking, tool orchestration, and adaptive error correction — capabilities that conventional evaluation systems barely cover.

2.3 What Stagnant Scores Actually Mean

When a next-generation model scores roughly the same as its predecessor on general-intelligence benchmarks, the reasonable interpretation is not “no progress,” but rather:

The capabilities measured by traditional evaluations are approaching the point of diminishing returns for current architectures, while another category of capabilities that truly determines real-world productivity is improving rapidly — but has not yet been effectively captured by mainstream evaluation systems.

This is analogous to evaluating a triathlete solely on their 100-meter sprint time. Their sprint performance may be unchanged, but improvements in swimming, cycling, and endurance are what determine whether they can finish the race at all.

3. Three Key Dimensions of the Capability Leap

Based on publicly available technical information and developer experience, the next-generation model has achieved significant advances in three dimensions.

3.1 Long-Horizon Task Planning and Execution

Traditional models excel at single-step optimization but tend to lose track of the overall goal or fall into local loops when faced with multi-step collaborative tasks. The new generation introduces more explicit task decomposition and state management mechanisms at the architectural level, enabling it to:

• Break down high-level goals into executable sub-task sequences.
• Evaluate progress after each step and dynamically adjust subsequent plans.
• Proactively attempt alternative paths when blocked, rather than failing outright.

This improvement is particularly evident in automation benchmarks. For instance, in end-to-end automation tasks, the new model’s completion rate jumped from under 20% for the previous generation to over 40%. In scientific computing and terminal operation tasks, the completion rate rose from just above 20% to over 60%. Behind these numbers lies a fundamental shift from “giving advice” to “actually getting things done.”

3.2 Tool Invocation and System Interaction

The core of modern agent applications is whether a model can reliably invoke external tools to perform actual work. This includes:

• API calls: understanding interface semantics, constructing valid requests, and handling responses.
• File operations: reading, writing, retrieving, and modifying structured and unstructured data.
• Environment awareness: determining the next action based on execution results, rather than blindly following a preset script.

A typical tool invocation loop can be expressed in pseudocode:

while not task_complete:
    observation = environment.get_state()
    plan = model.plan(goal, observation, history)
    action = model.select_action(plan, available_tools)
    result = environment.execute(action)
    history.append((action, result))
    if result.is_error():
        model.reflect_and_retry()

The new generation model performs more consistently within this loop: tool selection is more accurate, parameter construction has fewer errors, and error recovery is more timely. This directly explains its standout performance on automation and system-level tasks.

3.3 Complex Reasoning and Exploratory Tasks

In domains requiring deep reasoning and exploration, the new model also shows notable progress. On abstract reasoning and pattern discovery tasks, it approaches near-perfect scores on certain high-difficulty tests, indicating that it can effectively generate and validate hypotheses when confronted with unseen rules and structures.

The value of this capability lies in the fact that when a task has no ready-made answer and requires the model to independently explore a solution path, it no longer gets stuck easily.

4. Why Traditional Evaluations Miss These Changes

Once we understand the direction of the capability shift, it becomes clear why traditional evaluations fail to “see” these advances.

Dimension	Traditional Evaluation	Real Agentic Tasks
Task format	Single-turn Q&A	Multi-step execution
Environment interaction	None	Present
Tool use	Not involved	Core capability
Failure recovery	Not assessed	Critical metric
Deliverable	Text answer	Working result

Traditional benchmark design inherently excludes tool use, environment interaction, and long-horizon planning. As a result, a model can have completely flat scores on these benchmarks while its usefulness in real-world tasks improves substantially.

This is not to say traditional evaluations have no value — they remain effective for measuring knowledge breadth and foundational reasoning. The issue is that as the industry moves from conversational AI to execution-oriented AI, evaluation systems need to evolve in tandem.

5. Practical Implications for Developers

5.1 Redefining What Makes a “Good Model”

When selecting a foundation model, developers should gradually shift from “leaderboard scores” to “task completion rates.” Specifically, consider the following metrics:

• End-to-end task success rates in your own business scenarios.
• Tool invocation accuracy and error recovery capability.
• Goal persistence during long-task execution.
• Integration cost and stability with existing systems.

5.2 Building an Agent-Oriented Application Architecture

If your application involves automated workflows, multi-step tasks, or system interaction, consider the following patterns in your architecture:

• Task decomposition layer: converting user intent into executable sub-task sequences.
• Tool abstraction layer: uniformly wrapping external APIs, databases, and file operations.
• State management layer: tracking task progress and supporting interruption recovery.
• Evaluation and feedback layer: validating results at key checkpoints and triggering retries or human intervention.

5.3 Build Your Own Evaluation Set

Public leaderboards can only serve as a reference. What truly matters is an evaluation set built from your own business scenarios. Extract samples from real tasks, covering normal paths, edge cases, and failure scenarios, and evaluate model performance on a regular basis to establish a sustainable basis for model selection.

6. Conclusion

Flat general-intelligence scores do not mean a model has not progressed. On the contrary, when scores stagnate while real-world task completion capability improves dramatically, we are witnessing a shift in the center of gravity of model capability: from “can it answer questions” to “can it get things done.”

For developers and enterprises, this is both a challenge and an opportunity. The challenge is that old selection criteria may no longer apply. The opportunity is that those who first understand and embrace this shift will be positioned to build agent applications that solve real problems.

Evaluation systems will eventually catch up. But until then, the best benchmark is your own business scenario.