Benchmarks Plateau, Execution Scores Surge: Why Agentic Evaluations Are Redefining LLM Capability

Introduction: A Notable Divergence

Recent flagship model releases from leading vendors have sparked an interesting shift in industry discussion: on traditional comprehensive intelligence benchmarks, new-generation models show only marginal improvements over their predecessors—and in some dimensions, no progress at all. Yet on evaluations targeting real-world execution—automated workflows, terminal environment tasks, scientific computing, and tool calling—scores have jumped dramatically, with some public results approaching double the previous generation’s performance.

This structural divergence—flat composite scores but soaring execution scores—is not benchmark gaming. It signals a clear trend: the evaluation paradigm for large language models is moving from “can the model answer questions” to “can the model get things done end-to-end.” This article unpacks the technical logic behind this shift and offers developers a practical evaluation methodology they can implement directly.

Why Traditional Comprehensive Benchmarks Fail to Measure New-Generation Models

1. Saturation Effects: Collapsing Discriminative Power

Traditional comprehensive benchmarks rely heavily on knowledge Q&A, subject reasoning, and multiple-choice judgments—testing the ability to produce a single answer from a given input. After multiple generations of iterative training, top models have approached the ceiling of these benchmarks. The difficulty gradient of questions no longer distinguishes model capability effectively. When composite scores hover within a narrow range, the plateau likely reflects the measurement instrument’s limits, not stagnation in model capability.

2. Data Contamination: The Inherent Flaw of Static Questions

Questions from public benchmarks have almost certainly appeared in pretraining corpora, meaning models may have “memorized the answers.” This creates a double distortion: it may overestimate knowledge generalization, and when evaluation teams introduce new contamination-resistant question sets, score improvements appear sluggish. Static, public, long-unchanged question sets are inherently unsuitable for measuring models that keep improving.

3. Weighted Averages Mask Structural Capability Shifts

Composite intelligence scores are typically weighted averages of multiple sub-scores. When a new-generation model allocates substantial training compute toward tool use, long-horizon planning, and environment interaction—the “execution side”—these capabilities carry little or no weight in traditional benchmarks. The composite score simply cannot “see” this progress. Averages are smooth; structural capability changes are sharp.

4. Single-Turn vs. Multi-Turn: A Measurement Mismatch

Traditional benchmarks assume “one correct answer ends the task.” Real engineering tasks are multi-turn interactive processes: observe environment → decompose and plan → call tools → correct based on feedback → deliver results. Measuring multi-turn execution ability with single-turn exam questions is like hiring a driver with a written test—theoretical perfection does not mean the person can safely operate a vehicle.

What Agent-Based Evaluations Actually Measure

Dimension	Traditional Q&A Benchmarks	Agent Task Benchmarks
Input format	Single-turn question, one response	Multi-turn interaction with continuous environment feedback
Scoring method	Comparison against standard answers	Verifiable conditions on final state
Capabilities assessed	Knowledge recall, static reasoning	Planning, tool calling, error correction and recovery
Interaction environment	Stateless text	Sandboxed terminal, codebase, file system
Additional metrics	Accuracy	Completion rate, steps, cost, failure modes

Specifically, execution-oriented evaluations focus on six dimensions:

• Task completion rate: Scored by whether the final state satisfies verifiable conditions—e.g., whether files were written correctly or all tests passed.
• Long-horizon planning and state tracking: Whether the model maintains goal consistency across dozens of steps without drifting or forgetting.
• Tool call quality: Whether function selection is appropriate, parameters are constructed accurately, return values are validated, and retry or fallback strategies exist after failures.
• Environment interaction: Executing, observing, and debugging in real environments such as terminals, code repositories, and browsers.
• Cost and efficiency: Number of steps, token consumption, and latency to complete a task.
• Safety boundaries: Restraint in handling unauthorized operations and dangerous commands.

Technical Drivers: Why Execution Capability Suddenly Shifted Gears

1. Training Paradigm: From Imitation to Reinforcement

Pretraining primarily addresses knowledge and language; task execution ability comes largely from reinforcement learning with verifiable rewards. Execution tasks naturally offer objective terminal-state signals—whether code compiles, tests pass, or target files are generated correctly. These rewards are clear and automatable, enabling models to learn planning and error correction across large volumes of task trajectories.

2. The Compound Effect of Inference-Time Compute

Extended reasoning, self-reflection, and “plan–execute–verify” loops allow models to spend compute on intermediate steps. For single-turn Q&A, thinking longer yields limited gains. But for multi-step tasks, reliability improvements at each step compound across the entire chain—if per-step reliability rises from 90% to 99% on a ten-step task, end-to-end success jumps from roughly 35% to 90%.

3. Standardized Tool Calling and Expanded Context

Function-calling protocols have become more uniform, allowing models to read and write structured results reliably. Long context windows and external memory mechanisms preserve cross-step state, providing the infrastructure for long-horizon task execution.

4. Systems Engineering: Scores Reflect “Model + System”

A real agent system is the model plus scaffolding: sub-task decomposition, sub-agent orchestration, sandbox execution, and result verification. Evaluation scores typically include these system-level optimizations, so they measure overall execution capability rather than the raw model’s single-turn intelligence.

How Developers Can Build Their Own Evaluation Framework

1. Choose Dimensions Based on Business Scenarios, Not Composite Rankings

If your business is primarily knowledge Q&A, traditional benchmarks still offer reference value. If your business involves automated operations, data processing, or code generation, prioritize task completion rate, step efficiency, and failure modes over composite intelligence scores.

2. Build a Proprietary Evaluation Set

• Sample tasks from real tickets and logs, then anonymize and store them.
• Define verifiable completion criteria (terminal-state assertions) for each task.
• Maintain difficulty tiers: easy / medium / hard / adversarial samples.
• Rotate questions periodically to prevent overfitting to the evaluation set itself.

3. A Minimal Evaluation Skeleton

# A minimal agent evaluation skeleton
def run_case(case):
    env = Sandbox(case.setup)            # Sandbox environment, isolated from real systems
    trace = []
    for step in range(case.max_steps):
        action = agent(env.observe(), tools=env.tools)
        obs = env.step(action)
        trace.append((action, obs))
        if env.finished:
            break
    passed = case.verifier(env.final_state)   # Verifiable terminal-state scoring
    return {
        "passed": passed,
        "steps": len(trace),
        "tokens": env.token_usage,
        "fail_mode": classify_failure(trace) if not passed else None,
    }

The scoring principle: rather than letting the model self-evaluate, check objective terminal states—whether file contents are correct, whether database records meet expectations, or whether all unit tests pass.

4. Watch the Cost Curve

At the same completion rate, compare steps, token consumption, and retry rates across models. Execution-oriented models may be “slow and expensive.” Use “unit success cost”—the average cost per completed task—as the value metric, rather than looking only at raw token pricing.

5. Gradual Rollout and Online Observability

After offline evaluation passes, roll out gradually by traffic percentage. Combine with standard observability tooling such as call logs, distributed tracing, and monitoring alerts to track online task completion rates and failure attribution. Managed model service platforms (e.g., Baidu AI Cloud) typically provide hosted inference, batch evaluation, log services, and monitoring alerts out of the box—these can be reused directly to avoid building duplicate evaluation and operations infrastructure.

Safety and Rationality: Required Reading Beyond Benchmarks

• Least privilege: Agents hold real execution permissions and must follow the principle of least privilege, supported by sandboxing, human confirmation for dangerous operations, and full audit trails.
• Beware of benchmark overfitting: Leaderboard scores do not equal business value. A combination of dynamic question sets and manual sampling is more reliable.
• Failure-mode analysis matters more than total scores: Identifying where a task stalls—planning vs. tool parameters—yields more targeted optimization value than chasing a higher total score.

Conclusion

“Flat composite scores, soaring execution scores” is not a contradiction. It signals that the industry’s evaluation focus is migrating from “can the model answer questions” to “can the model get things done.” For developers, the pragmatic path is to stop chasing leaderboard rankings and instead build a proprietary evaluation system centered on task completion: sample real tasks, score with verifiable terminal states, weigh cost and failure modes equally, and translate model capability into measurable business outcomes. That is the real value of new-generation models.