From Benchmarks to Agents: The Next Paradigm for Large Model Evaluation

Introduction: An Unusual Preview

For a long time, pre-release messaging for new large-model versions has been highly homogeneous: larger parameters, faster inference, new highs on leaderboards. Users have long been accustomed to expecting the next generation through the same set of claims.

Recently, however, when a leading AI research organization discussed its next flagship model—referred to internally by a codename circulating in the industry—the core message was very different. The organization stated clearly that, based on preliminary internal evaluations, it could not yet rule out that the new model had reached a “Critical” level of cybersecurity capability, whereas its previous-generation model was rated only “High” under the same evaluation system.

That wording deserves repeated reading. It promised no score, but released a stronger signal: the capability curve may be switching from a “smooth rise” to a “step change.”

1. Why “Safety Level” Deserves More Attention Than “Leaderboard Score”

1.1 Risk tiers are another coordinate system for capability evaluation

The industry generally uses four risk levels—Low, Medium, High, and Critical (naming varies slightly by organization, but the logic is consistent): the higher the level, the greater the model’s capability to cause material impact without human intervention. What makes this coordinate system special is that it does not measure “how good” a model is, but “how much real-world consequence” it can produce.

Leaderboard scores are pushed higher every year by newer models, and their discriminative power keeps declining. Once a risk level is raised, however, it means the model has crossed certain capability thresholds with real consequences—for example, autonomously discovering and exploiting software vulnerabilities, or independently completing multi-step complex operations. Such capabilities are hard to quantify with a single benchmark, yet they directly determine what the model “can do” in the real world.

1.2 From High to Critical: crossing a threshold, not raising a score

The previous-generation model was only “High,” while the new generation is approaching “Critical.” The gap is not “getting more questions right,” but “being able to do things it previously could not.” An analogy: a student going from 85 to 92 is quantitative change; going from “completing an experiment under instruction” to “independently designing and completing an experiment” is qualitative change. The evolution of large models today sits on the boundary between these two kinds of change.

2. Two Logics of Progress: Linear Score Gains vs. Step-Change “Independent Completion of the Whole Task”

To understand the value of next-generation models, we first need to distinguish two fundamentally different types of technical progress:

• Type 1: Doing the same thing better. A few more percentage points on math, a higher pass rate on single coding problems, lower response latency. This kind of progress improves the experience curve but does not change the basic form of human-AI collaboration.
• Type 2: Moving from unable to do to able to do. The model can continuously plan, execute, and correct until it closes the loop and delivers a complete result. This kind of progress changes the division of labor itself.

The second type truly determines the upper bound of next-generation model value, because it directly answers the question users care about most: After I hand off a task, do I still need to watch it the whole time?

3. Agentic Coding: From Q&A Assistance to Delegated Collaboration

3.1 Today: called intelligent programming, but really “the human as project manager”

The mainstream form of coding assistance today is still a high-frequency Q&A loop: the developer describes a bug, and the model suggests a fix; the developer copies and runs it and gets an error; pastes the error back, and the model revises; tests fail, and the conversation continues.

Throughout the process, the model is an “advisor,” and the human is the “project manager”: breaking down tasks, running the environment, checking results, and making decisions. Even if the quality of the model’s one-shot code is high, collaboration efficiency remains locked by the fact that “the human must stay in the loop.”

3.2 Target form: give the goal and acceptance criteria, leave the rest to the model

The workflow pursued by agentic coding is:

read repository → locate issue → write patch → run tests → analyze failures → iterate → review results → output change report

What the developer provides is no longer “what to do next,” but “what must be achieved”:

# Delegated task ticket (illustrative)
task = {
    "repo": "internal/payment-service",
    "issue": "Order amount is occasionally miscalculated under concurrent scenarios",
    "acceptance": [
        "Reproduce and locate the root cause",
        "Submit a fix and pass all existing tests",
        "Add concurrency regression tests",
        "Output a change description and risk self-assessment"
    ],
    "constraints": {
        "sandbox": True,        # Execution allowed only in an isolated sandbox
        "network": "deny",      # External network access denied by default
        "max_runtime": "30m",   # Timeout automatically suspends and requests human takeover
        "require_approval": ["git push", "db migration"]  # High-risk actions require approval
    }
}

Half an hour later, what the developer returns to accept is not “a piece of code,” but a self-consistent deliverable containing root-cause analysis, a diff, and a test report. That is what a “next-generation model” should look like.

3.3 Reshaping the developer’s role

This does not mean developers are replaced; their role moves upward: from “the person who writes code” to “the person who sets goals, defines constraints, and accepts results.” Code review, architectural decisions, and requirement clarification become more important, not less. Skills engineers should build in advance include: writing precise requirements and acceptance criteria, designing automatically verifiable test systems, and reviewing model-generated changes.

4. A New Evaluation Paradigm: Continuous Working Duration and Number of Human Takeovers

4.1 Why traditional metrics start to fail

Leaderboard scores measure “quality of a single response,” while agentic scenarios care about “overall delivery of long-chain tasks.” A model that scores full marks on a single answer but drifts off after ten steps may have far lower production value than a model whose single responses are unremarkable but that can reliably close the loop.

4.2 Three metrics worth building in-house

• Continuous working duration: how long the model can keep advancing a task without human intervention;
• Average takeover interval: the effective work the model completes autonomously between two human interventions, analogous to mean time between failures (MTBF) in system reliability;
• Distribution of takeover reasons: classify each takeover as “unclear goal,” “blocked environment,” “insufficient permissions,” “model judgment error,” etc., to systematically expose weaknesses.

Teams should build a logging habit in daily use: each time you have the model execute a multi-step task, record the elapsed time, number of takeovers, and takeover reasons. This is closer to your real business value than any third-party leaderboard.

5. Stronger Capabilities Require Tighter Constraints: Safety Governance Upgrades in Parallel

Once capabilities cross a threshold, safety is no longer a “compliance check before release” but infrastructure that runs through training, evaluation, and deployment. Based on public practices from leading organizations, key measures include:

• Isolated environments: model inference and tool calls run in restricted sandboxes, with file systems, processes, and networks configured under least privilege;
• Network access control: default deny, allow as needed, and audit everything;
• Weight and artifact protection: model weights are treated as core assets with tiered access and leak-prevention controls;
• Behavior monitoring: full-chain logging for long-chain tasks, with real-time circuit breaking for anomalous behavior;
• Red-team evaluation and staged release: internal testing first, then limited traffic, then gradual rollout, with clear exit criteria at each stage.

One internal logic is worth noting: the more openly an organization discusses safety, the more likely it is to have greater confidence in the model’s true capability level. Safety investment and capability level are becoming two sides of the same coin.

6. How Developers and Enterprises Should Prepare

6.1 Technical side: treat “controllability” as a first-class citizen

• Design explicit acceptance criteria for agent tasks, and prioritize tasks that are automatically verifiable (with tests and static checks);
• Execute in sandboxes by default, and require human approval for sensitive operations (releases, migrations, deletions);
• Build complete task logs and audit trails, and achieve observability through general cloud infrastructure such as logging services and monitoring/alerting.

For example, general-purpose cloud capabilities such as large-model services, container platforms, object storage, logging, and monitoring can be used to build the isolated execution environment and audit system required for agent operations. When selecting a platform, focus on permission isolation, data residency, and audit completeness rather than comparing model parameters alone.

6.2 Process side: redesign human-AI division of labor

• Grade tasks by “goal clarity” and “verifiability”: delegate highly verifiable tasks boldly, and keep humans in the loop for tasks with low verifiability;
• Define responsibility boundaries clearly: before model output enters the main branch, it must be reviewed and approved by a human owner;
• Build a failure-case library: every takeover is a sample for improving prompts, toolchains, and processes.

7. Conclusion: Don’t Just Stare at Benchmarks—Watch How Long It Can Work Independently

Back to the initial question: Is the next generation of large models worth looking forward to?

If it brings only a few more percentage points on leaderboards, the excitement will be short-lived. But if it truly crosses the threshold of “independently completing the whole task”—even if initially only in restricted environments and for limited durations—the form of human-AI collaboration will be profoundly rewritten.

For practitioners, the most valuable question has changed: not “How high is its score?” but “After I assign a task, how long before I need to come back and take over?” Measuring around this question, building toolchains, and designing processes—that is how to stand firm in the agent era.