A Changing Evaluation Paradigm: From Static Benchmarks to Real Tasks

For a long time, public perception of LLM capability was shaped by static benchmarks: standardized question banks, multiple-choice tests, math proofs, and unit tests used to compare models by leaderboard scores. These evaluations are reproducible and cheap, but their weaknesses are just as obvious—fixed questions, fragmented scenarios, and short cycles make them easy to game through targeted optimization, and they say little about how a model performs in real production settings, where work is multi-step, long-horizon, and deeply interactive with external environments.

The rise of agent tasks has changed that picture. An agent task requires a model to do more than answer questions: it must plan steps autonomously, call tools, read and write files, execute commands, and retrieve information, carrying a multi-turn interaction through to a complete goal—fixing a bug, producing a data analysis report, or standing up a small service.

As a result, evaluation is shifting in three directions:

1. From curated question banks to real user sessions. Evaluation platforms now aggregate tasks submitted by real users worldwide and rank models through anonymous head-to-head battles and win-rate statistics, covering high-frequency work scenarios such as coding, data analysis, system operations, and document processing.
2. From single-turn to long-horizon. A single session may involve dozens of tool calls and feedback-driven corrections, much closer to a real production pipeline.
3. From model self-assessment to user confirmation. Whether a task is actually complete is increasingly decided by explicit user feedback rather than the model’s own judgment.

Sample size underpins the credibility of this kind of evaluation. Recently, an evaluation platform published rankings based on tens of thousands of real, long-horizon agent sessions from global users, and introduced a “net improvement rate” to measure how a model version’s win rate changes across iterations in real sessions. Compared with static benchmarks, sample sizes at the ten-thousand level shrink confidence intervals to a usable range and turn “is the new version actually better?” into a quantifiable question.

Core Metrics: What Agent Evaluation Actually Measures

Producing comparable rankings on real sessions requires a structured metric system. Drawing on mainstream industry practice, four categories of metrics form the backbone of agent evaluation.

1. Task confirmation success rate

This is the hardest metric: the share of tasks that users explicitly confirm as complete. It differs fundamentally from “the model believes it is done.” One of the most common agent failure modes is pretending to be done—code that never actually ran, files that were never written, yet a completion report sent to the user. Anchoring the metric to user confirmation forces models to develop self-verification: proactively running tests, checking outputs, inspecting side effects, and declaring completion with evidence rather than guesses.

2. Shell command execution reliability

If task confirmation measures the outcome, command reliability measures process quality. Agents interact with real environments primarily by executing shell commands, and the reliability of this path caps what tasks they can complete. Common failure modes include syntax or argument errors, wrong assumptions about the runtime environment, missing permission handling, accidentally destructive operations, and failure to recover after an error. Improvements here usually signal systematic progress in environment awareness, error recovery, and risk avoidance—not simply having memorized more commands.

3. Steerability

An agent is not a one-shot black box; it is a multi-turn collaboration partner. Steerability measures whether, given corrective feedback, the model can adjust course precisely while preserving prior progress. Poor steerability shows up as two extremes: stubbornness—spinning on the same wrong path—or oscillation—one piece of feedback wipes out all previous work. High steerability requires the model to understand the boundary of the feedback: what to change, what to keep, and why.

4. Satisfaction: praise vs. complaints

The metrics above measure “doing it right”; satisfaction measures “how it feels to use.” By running sentiment classification on post-session user feedback and tracking the relative share of praise and complaints, this dimension captures the more subjective experience: whether communication is clear, whether the process is transparent, and whether results exceed expectations. It often correlates imperfectly with technical metrics, yet it directly shapes a model’s long-term reputation.

The pseudocode below illustrates how session-level metrics can be computed:

# Illustrative: computation logic for session-level agent evaluation metrics (pseudocode)
def evaluate_session(session):
    # 1. Task confirmation success rate: based on explicit user confirmation, not model self-assessment
    confirmed = session.user_explicitly_confirmed_done
    # 2. Shell command reliability: share of environment-operation commands that pass on the first attempt
    total = session.count_tool_calls(kind="shell")
    ok = session.count_tool_calls(kind="shell", status="success")
    shell_reliability = ok / max(total, 1)
    # 3. Steerability: whether the model returns to the right track within a limited number of turns after user correction
    steerability = session.steer_resolved_within(max_turns=3)
    # 4. Satisfaction: sentiment classification of session feedback, tracking the praise-to-complaint ratio
    satisfaction = sentiment_ratio(session.final_feedback)
    return {
        "confirmed_success": confirmed,
        "shell_reliability": shell_reliability,
        "steerability": steerability,
        "satisfaction": satisfaction,
    }

These metrics are only meaningful when compared on the same leaderboard and the same task distribution; an absolute number for a single model carries limited information on its own. The valuable approach combines vertical comparison across versions (net improvement rate) with horizontal comparison among contemporary models (relative ranking). Only together do they reveal whether a capability gain is real progress or evaluation noise.

The Architectural Foundation: Three Technical Pillars

Evaluation metrics reflect external behavior; the architecture behind that behavior matters just as much to developers. Today’s flagship open-weight models show three clear technical trends.

1. Mixture of Experts: decoupling capacity from cost

MoE has become the dominant design for very large models. The core idea: replace the feed-forward network with multiple “expert” subnetworks, and let a router dynamically select a small subset of experts per token. Total parameter count then determines the model’s knowledge capacity, while the parameters activated per inference determine compute cost—the two are decoupled.

Public figures suggest flagship open-weight models now follow a typical configuration: total parameters at the trillion scale, roughly hundred-billion-scale parameters activated per inference, and an activation ratio usually below ten percent. The direct benefit: knowledge capacity on the scale of a very large model, with inference cost close to that of a mid-sized dense model.

For agent workloads, this architecture is especially valuable: agent sessions are far longer than ordinary chat, with frequent tool calls and concurrent requests, so inference cost and latency directly determine product viability. MoE also brings new engineering challenges—experts must stay resident in GPU memory, driving up memory footprint; expert parallelism and load balancing are required; and cross-node communication overhead is significant. All of this raises the bar for deployment infrastructure.

2. Million-token context windows: the memory substrate for long-horizon tasks

Agent tasks consume context at a startling rate: system prompts, tool definitions, multi-turn interaction history, and the returned results of every tool call (which may be an entire log or source file) stack up and can easily exceed hundreds of thousands of tokens. Meanwhile, tasks such as “understand an entire code repository,” “analyze a multi-hundred-page research report,” or “conduct research on long-period data” inherently demand ultra-long context processing.

Million-token context windows exist precisely for these workloads. But a sober view is needed: supporting long context is not the same as using it well. Industry experience consistently shows that with very long inputs, models use information in middle positions less efficiently (the well-known “lost in the middle” problem), and attention compute grows with length. In production, ultra-long context is therefore typically combined with context compression, retrieval augmentation, and key-information anchoring rather than raw length for its own sake.

3. Native multimodality and agent-oriented reinforcement training

Early multimodal approaches mostly bolted external encoders onto a language model. Native approaches that introduce image-text and other multimodal signals during pretraining generally show more coherent cross-modal understanding. For agents, this means the model can directly read screenshots, charts, design mockups, and UI elements—enabling UI automation testing, chart interpretation, and creative design assistance.

The other key path is post-training reinforcement aimed at agent tasks: reinforcement learning that uses verifiable outcomes (such as unit tests passing or commands executing successfully) as reward signals, so the model learns tool use, error recovery, and long-horizon planning from real environment feedback. Arguably, agent capability improves half through architecture and half through this “learning in the environment” training paradigm.

The Open-Weight Ecosystem and a Disciplined Selection Process

1. What open weights mean for developers

Open-weight models publish their parameters for download and use. The value shows up on at least four levels:

• Data compliance and privacy: regulated industries such as finance, healthcare, and government can deploy in their own environments, keeping data in-domain;
• Deep customization: continued pretraining or fine-tuning on business data to accumulate domain-specific knowledge;
• Cost and supply control: at large inference scale, self-hosted clusters can offer better marginal cost, free from a single provider’s quota limits;
• Technical autonomy: avoids vendor lock-in and supports multi-cloud and hybrid deployment strategies.

One caveat: “open weights” does not mean “free, unrestricted use.” Before commercial deployment, verify the license’s granted scope and any additional terms.

2. A practical selection checklist

With so many open-weight models available, developers should build an evaluation framework around these dimensions:

• Prioritize real-task evaluations: look at agent leaderboards built on real user sessions, not just static benchmark scores;
• Read metric structure, not single scores: combine task confirmation rate, command reliability, steerability, and satisfaction to locate weaknesses;
• Check deployment cost: distinguish total from active parameters, and assess memory requirements, quantization support, and inference throughput;
• Read the license: confirm commercial-use terms and restrictions on derivative models;
• Build a private evaluation set: construct a regression set from your own real business tasks—no public leaderboard can replace this step.

3. A common path from weights to online service

The typical engineering path: obtain the model through an open-weight hosting platform; deploy inference on self-managed GPU clusters or cloud-hosted services; and establish solid online observability. On the cloud side, the common pattern is to manage weights and training data with object storage, orchestrate inference services with container platforms and elastic scaling, track latency, throughput, and error rates with logging and monitoring/alerting systems, and use canary releases to control the risk of new version rollouts. Major cloud platforms—including Baidu AI Cloud—provide these standard capabilities, and developers can combine them as needed rather than reinventing the wheel.

Building Production-Grade Agent Applications

Model selection is only the starting point. Turning agent capability into reliable productivity depends just as much on engineering.

1. Tool and environment design

• Right-size tool granularity: too coarse limits flexibility; too fine makes decision chains long and brittle;
• Write precise descriptions: parameters, return formats, and error codes should be documented clearly in the tool interface—the model’s understanding of a tool directly determines call quality;
• Make errors readable: return structured, comprehensible error messages so the model can “understand why it failed”—only then can it self-repair.

2. Sandbox isolation and risk control

# Illustrative: agent execution loop with checkpointing and rollback (pseudocode)
state = init_task_state()
checkpoint = snapshot(state)                # initial snapshot
while not state.done:
    action = llm.plan_next_action(state)    # the model decides based on the current state
    if is_risky(action):
        checkpoint = snapshot(state)        # snapshot before any high-risk operation
    result = sandbox.execute(action)        # execute in an isolated sandbox
    if result.failed and result.recoverable:
        state = restore(checkpoint)         # roll back if the failure is recoverable
        state.hint = result.error           # feed the error message back to the model
    else:
        state.observe(result)

Every environment operation should run in a sandbox with restricted resources and permissions, following the principle of least privilege. For high-risk actions—deletion, overwriting, bulk changes—introduce confirmation gates and snapshot/rollback mechanisms, so the cost of a model mistake stays within recoverable bounds.

3. Steerability and feedback loops

At the product level, keep an entry point for users to correct course at any time, and translate corrections into explicit state updates. For long tasks, set interim checkpoints where users confirm direction before continuing, so drift is caught early rather than discovered after running far off course.

4. Evaluation loops and continuous iteration

Build a unified online/offline evaluation system: sample real sessions online at a set ratio and monitor them continuously with a mix of human review and automated metrics (task confirmation rate, command success rate, steerability, satisfaction); maintain an offline regression set and rerun it whenever you change models, prompts, or tool definitions. Evaluation is not a one-time event—it is a continuous process that moves in step with product iteration.

5. Cost and latency optimization

Agent applications are cost-sensitive. Common optimizations include context trimming and history compression, prefix caching and reuse, streaming output to improve perceived latency, and a layered “small-model routing with large-model fallback” architecture—simple steps go to low-cost models, while complex decisions escalate to flagship models.

Conclusion

Agent evaluation is moving from the exam hall to the field: real users, real tasks, long-horizon sessions, and success judged by user confirmation. This paradigm brings “model capability” and “production value” closer than ever before. At the same time, open-weight models—powered by MoE architectures, ultra-long context, and native multimodality—are closing the gap with closed models in demanding agent scenarios. The competitive focus of the open ecosystem has shifted from “who chats more fluently” to “who gets the job done.”

For developers, rather than chasing any single leaderboard position, it pays to understand the logic behind the metrics: task confirmation rate teaches self-verification; command reliability teaches respect for the environment; steerability teaches respect for feedback; satisfaction teaches attention to experience. Internalizing these principles into your own agent engineering practice is the greatest dividend these leaderboards offer the industry.