Open-Source LLM Agent Evaluation: Real-World Metrics and the Architecture Behind the Gains

How Agent Evaluation Is Shifting from Benchmarks to Real-World Use

The way we measure LLM capabilities is undergoing a fundamental paradigm shift. Early static benchmarks—knowledge QA on fixed datasets, single-turn code generation, and the like—are easy to compare across models, but they suffer from three hard-to-avoid limitations:

• Data contamination: Public test questions may leak into training corpora, inflating scores while masking weak real-world capability;
• Single-turn static interaction: They fail to reflect how stable a model is across multi-turn tool calls, error recovery, and other complex chains;
• Disconnect from real workflows: A “perfect score” in a lab setting doesn’t necessarily mean the model is “useful” on real user tasks.

Recently, a major international model evaluation platform released an agent leaderboard based on real usage data: rankings are driven by more than 17,500 real, long-horizon agent task sessions from global users, rather than artificially constructed test questions. A recently released open-weights model topped the open-model field and ranked first in all three categories—work, coding, and chat—with its net improvement rate jumping from +0.9% to +5.1% over the previous generation. The ranking itself is noteworthy, but more valuable is the evaluation methodology and model technology roadmap behind it. This article walks through both.

Five Key Metrics: A Multidimensional Profile of Agent Capability

Real-world agent usage is highly heterogeneous: some users ask the model to write scripts, others to troubleshoot production incidents, others to compile research reports. A single score cannot capture this complexity, which is why mature agent evaluations adopt multi-metric systems. Based on the data disclosed in this evaluation, let’s examine what each metric means and the engineering takeaways.

1. Net Improvement Rate: How Much Better Is This Version than the Last?

Net improvement rate is defined as: on the same task distribution, the share of sessions where the new model improved, minus the share where it regressed. A simplified calculation:

def net_improvement_rate(sessions):
    """sessions: each record contains a score_delta between old and new model ratings"""
    improved  = sum(1 for s in sessions if s.score_delta > 0)
    regressed = sum(1 for s in sessions if s.score_delta < 0)
    return (improved - regressed) / len(sessions)

The jump from +0.9% to +5.1% shows intuitively that the new model is “better across most real tasks” rather than “dramatically better on a few questions.” The takeaway for users: when evaluating model iterations, focus on the breadth and average of improvements, not just peak capability.

2. Command Execution Reliability: The Foundation of an Agent’s Hands-On Ability

In coding and ops-oriented agent tasks, models need to generate and execute shell commands. Command execution reliability measures the proportion of model-generated commands that execute correctly and achieve the intended outcome. In this evaluation, the new model reached +7.9%, first among open models, while the previous generation sat at -4.5%, ranking 13th. The leap from negative to nearly +8% points to systematic improvements in environment awareness, parameter inference, and error recovery.

The warning for developers is clear: command execution is where agents most easily fail—one wrong rm or a badly assembled pipeline can cause irreversible damage. In production, command execution must be confined to an isolated sandbox, backed by allowlists and human confirmation mechanisms.

3. Praise-to-Complaint Ratio: A Direct Signal of User Satisfaction

This metric tracks the difference between positive and negative user feedback within sessions. The new model scored +7.7%, up sharply from +1.3% for the previous generation. Compared with machine scoring, spontaneous user praise and complaints are closer to real experience—but they are also affected by noise such as task difficulty and user expectations, so they work better as a long-term trend signal than as a single-point conclusion.

4. Steerability: Understanding Feedback and Adjusting Course

Steerability measures a model’s ability to adjust its execution direction based on mid-task user feedback. The new model hit +6.5%, first among open models. The essential difference between agents and chatbots is the length of the task chain: once the direction goes off track, every subsequent step amplifies the deviation. A model with strong steerability can turn corrective instructions like “no, I want a different approach” into effective path corrections, reducing the cost of starting over.

5. Confirmation Success Rate: The Final Test of Task Closure

This metric tracks the proportion of sessions where users explicitly confirm task completion. The new model scored +3.9% versus +3.8% for the previous generation. A 0.1 percentage-point gain looks tiny, but at a high base, every increment represents the polishing that takes a model from “almost done” to “truly deliverable.” It’s a reminder: the last mile of agent deployment often lies not in dazzling capability ceilings, but in delivery certainty.

Three Architectural Technologies Behind Agent Capability

1. Mixture-of-Experts: Balancing Trillion-Scale Total Parameters with Hundred-Billion-Scale Activation

According to the model’s public technical documentation, it adopts a Mixture-of-Experts (MoE) architecture with 2.8 trillion total parameters and roughly 104 billion activated per inference. The core idea of MoE is decoupling capacity from compute: total parameters determine the model’s knowledge capacity, while activated parameters determine the computational cost of each inference. A simplified forward pass:

# Simplified MoE forward pass (pseudocode)
gate = softmax(self.router(x), dim=-1)         # Score all experts
topk_val, topk_idx = gate.topk(k=2, dim=-1)    # Activate only the Top-2 experts
y = sum(v * self.experts[i](x)
        for v, i in zip(topk_val, topk_idx))

The engineering challenges of this route include expert load balancing, training stability, and inference parallelism strategies, but the payoff is clear: the model can cover long-tail knowledge and complex reasoning patterns while keeping inference overhead within an acceptable range.

2. Million-Token Context: The Memory Foundation for Long-Horizon Tasks

Agent tasks often span dozens of tool calls, with intermediate results, file contents, and error logs continuously consuming context. A context window of roughly 1 million tokens gives the model the ability to “remember” the entire task history, reducing rework caused by key information being truncated. That said, long context brings engineering pressure on attention computation and KV cache memory, typically requiring techniques such as sparse attention and KV cache management to balance quality and cost.

3. Native Multimodality: From “Understanding” to “Operating”

Native multimodality means modalities such as vision and text are modeled jointly during pretraining rather than stitched together afterwards. For agents, this directly expands the set of actionable tasks: understanding UI screenshots, analyzing charts, assisting with 3D creation, and more. Combined with this evaluation’s emphasis on coding, financial research, and factual accuracy, the evolution of model capability is clearly shifting from “being able to chat” to “being able to get things done.”

The Ecosystem Value of Open Weights

For enterprises and developers, open weights deliver at least four benefits: private deployment to meet data compliance and security requirements; fine-tuning and customization with domain data to build proprietary capabilities; auditability and reproducibility to support security assessment and result verification; and flexible deployment paths. Cloud platforms—Baidu AI Cloud among them—typically provide common capabilities such as GPU compute, model hosting, logging, and monitoring with alerting, so developers can productionize open-source models with elastic scaling without building their own data centers, and focus their energy on business logic itself.

Developer Practice: Building a Scientific Evaluation and Selection Process

A leaderboard is only a first filter. Real selection decisions should rest on your own evaluation system. Here is a lightweight evaluation skeleton:

import json
EVAL_TASKS = [json.loads(l) for l in open("tasks.jsonl")]
def run_agent(model, task, max_steps=30):
    """Run the agent in a sandbox; return the execution trajectory and final result"""
    traj = []
    for _ in range(max_steps):
        action = model.decide(task.goal, traj)
        if action.type == "finish":
            return traj, action.result
        obs = task.sandbox.execute(action)      # Execute in an isolated environment
        traj.append((action, obs))
    return traj, None
def evaluate(model, tasks):
    confirmed, ok, total = 0, 0, 0
    for t in tasks:
        traj, result = run_agent(model, t)
        confirmed += int(t.user_confirm(result))
        for a, o in traj:
            if a.type == "shell":
                total += 1
                ok += int(o.exit_code == 0)
    return {"confirmed_rate": confirmed / len(tasks),
            "cmd_reliability": ok / max(total, 1)}

Pair it with three principles: build the evaluation set from real tickets, covering task completion, command reliability, and steerability/correction; canary-test new models, comparing on small traffic before full rollout; and close the feedback loop, turning user correction records into new evaluation cases so the evaluation set grows together with the business.

Conclusion

Leaderboard rankings will shift, but evaluation methodology and architectural understanding are long-term assets. The core signal from this evaluation is that agent capability is being redefined by “real usage data,” with MoE, ultra-long context, and native multimodality forming the current technical mainline. For developers, rather than chasing a single leaderboard position, it is more valuable to understand the engineering meaning behind each metric and build your own evaluation, canary-release, and feedback loops around your own scenarios. That is the reliable path to turning LLM capability into real productivity.