Evaluating Open-Source LLM Agents: Key Metrics and Selection Practices

1. From Conversation to Execution: Agents as the New Focus of LLM Competition

Large language model (LLM) technology is undergoing a profound paradigm shift: moving from “being able to chat” to “getting things done.” Traditional conversational evaluations focus on the quality of single-turn responses, whereas agent scenarios impose entirely different requirements on models. Agents must understand complex objectives, autonomously decompose tasks, invoke external tools (such as shell commands, file systems, and browsers), self-correct based on environmental feedback, and maintain stability over long-term interactions spanning dozens of turns. Evaluating these capabilities is significantly more difficult than assessing static knowledge Q&A benchmarks, because the ultimate success of a task depends on whether the entire execution chain forms a closed loop.

Recently, an international evaluation platform’s agent leaderboard revealed a set of data worth examining deeper: an open-weight model achieved a +5.1% net improvement rate across over 17,500 real-world, long-term agent task sessions. This marks a substantial increase from its previous generation’s +0.9%, ranking first among open-weight models in three major categories: work, programming, and chat. The model weights have also been made available for download on public model hosting communities. This data sends a clear signal: open-weight models are rapidly closing the gap with top-tier closed-source models in terms of agent capabilities, significantly improving the feasibility for enterprises to build their own agent applications based on open-source models.

For developers and enterprise technical decision-makers, a more valuable question than “who ranks first” is: What exactly are these leaderboard metrics measuring? Which capabilities are truly critical to your business? And how can you reproduce credible evaluation conclusions within your own business context? This article will explore these questions step by step.

2. Understanding the Core Metrics of Agent Evaluation

Unlike traditional static benchmarks, mainstream agent evaluations generally adopt a dynamic mechanism of “real tasks + anonymous pairwise comparison + user voting.” Users submit real tasks, two anonymous models execute them simultaneously, and users vote for the better performer. The results are then aggregated into a relative win rate and a net improvement rate. Understanding the following key metrics is the prerequisite for correctly interpreting leaderboards.

1. Net Improvement Rate: The Yardstick of Overall Capability. This metric measures the model’s win rate minus its loss rate relative to a strong baseline, directly reflecting the extent to which “real users think it is better.” The leap from +0.9% to +5.1% is not a test-cramming improvement, but a comprehensive result of全面提升 in task closure capability, interaction quality, and stability.

2. Command Execution Reliability: The Watershed for Engineering Usability. Agents rely heavily on executing operating system commands to perform file operations, environment configuration, and build deployments. This metric measures the proportion of commands that execute successfully on the first try. The most notable detail in this case is that the new version achieved +7.9% (first among open-weight models), whereas the previous generation was only -4.5%. This indicates a qualitative leap in its command generation capability, transitioning from “frequent errors and repeated retries” to “highly reliable, one-pass success.” The cost of command failure goes far beyond wasting time—failed commands pollute the environment state, triggering cascading task failures, which is one of the biggest risk sources for deploying agents in production environments.

3. Controllability: The Valve for Collaboration Costs. This metric measures whether the model can promptly adjust its execution direction after receiving mid-task feedback from the user. Agent tasks have long cycles, and users’ initial intent expressions are often incomplete. The “speed of correction response” directly determines interaction costs. The case model reached +6.5% in this metric, ranking first among open-weight models, meaning that when a user says “No, try a different approach,” the model can quickly get back on track instead of stubbornly pursuing an erroneous path to the end.

4. Praise-to-Complaint Ratio: A Proxy Signal for Experience Quality. This metric calculates the ratio of users’ active praise to complaints during a session (the case model scored +7.7%, while the previous generation was only +1.3%). It captures situations where “the task was completed, but the experience was poor,” serving as a complement to objective metrics.

5. Confirmed Completion Rate: The Hard Metric for Business Value. This metric tracks the ratio of users explicitly confirming that a task was truly completed (the case model scored +3.9%, ranking fourth among open-weight models). Good scores in the previous metrics do not guarantee successful task closure; it is normal for this rate to be slightly lower than satisfaction metrics—users sometimes praise a session based on a friendly attitude even if it did not fully solve the problem. This metric should be the top priority during model selection.

A practical interpretation mnemonic: The confirmed completion rate indicates the business baseline, command reliability and controllability indicate engineering usability, satisfaction indicates user experience, and the net improvement rate indicates the overall capability level.

3. Underlying Architectural Elements Supporting the Capability Leap

Leaderboard scores are the results; architectural designs are the causes. Based on the public technical materials of this open-source model, three design choices are worth the attention of anyone making selection decisions.

1. Mixture-of-Experts (MoE) Architecture: Rebalancing Capacity and Cost. The model has a total parameter count of approximately 2.8 trillion, but only activates about 104 billion parameters during a single inference. The essence of MoE lies in “exchanging sparse activation for large capacity.” Expert networks are routed on demand, endowing the model with a trillion-level knowledge capacity while keeping inference overhead close to that of a dense 100-billion-parameter model. The insight for enterprises is: when calculating inference costs, do not just look at the total parameter count, but comprehensively consider the activated parameter count and the actual token throughput.

2. Million-Level Context Window: The Memory Foundation for Long-Cycle Tasks. Agent task session lengths can easily reach hundreds of thousands of tokens—comprehensive codebase understanding, multi-turn tool invocation trajectories, and long document analysis are all standard occurrences. A context window of roughly 1 million tokens means the model can maintain a global perspective of the entire task without complex retrieval pipelines, significantly reducing the probability of “forgetting the initial objective” or “losing intermediate results.” However, note that window length does not equal long-text understanding quality. During selection, you should still use “needle in a haystack” style tasks to empirically test the recall rate of mid-section information.

3. Native Multimodality and Domain Depth. The model has been specifically enhanced in areas such as programming, 3D creation, financial research, and factual accuracy. Native multimodality allows the agent to directly process inputs like screenshots, charts, and scanned documents, eliminating the need for external recognition pipelines and significantly reducing costs for automated office and data research scenarios. The enhancement of factual accuracy directly lowers the risk of the agent taking “hallucinatory actions”—the cost of executing a command based on a hallucination is far higher than saying something incorrect.

4. How Enterprises Can Build Their Own Evaluation and Selection Process

Leaderboards should only be used for initial screening. For enterprise deployment, it is recommended to follow a four-step method: “initial leaderboard screening -> business evaluation set re-testing -> gradual rollout -> continuous monitoring.”

Step 1: Define a Business-Oriented Evaluation Set. Extract 50 to 200 representative tasks from real business operations, covering typical workflows and edge cases, and adjust weights based on business distribution. If your business primarily involves code refactoring, you should increase the proportion of multi-file refactoring tasks rather than general Q&A.

Step 2: Execute Engineered Evaluation. Below is a sample pseudocode demonstrating how to conduct pairwise comparisons and calculate the confirmed completion rate for candidate models:

# Sample pseudocode: Pairwise evaluation framework for agent tasks
def run_task(model_client, task):
    """Drives the candidate model to execute a single agent task, recording the full trajectory"""
    session = model_client.create_session(tools=["shell", "file", "browser"])
    traj = session.execute(task.prompt, max_steps=task.max_steps)
    return {
        "task_id": task.id,
        "steps": traj.step_count,
        "cmd_ok": traj.command_success_ratio,    # First-pass command success rate
        "final": session.environment_snapshot()   # Final state, used for objective verification
    }
def pairwise_eval(model_a, model_b, tasks):
    win, lose, confirmed = 0, 0, 0
    for task in tasks:
        ra, rb = run_task(model_a, task), run_task(model_b, task)
        ok_a, ok_b = task.assert_final(ra["final"]), task.assert_final(rb["final"])
        if ok_a and not ok_b:
            win += 1
        elif ok_b and not ok_a:
            lose += 1
        else:  # In case of a tie, introduce blind human evaluation to capture subjective signals like satisfaction and controllability
            vote = judge_blind(ra, rb, task)
            win += (vote == "a"); lose += (vote == "b")
        confirmed += int(ok_a) + int(ok_b)
    return {
        "net_improvement":  (win - lose) / len(tasks),   # Net improvement rate
        "confirmed_success": confirmed / (2 * len(tasks)) # Confirmed completion rate
    }

Here, the “final state assertion” corresponds to the leaderboard’s confirmed completion rate, a key design to prevent subjective bias; the “command success rate” corresponds to command execution reliability; and the blind evaluation phase corresponds to satisfaction and controllability signals.

Step 3: Calculate Real Costs. Estimate inference costs by combining activated parameter counts, concurrency scale, and context length. Focus on stress-testing the memory footprint and prefill overhead of MoE models in long-context scenarios. At the same time, be sure to evaluate the “cost of failed retries”—for models with poor command reliability, their nominal unit price advantage will often be completely wiped out by the number of retries.

Step 4: Gradual Rollout and Continuous Monitoring. Open-weight models are typically integrated into production via private deployment or cloud inference services. On mainstream cloud platforms that provide GPU compute, container platforms, and monitoring/alerting capabilities (such as Baidu AI Cloud), it is recommended to integrate the model service into a unified logging and metrics system. Focus on monitoring four key metrics: task completion rate, manual intervention rate, average interaction turns, and tool invocation failure rate, using them as regression gates for subsequent version upgrades.

5. Common Pitfalls and Best Practices

• Pitfall 1: Leaderboard Dogmatism. Leaderboards reflect the average experience of all users, but your business distribution may be highly skewed. A top-ranking model might completely underperform on specific vertical tasks.
• Pitfall 2: Focusing Only on Single-Turn Quality, Ignoring Long-Term Stability. Assuming a single-step reliability of 95%, the overall success rate in a 30-step task drops to only about 21% (0.95 to the power of 30). Managing long-tail errors is of extremely high value.
• Pitfall 3: Ignoring Controllability. A highly capable but “stubborn” agent will have a much higher accident rate in high-privilege scenarios (such as executing commands in production environments) than a slightly less capable model that corrects its course quickly.
• Pitfall 4: Neglecting Evaluation Set Leakage. Questions from static benchmarks may have long been included in training data. Anonymous pairwise evaluations based on real sessions have stronger anti-leakage capabilities, and enterprise evaluations should also try to use private data.

Best practices can be summarized in one sentence: Use real long-cycle tasks for evaluation, use final state assertions as the judge, and use gradual rollouts and monitoring as insurance.

6. Conclusion

Open-weight models setting new records on agent leaderboards signifies that “building agents with open-source models” is transitioning from the technical validation phase to the production-ready phase. For enterprises, rather than fixating on leaderboard rankings, it is more productive to adapt the leaderboard’s metric system into your own business context: command reliability determines the engineering baseline, controllability determines the collaboration experience, and the confirmed completion rate determines business value. Building private evaluation capabilities around these three points is the fundamental method for consistently making the right model selection decisions amidst rapid model iteration cycles.