Evaluating and Selecting Open-Source Large Language Models for AI Agents: A Practical Guide

1. From Chatting to Doing: The Shift in Evaluation Paradigms

The focus of Large Language Model (LLM) applications is shifting from single-turn Q&A to Agent tasks. Instead of merely generating text, models are now expected to understand objectives, decompose steps, call tools, execute commands, self-correct based on feedback, and deliver final results through multi-turn, long-cycle interactions. This shift directly challenges traditional model evaluation methods.

The limitations of static benchmark tests (fixed question banks with standard answers) are significantly amplified in Agent scenarios:

• Distribution Distortion: Questions in a benchmark differ greatly from real work tasks in terms of complexity, ambiguity, and multi-turn nature. A real programming task might span dozens of files, multiple rounds of command execution, and constantly changing user instructions—a complexity that benchmarks struggle to replicate.
• Vulnerability to Targeted Optimization: Models can overfit to benchmark questions. A high score does not necessarily translate to good performance in actual tasks.
• Lack of Process Signals: Agent tasks often fail at a specific intermediate step. Static evaluations only show the final outcome (right or wrong) without indicating which step broke, making it impossible to guide troubleshooting and improvement.
• Lack of User Dimension: Real users change requirements mid-way, add constraints, and express dissatisfaction. These dynamic interactions cannot be simulated by static question banks.

Consequently, the industry is moving toward dynamic evaluation approaches centered around real user sessions. Multiple candidate models are put into the same batch of real task flows, driven by real users over long-cycle sessions, and rankings are calculated based on battle results and user feedback. Taking a recent Agent evaluation leaderboard as an example, its ranking is built on tens of thousands of real, long-cycle Agent sessions, categorized by work, programming, and chat. This “combat-ready” evaluation provides significantly higher reference value for model selection than static benchmark scoring.

2. Five Core Metrics for Understanding Agent Evaluation

When looking at an Agent leaderboard, developers need to understand the meaning behind the metrics rather than just memorizing rankings.

1. Net Improvement

This metric measures a model’s net winning margin against opponents in the same batch of real task battles. It answers the question: “When putting two models into the same batch of real tasks, which one do users trust more with the tasks, and which one delivers better results?” It is more explanatory than a single-point score. By comparing the Net Improvement rate between sequential generations of the same model series, you can determine if an iteration brought actual gains rather than relying on the vendor’s claimed improvements.

2. Bash Reliability

One of the most common scenarios for Agent deployment is interacting with the Shell: deploying services, processing data, and managing environments. This metric measures the stability of a model’s generated and executed commands—whether commands pass on the first try, whether it can diagnose failures and retry, and whether it causes destructive side effects. This is often underestimated: a wrong delete command or an unlocked write operation causes far more damage than answering a trivia question incorrectly. When evaluating this metric, you should also consider the model’s self-restraint against dangerous commands and its awareness of idempotency.

3. Praise vs. Complaint

This is a direct signal of user satisfaction. It is not uncommon to have a model with perfect technical metrics but a terrible user experience—for example, being correct but overly verbose, or changing the direction correctly but asking for repeated confirmations. Introducing subjective user evaluations is a necessary supplement to objective metrics. In internal enterprise evaluations, this can be substituted with ticket satisfaction, session ratings, or user retention signals.

4. Steerability

This measures a model’s ability to adjust its execution direction based on user feedback. In long-cycle tasks, requirement changes are the norm—users often say, “Wrong direction, change it to this” after seeing intermediate results. A model with poor steerability will either stubbornly stick to its guns or completely start over, both of which waste compute power and time. During selection, you can verify this with a simple experiment: assign a task to the model, inject two or three directional modifications midway, and observe its compliance rate and execution efficiency.

5. Confirmed Success

This is the proportion of tasks explicitly confirmed as “completed” by the user, making it the metric closest to “actual delivery.” There is a systematic gap between this and “the model thinks it’s done”: models tend to declare success prematurely, while users find missing edge cases during acceptance. Engineering-wise, this reminds us to design explicit acceptance nodes for Agent systems rather than letting the model grade itself.

It must be emphasized that these five metrics are neither equivalent nor interchangeable. High reliability does not mean good steerability, and high satisfaction does not mean a high success rate. The rational approach is to weight and combine them according to your business task structure—for example, weighting reliability higher in Ops scenarios, and weighting satisfaction and steerability higher in creative and content scenarios.

3. Architectural Perspective: Why MoE and Long Context are Standard for Agent Models

Evaluation metrics depict the outcome, but architecture determines the upper limit and cost of that outcome. Currently, top open-weight models have almost unanimously adopted two technical paths: Mixture of Experts (MoE) architecture and ultra-long context.

1. MoE: Decoupling “Capability” and “Cost” via Sparse Activation

The core idea of MoE is that the total parameter count of a model can reach the trillion level to expand knowledge capacity, but each token is routed to only a small fraction of experts (tens to hundreds of billions of parameters), keeping the computational overhead of a single inference within an acceptable range. A simplified routing illustration is as follows:

# MoE layer illustration: only Top-k experts are activated per token
def moe_forward(x, router, experts, k=2):
    logits = router(x)                 # [n_tokens, n_experts]
    weights, ids = top_k(logits, k)    # Select k experts for each token
    y = zeros_like(x)
    for eid, expert in enumerate(experts):
        mask = (ids == eid)
        if not mask.any():
            continue                   # Unselected experts do not participate in computation
        y[mask] += expert(x[mask]) * weights[mask]
    return y

Its engineering significance can be summarized in three points: First, the total parameter count determines the knowledge capacity and complex task modeling ability, serving as the capability ceiling. Second, the activated parameter count determines the computation per token; hundred-billion-level activation makes trillion-level models serviceable within a reasonable hardware budget. Third, all expert parameters must reside in VRAM or be efficiently scheduled. Memory planning during deployment must be calculated based on total parameters—this is the most common pitfall for beginners.

2. Million-Token Context: The Agent’s “Working Memory”

Agent tasks are inherently long-cycle: a real programming task might involve dozens of files, multiple rounds of command output, and continuously appended user instructions, easily exceeding the traditional 32K/128K window. Expanding the context to the million-token level means the entire codebase, batches of documents, and complete session history can be loaded directly into the context. This reduces the engineering complexity of external retrieval and prevents the need to aggressively compress history between tool calls, lowering the probability of losing critical information. However, long context is not free: the VRAM footprint of the KV cache and the attention computation grow with length, requiring framework-level optimizations (such as KV cache quantization, sparse attention, and paged management). Moreover, “fitting it in” does not equal “seeing it clearly”; the information dilution problem in the middle of the context still exists. Critical information should still be placed in effective positions through good prompt structure and context management strategies.

3. Native Multimodality: Agents Need to “See” the World

Error screenshots, design drafts, and data charts—these non-text inputs are ubiquitous in real tasks. Native multimodal capabilities allow the model to process multimodal inputs within the same weights, avoiding the error cascade and latency stacking brought by external OCR or vision models. This is a direct bonus for typical Agent scenarios like programming, data analysis, financial research, and content creation.

4. Enterprise Implementation: Building Your Own Evaluation and Selection System

Leaderboards provide an industry coordinate system, but “being on the list” does not mean “suitable for your business.” It is recommended to build an internal evaluation system following these steps.

1. Build an Internal Evaluation Set. Sample from real business sessions (ensure anonymization), covering high-frequency, high-value, and high-risk tasks. Each task should retain the complete multi-turn context and the user’s mid-way instruction changes, rather than just the final problem description. The scale does not need to be large; hundreds of high-quality, reproducible tasks often provide more differentiation than tens of thousands of rough samples.

2. Translate Leaderboard Metrics into Business Metrics. Confirmed Success corresponds to business receipts or acceptance ticket pass rates; Bash Reliability corresponds to the end-to-end success rate and rollback rate of automated task chains; Steerability corresponds to the “compliance rate of mid-way instruction changes”; Satisfaction corresponds to internal user ratings or ticket closure reason distributions. A schematic evaluation loop can be organized as follows:

for task in eval_set:                  # Internal evaluation set
    traj = agent.run(task.prompt)      # Complete multi-turn trajectory, including tool calls
    for revision in task.midway_edits: # Inject mid-way instruction changes
        traj = agent.revise(traj, revision)
    result = judge(traj, task.rubric)  # Evaluate item by item according to the acceptance checklist
    metrics.update(task.category, result)

3. Make Key Decisions for Selection and Deployment. Between open-weight and hosted APIs, scenarios involving sensitive data, requiring deep customization, or needing private deployment are suitable for open-weight models. Scenarios pursuing rapid go-live and elastic scaling are suitable for hosted services. The two can also be combined—sensitive tasks go through private models, while general tasks go through hosted services. Privately deploying open-weight models typically requires a GPU compute cluster, a container platform for scheduling and orchestration, object storage for model weights and evaluation data, along with logging and monitoring/alerting to ensure online service quality. Taking public cloud providers like Baidu AI Cloud as an example, these capabilities have corresponding standard product categories, allowing enterprises to procure progressively based on business scale and avoid heavy upfront asset investments. Finally, the security baseline: when an Agent has the permission to execute Shell commands, the risk is real. Ensure you configure least privilege, command blacklists/whitelists, sandbox isolation, and operation auditing for the execution environment, upgrading “Bash Reliability” from a model metric to a systems engineering metric.

5. Conclusion

The essence of Agent evaluation is breaking down the vague question of “is the model good” into a set of measurable engineering problems: “can it deliver stably in real tasks,” “can it understand adjustments,” and “are users satisfied.” The continuous progress of open-weight models in real-session evaluations validates the ecological vitality of this technical path. For enterprises, the more valuable action is not chasing leaderboard rankings, but establishing your own evaluation coordinate system as soon as possible—ensuring that every model upgrade is evidence-based and every cost investment corresponds to verifiable business returns.