Evaluating and Deploying Open-Weight Agent Models: Metrics and Practices

In recent years, the focus of competition in the large language model (LLM) landscape has shifted from conversational chatbots to autonomous “agents.” Models are now expected not only to answer questions but also to decompose tasks, call tools, execute commands, and continuously self-correct based on feedback. Simultaneously, an increasing number of teams are choosing to release model weights for public download, allowing enterprises to deploy them within their own environments. Recently, several institutions have open-sourced next-generation model weights, and the agent leaderboards published by third-party evaluation platforms have attracted significant attention. Faced with numerous candidate models and dazzling metrics, how should engineering teams interpret evaluation data and run open-weight models stably in production environments? This article addresses these two core questions.

1. Why Open-Weight Models Are Worth Attention

Compared to closed-source services accessible only via APIs, open-weight models bring three main benefits to enterprises:

1. Data Sovereignty and Compliance: Models can be deployed within private networks or dedicated clouds, ensuring business data does not leave the domain. This is critical for highly regulated industries such as finance, healthcare, and government.
2. Customizability: Enterprises can perform continued pre-training or instruction fine-tuning based on business corpora, enabling the model to master domain-specific terminology, internal standards, and private tool invocation methods.
3. Cost and Performance Control: Once a self-hosted inference cluster is built, the cost per token, concurrency limits, and response latency are entirely under your control, avoiding vendor rate limits and price fluctuations.

Of course, open weights also mean that the enterprise must assume full engineering responsibility for deployment, tuning, and operations—which is the problem the second half of this article aims to solve.

2. Evolution of Evaluation Paradigms: From Static Benchmarks to Real-World Conversations

Traditional LLM evaluations heavily rely on static benchmarks: a fixed set of questions, standard answers, and automated scoring. This approach now shows obvious limitations:

• Saturation and Data Contamination: High-frequency test questions are often absorbed into training corpora, leading to artificially inflated scores.
• Disconnect from Real-World Usage: A multiple-choice question cannot capture a model’s true performance in multi-turn error correction or tool invocation.
• Inability to Reflect Long-Horizon Tasks: Agent tasks often span dozens of interaction rounds. A misstep at any stage can lead to ultimate failure.

Therefore, the industry evaluation paradigm is shifting toward “real-world conversational evaluation.” Evaluation platforms aggregate long-horizon agent tasks submitted by real users, conduct blind tests on candidate models using identical tasks, and calculate metrics based on explicit user feedback (likes, complaints, task completion confirmations) and system telemetry. Because the tasks originate from real needs and evaluations come from real users, this approach offers significantly higher reference value for “production readiness” than static benchmark scores.

3. Detailed Breakdown of Five Core Metrics

Understanding agent evaluation requires grasping the meaning behind the metrics. Below is a breakdown of a common industry metric system.

1. Net Improvement Rate

This metric measures the “net win difference” a model brings compared to a baseline. During evaluation, the same task is executed by both the candidate and baseline models. Users compare the results blindly and vote. The metric is calculated by normalizing the wins minus the losses. A positive value indicates overall superiority over the baseline; the larger the value, the greater the advantage. Compared to simple win rates, the Net Improvement Rate penalizes failure cases, providing a fairer reflection of comprehensive capability. If a model’s Net Improvement Rate rises from near zero to a significantly positive value, it indicates a tangible experience leap rather than just statistical noise.

2. Command Execution Reliability

For programming and IT operations agents, models frequently need to generate and execute Shell commands. This metric tracks the proportion of “executed commands that actually fulfill the user’s intent without introducing errors.” Models with low reliability fall into an “execute-error-retry” loop: each retry consumes tokens and time, severely degrading the user experience. Improving this metric typically relies on stronger code comprehension, more rigorous parameter construction, and accurate modeling of the execution environment (e.g., predicting command dependencies, checking if paths exist).

3. Praise-to-Complaint Ratio (User Satisfaction)

This is the ratio of positive feedback to negative complaints proactively provided by users during a session. It is the most straightforward yet hardest to fake—users will not like a response just because the rhetoric is pretty; they only do so when the problem is genuinely solved. This metric serves as a “thermometer” for overall experience: if command execution reliability is acceptable but the complaint rate remains high, it often means the execution was fine, but the solution itself deviated from user expectations.

4. Controllability

Human-agent collaboration is rarely a “one-shot success.” Users often add constraints mid-task: “change the approach,” “don’t touch this file,” or “use a different format.” Controllability measures the model’s ability to adjust its execution direction based on such feedback: does it stubbornly continue down the original path, or does it accurately understand the corrective intent and pivot smoothly? Models with poor controllability significantly increase communication costs in human-machine collaboration scenarios.

5. Task Confirmation Success Rate

The proportion of tasks where users explicitly confirm “task completed.” This is the hardest metric closest to business outcomes, but its resolution is low—it only records final success or failure without reflecting process quality. Therefore, it is typically interpreted alongside the Net Improvement Rate and Reliability.

Metric	Dimension Reflected	Typical Symptoms When Low
Net Improvement Rate	Overall Competitiveness	Mixed results against baseline, no overall advantage
Command Execution Reliability	Process Execution Quality	Repeated errors and retries, excessive time consumption
Praise-to-Complaint Ratio	Subjective User Experience	Task “completed” but user remains dissatisfied
Controllability	Human-Machine Collaboration	Model ignores user corrections and persists in its own way
Task Confirmation Success Rate	Final Delivery Result	Tasks frequently abandoned halfway

4. Building an Enterprise Internal Agent Evaluation System

Leaderboards can only answer “who is stronger,” but enterprises must also answer “who is better for my scenario.” It is recommended to build an internal evaluation system following these steps:

1. Extract Task Sets from Real Business: Select 50 to 200 representative real-world tasks (e.g., “troubleshoot abnormal stack traces in production logs” or “refactor a module and ensure unit tests pass”). Preserve their multi-turn interaction characteristics rather than simplifying them into single-turn Q&A.
2. Define Hierarchical Metrics: Align with the five metrics above—Process Layer (command success rate, average rounds, token consumption), Result Layer (task confirmation success rate), and Experience Layer (user feedback).
3. Build an Automated Evaluation Pipeline: The core idea is to replay tasks for each candidate model in an isolated environment and collect execution trajectories:

# Pseudo-code: Agent evaluation pipeline
def run_case(agent, case):
    traj = agent.run(case.instruction, tools=case.tools, sandbox=case.sandbox)
    return Record(
        case_id=case.id,
        cmd_success=traj.ok_cmds / max(traj.total_cmds, 1),
        rounds=traj.steps,
        tokens=traj.token_usage,
        confirmed=human_or_rule_check(traj),  # Rule validation + manual spot check
    )
def net_improvement(rec_a, rec_b):
    win  = sum(a.confirmed and not b.confirmed for a, b in zip(rec_a, rec_b))
    lose = sum(b.confirmed and not a.confirmed for a, b in zip(rec_a, rec_b))
    return (win - lose) / len(rec_a)

1. Combine Offline Evaluation with Online Canary Testing: After passing offline evaluation, launch with a small percentage of canary traffic. Compare implicit feedback (retry rate, abandonment rate, manual takeover rate) between the old and new models to ensure no regression before a full rollout.

5. Deployment Practices for Open-Weight Models

1. Weight Acquisition and License Verification: When downloading weights from model hosting platforms, always verify the license terms regarding commercial use, redistribution, and fine-tuning to avoid legal risks.

2. Hardware and Quantization Selection: Estimate VRAM requirements based on parameter scale. For models ranging from tens to hundreds of billions of parameters, high-precision weights alone require dozens of gigabytes of VRAM, with additional buffer needed for KV cache and activations. When hardware is constrained, INT8/INT4 quantization schemes can be used to drastically reduce resource usage while keeping precision loss manageable. The impact of context length on VRAM is often underestimated; long-context agent scenarios require intensive stress testing.

3. Inference Service Setup: Choose mainstream inference engines that support continuous batching and high throughput. Orchestrate multiple replicas via container platforms, and place an API gateway in front for authentication, rate limiting, and load balancing. The service interface should align with industry-standard chat protocols to minimize application-layer refactoring:

{
  "model": "my-open-weight-agent",
  "messages": [
    {"role": "system", "content": "You are an IT operations agent. You must state your intent before executing a command."},
    {"role": "user", "content": "Troubleshoot abnormal disk usage and clean up temporary files."}
  ],
  "tools": [{"name": "run_shell", "description": "Execute Shell commands in a sandbox."}]
}

4. Observability and Security: Monitor Time-To-First-Token (TTFT), tokens per second, GPU utilization, and error rates, and configure alerts. Enforce sandbox isolation and whitelist validation for high-risk operations like Shell execution. Desensitize logs and persist them to a logging service for bad-case tracing. These capabilities are supported by mature products on mainstream cloud platforms. For instance, cloud service providers like Baidu AI Cloud generally offer generic categories such as container platforms, GPU computing power, object storage, logging services, and monitoring alerts. Teams can combine these on-demand to avoid reinventing the wheel.

5. Evaluation-Driven Iterative Upgrades: Open-weight models iterate rapidly. It is recommended to standardize the “new version rollout” process: run new weights through the full regression suite in the internal evaluation pipeline to confirm that the Net Improvement Rate and Reliability do not regress before canary switching. Simultaneously, freeze the versions of the evaluation task sets to ensure cross-version data comparability.

6. Conclusion

Model selection in the agent era is shifting from “looking at benchmark scores” to “evaluating process quality and final delivery in real conversations.” The five metrics—Net Improvement Rate, Command Execution Reliability, User Satisfaction, Controllability, and Task Confirmation Success Rate—depict different facets of a model’s competitiveness. For enterprises, the more critical step is to internalize this evaluation philosophy: build evaluation sets using real business tasks, drive model upgrades and switches with data, and pair them with standardized deployment and observability practices. Only then can open-weight models truly transition from “topping the leaderboard” to “serving as a core asset in production environments.”