Evaluating Open-Source AI Agent Models and Implementing Hosting Services

1. From Conversational Ability to Task Execution: Agents as the New Focus

Over the past two years, the focus of large language model (LLM) evaluation has shifted from single-turn Q&A to long-horizon task execution. Developers are no longer just concerned with whether a model can write a snippet of code or answer a factual question correctly; they care about its ability to continuously complete a closed loop in a real working environment: understanding requirements, decomposing steps, calling tools, verifying results, and delivering the final output. This is the fundamental difference between Agentic capabilities and traditional conversational abilities: the former aims to “get things done,” while the latter aims to “talk nicely.”

Meanwhile, open-weight agent models are rapidly closing the gap with closed-source models. According to public leaderboards from several third-party evaluation communities, top open-source models are approaching mainstream closed-source solutions in categories like office tasks, coding, and general conversation, with some dimensions even overtaking them. For enterprises, this brings two direct changes:

• Expanded options: Open weights allow for private deployment and fine-tuning customization, while managed services offer out-of-the-box usage. Both routes have their applicable scenarios.
• Higher selection threshold: There is often a gap between claimed model capabilities and real-world task performance. Therefore, “how to scientifically evaluate an agent model” has become a prerequisite before selection.

2. Core Dimensions of Agent Evaluation: Four Key Metrics

Unlike traditional static benchmarks, credible agent evaluation relies on a large volume of real, long-horizon user sessions: users come with real tasks, the model executes the full process, and users vote in pairwise comparisons to determine the winner. The sample size typically needs to reach tens of thousands of sessions to ensure statistical significance. Under this methodology, the following four metrics are most valuable.

1. Command Execution Reliability (Shell/Bash Reliability)

This metric measures the accuracy and robustness of a model when generating and executing commands in a terminal environment: whether commands run successfully on the first try, whether the model can self-diagnose and correct failures, and whether it accidentally mismanipulates files or pollutes the environment. It is usually presented as a “net score”—a positive value indicates more successes and positive feedback than failures and negative feedback, while a negative value indicates insufficient reliability, often meaning the model will perform unpredictable destructive operations.

Command execution is the foundation of coding and automation agent tasks. Data processing, build releases, and environment operations all rely on a command-line closed loop. If this metric does not pass, all other capabilities are irrelevant.

2. Steerability

Real-world tasks rarely go smoothly from start to finish. Users will add constraints mid-execution, correct the direction, or even overturn previous plans. Steerability measures the quality of a model’s response to such feedback: does it “understand and adapt quickly,” or is it stubborn and erratic?

For long-horizon tasks, steerability often determines the final delivery experience more than the quality of the first-round output. A model with a slightly inferior initial proposal but rapid course-correction is usually better in practice than one that “gives a beautiful one-shot solution but refuses to follow instructions.”

3. Confirmed Success

The most common failure mode for agents is not an explicit error, but “appearing to be done”—the model claims the task is finished, but files are missing, results are wrong, or the process is incomplete. The confirmed success rate requires task outputs to undergo objective verification: tests pass, files exist, and results are reproducible, rather than judging completion solely based on the model’s self-reporting.

This metric directly corresponds to delivery certainty in enterprise scenarios and is the core threshold for evaluating whether an agent can enter a production environment.

4. User Satisfaction (Praise vs. Complaint Ratio)

Under an anonymous voting mechanism, users can actively choose “praise” or “complaint” after completing a session. The difference between the two constitutes net satisfaction. It is the comprehensive signal closest to the real user experience: even if all hard metrics meet standards, if the interaction is stiff, responses are verbose, or the rhythm is sluggish, satisfaction will drop significantly.

Comprehensive Metric: Net Improvement Rate

By aggregating the wins and losses of all sessions, the net improvement rate of the model relative to the baseline can be obtained. When interpreting this, it is recommended to focus on three points:

1. Check the sign: Whether it generally outperforms the comparison baseline.
2. Check the dimension distribution: Which task categories it excels in and which execution phases it is weak in.
3. Check generational changes: The improvement margin of the new version versus the old version, which reflects the team’s iteration quality better than absolute scores.

3. Open Weights vs. Managed Services: The Trade-off Logic of a Dual-Track Model

The industry is currently forming a dual-track delivery model of “open weights + managed agent services”: model weights are openly available for download on open-source platforms, supporting private deployment and secondary fine-tuning; simultaneously, official hosted versions are provided for users to use immediately upon registration. The trade-offs between the two routes can be summarized in a decision checklist:

Consideration	Lean towards Open-Source Self-Hosted	Lean towards Managed Service
Data Compliance	Strict compliance requirements, data cannot leave the domain	Non-sensitive scenarios, rapid validation
Customization Level	Requires fine-tuning, deep integration with internal toolchains	General capabilities are sufficient
Team Capabilities	Possesses inference optimization and ops capacity	Wants zero-ops, focus on business
Cost Structure	Long-term high volume, self-hosting amortizes costs	Volatile usage, pay-as-you-go is more cost-effective

Note that these two routes are not mutually exclusive. A common mature combination in practice is: first use a managed service to quickly run through the business closed loop and validate value, then use open weights to self-host to meet compliance and cost demands. This “validate first, self-host later” path can significantly reduce the upfront risk of technical investment.

4. Reference Architecture for Enterprise-Grade Agent Services

Regardless of the chosen delivery model, a usable agent service needs to address five layers of problems:

1. Model Inference Layer: A high-performance inference engine hosts tens of billions of parameter models, requiring batch scheduling, KV cache management, and quantization acceleration.
2. Tool Execution Layer: Provides a sandbox environment for command execution and code running, implementing resource limits, network isolation, and operation auditing.
3. Context and Memory Layer: Task state persistence, cross-session memory, and file artifact management.
4. Orchestration Layer: Task decomposition, multi-step planning, failure retries, and human-in-the-loop confirmation nodes.
5. Observability Layer: Logging, monitoring alerts, and cost accounting.

On cloud platforms like Baidu AI Cloud, the above capabilities can be mapped to combinations of general cloud product categories: GPU computing power and container platforms host inference and sandbox workloads, object storage preserves task artifacts, message queues decouple asynchronous execution of long tasks, and log services with monitoring alerts ensure online observability. Teams do not need to build infrastructure from scratch and can focus their energy on the agent logic itself.

A typical task dispatch API can be abstracted as follows:

POST /v1/agent/tasks
{
  "objective": "Aggregate yesterday's sales data and generate a daily report",
  "tools": ["sql_query", "spreadsheet", "mailer"],
  "constraints": {
    "max_steps": 30,
    "timeout_seconds": 600,
    "require_confirmation": ["send_email"]
  }
}

The corresponding minimal authorization configuration on the sandbox side:

sandbox:
  cpu_limit: "2"
  memory_limit: "4Gi"
  network_policy: "deny-by-default"   # Deny all by default, allow via whitelist
  filesystem: "ephemeral"             # Read-only base image + temporary writable layer
  audit: "all-commands"              # Full command audit

5. Engineering Best Practices and Common Pitfalls

1. Evaluate before going live: Build an evaluation set consistent with your business task distribution, covering both normal paths and boundary scenarios. Execute regression evaluations for every model or prompt change to prevent “local optimization, global degradation.”
2. Sandbox first, least privilege: The command execution environment must be isolated from the production network; sensitive actions like data deletion, outbound sending, and financial operations must have manual confirmation nodes.
3. Design for idempotency and checkpoint resumption: Failures are inevitable in long-horizon tasks. Step-level idempotency prevents data pollution from repeated executions, and checkpoint resumption can significantly reduce retry costs.
4. Monitor “Confirmed Success” instead of just “Reply Rate”: After launch, continuously track the real task completion rate, human intervention rate, and token cost. These metrics reflect service quality better than sampled experiences.
5. Beware of leaderboard overfitting: Third-party leaderboards are suitable for initial candidate screening, but final selection should be based on your own evaluation set—the task distribution on a leaderboard is almost certainly different from the real distribution of your business.

6. Conclusion

Open weights have rapidly lowered the technical barrier to agent capabilities, while managed services have further lowered the barrier to usage. Together, they are pushing agents from “demos” to “production.” For development teams, a more important question than “which model scores higher” is: Is my evaluation methodology credible? Is my execution environment secure? Is my iteration path a closed loop? Answering these three questions well ensures that the dividends brought by each round of model progress can truly be transformed into business value.