Open-Source AI Agent Models: Evaluation Metrics and Cloud-Hosted Architecture Guide

The Evolution and Practice of Open-Source Large Language Models in the Agent Domain

In recent years, the focus of large language model (LLM) development has shifted from simple conversational Q&A to complex task execution. Agent models equipped with planning, tool-calling, and code execution capabilities are becoming the core engine of enterprise workflow automation. However, evaluating agent models is far more complex than evaluating traditional text generation, and enterprises face deployment challenges such as the difficulty of building the underlying runtime framework and controlling security boundaries. This article starts with an analysis of agent model evaluation dimensions and then explores the design and practice of cloud-hosted agent architecture.

1. Core Evaluation Dimensions and Performance Evolution of Agent Models

In the conversational model era, we typically focused on a model’s natural language understanding, knowledge coverage, and logical reasoning. In the agent era, however, the evaluation criteria have fundamentally changed. Authoritative agent leaderboards in the industry use real, long-horizon user session data to quantitatively assess a model’s overall capabilities. Evaluation looks not only at whether a task is completed, but also at whether the process is stable and meets user expectations.

Specifically, the following core metrics form the foundation for measuring agent model capabilities:

1.1 Overall Net Improvement Rate and Confirmation Success Rate

When executing long-horizon tasks, agents often go through multiple tool calls and state transitions. The overall net improvement rate measures how much positive improvement a model delivers in final outcomes over a complete task cycle, compared with a baseline or the previous generation of models. The confirmation success rate directly reflects the proportion of completed tasks confirmed as valid and up to standard by users or automated systems. A high confirmation success rate indicates strong reliability in complex instruction following and goal decomposition.

1.2 Shell Command Execution Reliability

This is a key dividing line between agents and ordinary conversational models. Agents need to generate and execute shell commands to operate file systems, provision environments, or run scripts. A model must not only generate syntactically correct commands, but also handle exceptions during execution, assess state, and perform self-repair. From early models that frequently got stuck in command execution loops to today’s advanced open-source models achieving significant positive gains on this metric, the progress reflects major improvements in code execution feedback loops.

1.3 Controllability and User Satisfaction

An agent executing a task is like an autonomous vehicle: users need the ability to “take over” or “change course” at any time. Controllability measures how flexibly a model adjusts its execution direction in response to mid-task user feedback, corrective instructions, or new constraints. A good agent should incorporate new instructions seamlessly rather than stubbornly sticking to its original plan. Meanwhile, user satisfaction—measured by the ratio of praise to complaints—directly reflects experience quality in real-world scenarios.

2. Architecture Design and Core Value of Cloud-Hosted Agent Services

Although open-source agent models continue to advance, putting them into production often requires substantial effort to build the harness (runtime framework) that supports agent execution, including context management, tool orchestration, concurrency control, and exception recovery. To lower this barrier, the industry has gradually embraced the “hosted agent” cloud service model.

Hosted agent services deliver the agent runtime framework as a cloud-native capability, allowing enterprises and developers to benefit from stable, 24/7 agent execution without building and maintaining the underlying runtime environment from scratch.

2.1 Elastic Task Execution and Multi-Agent Collaboration

In complex business scenarios, a single agent often struggles to complete full end-to-end tasks. Cloud-hosted frameworks natively support long-horizon task execution and multi-agent collaboration. Architecturally, this typically relies on a centralized orchestrator that decomposes complex tasks into subtasks, assigns them to agents with different skills, and synchronizes state and communication between agents through message queues or shared memory spaces. The cloud platform manages agent lifecycles, schedules compute resources, and guarantees isolation under high concurrency.

2.2 Scenario-Based Integration and On-Demand Configuration

Hosted services allow enterprises to configure agents on demand and offer flexible integration options. Internally, agents can be integrated into enterprise systems via APIs to handle business monitoring, automated operations response, and log analysis. Externally, they can be embedded into end-user-facing products as investment research assistants, intelligent customer service, and more. This configuration is typically done through declarative configuration files, which greatly lowers the development barrier.

3. Enterprise-Grade Security Boundaries and Compliance Auditing

Once an agent can execute shell commands, access the network, and manipulate data, security becomes the primary enterprise concern. An uncontrolled agent can cause data leaks, accidental system operations, or even production incidents. A mature hosted agent service must therefore embed security mechanisms at every layer of its architecture.

3.1 Runtime Isolation and Permission Control

To prevent unauthorized operations, hosted services provide an independent sandbox environment for each execution instance. Sandboxes enforce resource isolation at the operating system or container level, restricting an agent’s access to the host file system and network. Enterprises can set strict operational boundaries for agents in different business scenarios—for example, allowing access only to files in specific directories, or permitting requests only to whitelisted API domains. The system should also provide an emergency kill switch, allowing administrators to immediately terminate sessions when abnormal behavior is detected.

3.2 End-to-End Traceability and Resource Auditing

Enterprise compliance requires that data operations be traceable. Hosted services must support end-to-end trace queries of the execution process, recording every tool call, model input and output, and execution duration, and must support exporting audit logs. Combined with the cloud platform’s general-purpose logging services and monitoring and alerting capabilities, administrators can observe resource consumption, API call frequency, and cost spending for each agent project in real time, achieving visualized cost control.

3.3 Pre-Inference Interception and Data Masking

There is a risk of sensitive data leakage both before model inference and after tool results are returned. Advanced hosted architectures typically introduce an independent inference hook component. Acting as middleware, this component inspects inputs before they are submitted to the model, intercepting data that may contain user privacy or business secrets; it also filters tool return results, masking or blocking content according to rules defined by the enterprise. This design decouples security policies from model logic, making it easy for enterprises to integrate their own security services or implement custom compliance rules.

4. Toolchain Ecosystem and Developer Integration Practices

An agent’s capability boundary depends largely on the tool ecosystem it can call. To accelerate business adoption, hosted services typically come with a rich prebuilt tool library and support custom enterprise extensions.

4.1 Out-of-the-Box Skills and Plugin Library

Platforms typically provide hundreds of general-purpose skills and dozens of plugins, covering high-frequency task scenarios from web search and professional knowledge base queries to office document processing and automated content generation. These tools are optimized by the platform with standard input and output interfaces, and developers only need to declare them in the agent configuration to use them.

4.2 Custom Tool Integration and Development Framework

For enterprise-specific business systems, developers can easily integrate their own tools and data sources. Typically, the cloud platform provides a standard tool development specification, requiring developers to define the tool’s description, input parameter schema, and authentication method. The following is an example custom tool declaration configuration:

{
  "tool_name": "internal_inventory_checker",
  "description": "Query real-time inventory data from the internal ERP system",
  "endpoint": "https://internal-api.example.com/v1/inventory",
  "auth": {
    "type": "bearer_token",
    "token_secret_ref": "vault://secrets/erp_token"
  },
  "input_schema": {
    "type": "object",
    "properties": {
      "sku_id": { "type": "string", "description": "Product SKU ID" }
    },
    "required": ["sku_id"]
  },
  "response_parser": "custom_erp_json_parser"
}

4.3 Architecture Design Guides and Cost Estimation

To further improve the developer experience, the platform’s documentation site should provide not only API references but also complete agent architecture design guidance, configuration tutorials, and a runtime cost estimation model. Enterprises can even copy mature industry agent templates directly into their own project spaces and adapt them with minor modifications to connect internal business systems. Combined with general-purpose cloud services such as container platforms and object storage—for example, those provided by Baidu AI Cloud—enterprises can build modern agent workflows that balance elasticity, security, and cost efficiency.

5. Conclusion

The evolution from base large language models to agent models marks AI’s leap from “information provider” to “task executor.” Continued breakthroughs by open-source models in core metrics such as shell execution reliability and controllability provide a solid foundation for complex business automation. Meanwhile, the rise of cloud-hosted agent services—by abstracting away the complexity of underlying frameworks and providing robust sandbox security mechanisms and a rich tool ecosystem—greatly lowers the barrier to enterprise agent adoption. Looking ahead, as multi-agent collaboration architectures mature further, agents are poised to become deeply embedded in enterprises’ core production workflows, serving as a new generation of infrastructure driving digital transformation.