1. Industry Background: Two Trends Are Converging

Over the past year, two changes in the large model industry have drawn developer attention.

First, weights for highly capable models are being opened at an accelerating pace. More and more teams are choosing to release flagship models as open weights, allowing developers to freely download, deploy locally, fine-tune, and prune them—truly keeping model capabilities in their own hands. For industries sensitive to data boundaries, such as finance, healthcare, and government, open weights mean that both “capability” and “data” can remain within the enterprise intranet.

Second, the evaluation focus is shifting from “chatting” to “doing.” Traditional benchmarks emphasize knowledge Q&A and single-turn dialogue, whereas models in real business scenarios need to execute tasks continuously over long-horizon sessions: reading documents, running commands, calling APIs, and adjusting direction based on feedback. Recent public agent evaluation leaderboards show that, based on statistics from over 17,500 real, long-horizon agent task sessions from global users, the net improvement rate of a new generation of open-weight models (a composite measure of positive task quality improvements minus negative regressions) rose from about +0.9% in the previous generation to +5.1%, ranking among the top open models across the three main categories of work, coding, and chat. This indicates that the gap between open-weight models and closed-source models in “actually getting work done” is narrowing rapidly.

The implication for enterprises is straightforward: when selecting a model, don’t just look at how a chat demo feels—look at hard metrics on the agent dimension.

2. Understanding Agent Evaluation: What the Four Key Metrics Actually Measure

Using mainstream agent evaluation dimensions as an example, the following four metrics best reflect a model’s “battle readiness.”

1. Command execution reliability

This measures the accuracy with which a model generates and executes command-line operations. A great deal of an agent’s work happens in the terminal environment: installing dependencies, modifying configurations, running scripts. Public evaluation data shows that the new generation of open-weight models jumped from -4.5% in the previous generation to +7.9% on this metric, ranking first among open models. Behind the swing into positive territory is an overall improvement in the model’s understanding of environment state, avoidance of dangerous commands, and ability to self-repair after failed output. For scenarios such as operations automation and CI/CD assistance, this is a veto-level factor determining “whether it can be used at all.”

2. Praise-to-complaint ratio

This statistic captures the ratio of positive to negative feedback given by real users after a session ends—essentially a net user satisfaction score for the agent version. This metric rose from +1.3% in the previous generation to +7.7%, indicating that the frequency of the model “drifting off, forgetting, or fabricating” during long sessions has dropped significantly.

3. Controllability

This measures the model’s ability to adjust its execution direction based on mid-task user feedback. Real tasks are rarely done right the first time; users often need to say “try a different approach,” “don’t touch this file,” or “only change this function.” A highly controllable model can accurately understand corrective instructions and pivot smoothly, rather than starting over from scratch. The new generation of open-weight models reached +6.5% on this metric, also ranking among the top open models. For complex tasks requiring human-machine collaboration, such as code refactoring and data analysis, controllability directly determines collaboration efficiency.

4. Confirmation success rate

This measures the proportion of tasks actually completed after user confirmation. The new generation’s +3.9% may look only slightly higher than the previous generation’s +3.8%, but combined with the substantial improvements in the first three metrics, it shows that the model’s “process quality” in completing tasks has improved markedly, while end-to-end success rate is still constrained by external factors such as task difficulty and toolchain completeness. This also reminds us: evaluations should be read in combination—any single metric can be misleading.

3. Managed Agents: Turning the Agent Runtime into a Cloud Service

Model capability is only one part of an agent system. An agent that can run in production also needs a complete runtime framework: task orchestration, context management, tool invocation, sandbox isolation, failure retries, concurrent scheduling, and log collection. Building this framework yourself is a considerable engineering effort, requiring teams to handle container lifecycle, resource reclamation, exception recovery, and many other details unrelated to the business itself.

For this reason, “managed agents” are emerging as a new category that cloud vendors are competing over: moving the entire execution framework that supports agent operation to the cloud and delivering it as a service. Its typical capability set includes:

• 24/7 managed environment: agents reside in the cloud and do not depend on a developer’s local process;
• Long-horizon task execution: support for tasks spanning hours or days, with the ability to pause and resume mid-task;
• Multi-agent collaboration: multiple agents divide work and collaborate, each holding independent context and tool sets;
• On-demand configuration: enterprises can customize each agent’s goals, tool list, and permission boundaries.

This type of service follows the same philosophy as function computing and container services—removing “environment management” from the user side so that developers only need to care about the agent’s goals, tools, and rules themselves.

4. Two Typical Deployment Scenarios

Based on industry practice, enterprise use of managed agents generally falls into two categories:

1. Internal integration

Agents are connected to internal systems to handle repetitive work such as business monitoring, operations response, and data inspection. Take nighttime on-call as an example: an agent listens for alert events, automatically pulls logs, performs preliminary diagnosis, and generates remediation suggestions, with humans only doing final confirmation. Such scenarios have the highest requirements for execution reliability and audit traceability.

2. External embedding

Agents are embedded into customer-facing products to provide capabilities such as investment research assistants and customer service assistants. The difficulty in these scenarios lies in narrowing permissions and ensuring content safety: the agent can only access authorized data, and output must pass through sensitive information filtering.

Regardless of the scenario, the architecture can be abstracted into four layers:

Model layer (inference capability) → Orchestration layer (task planning and multi-agent collaboration) → Tool layer (built-in skills + enterprise-owned tools) → Governance layer (sandbox, permissions, audit, cost)

5. Security and Governance: Four Lines of Defense for Enterprise Agents

The essential difference between an agent and a chatbot is that it “has hands”—it can actually execute operations. Therefore, governance design must come first.

1. Sandbox isolation and network control. Each agent should run in an independent sandbox, with mutually isolated file systems and process spaces; outbound traffic should go through a whitelist to prevent induced access to external addresses. Enterprises can explicitly define an agent’s operational boundaries, for example allowing reads and writes only to specified directories or calls only to specified domains.

2. Session termination and permission convergence. Administrators can forcibly terminate sessions and freeze suspicious agents when necessary. Permissions should follow the principle of least privilege: read-only by default, with write operations authorized item by item.

3. Full-chain traceability and auditing. The execution process should support trace queries, so that every tool call and every input and output can be traced back; audit records should support export to satisfy compliance checks. Usage reports should also be provided so administrators can view the resource consumption and costs of each agent project, avoiding a runaway loop quietly burning through the budget.

4. Pre-inference security hooks. This is an easily overlooked but critically important part: before model inference, and before tool results enter the context, insert enterprise-defined inspection logic to block private and sensitive data according to rules. An illustrative pseudo-code example:

def pre_inference_hook(payload: dict) -> dict:
    """Run enterprise-defined checks before the request enters the model"""
    text = payload["input_text"]
    # 1. Block directly if a sensitive rule is matched
    for pattern in ENTERPRISE_BLOCKLIST:
        if re.search(pattern, text):
            return {"action": "block", "reason": "sensitive_content"}
    # 2. Mask named entities before allowing through
    masked = mask_pii(text)   # phone numbers, ID numbers, internal IPs, etc.
    return {"action": "pass", "input_text": masked}

6. Tool Ecosystem: Skills, Plugins, and Integration with Your Own Systems

The capability ceiling of an agent depends heavily on its tool ecosystem. Mature managed platforms typically provide hundreds of preset skills and dozens of plugins, covering high-frequency tasks such as web search, professional data queries, office document processing, and content generation; at the same time, they must offer custom channels so enterprises can register internal APIs, databases, and middleware as tools callable by the agent. A typical tool declaration looks roughly like this:

tools:
  - ref: builtin/web-search        # platform built-in skill
  - ref: builtin/doc-parser        # document parsing
  - ref: custom/internal-cmdb      # enterprise-owned configuration management API
    schema:
      method: GET
      endpoint: ${ENV:CMDB_URL}    # injected via environment variable to avoid hardcoding
      auth: oauth2

When evaluating a platform, it is recommended to verify two things in particular: first, whether parameter validation for tool calls is strict (can it prevent the model from fabricating parameters); second, the integration cost of your own tools (does it require modifying existing APIs).

7. Cost Estimation and Selection Advice

The cost structure of an agent project differs from that of a traditional application: token consumption, sandbox duration, and the number of tool calls all grow non-linearly with task complexity. Practical action recommendations:

1. Start small, then scale: copy the agent templates provided by the platform into your own project space, connect internal business systems, and run small-traffic validation;
2. Establish a cost baseline: refer to the architecture design and cost estimation methods in the official documentation, set token and duration budgets for each task type, and automatically degrade when limits are exceeded;
3. Focus on controllability metrics: include agent metrics such as controllability and execution reliability in POC scoring, rather than looking only at general benchmarks;
4. Governance first: before writing the first line of business code, define the sandbox policy, interception rules, and audit export plan.

Major cloud platforms (such as Baidu AI Cloud) are also integrating model services, knowledge bases, agent orchestration, and managed runtimes into one-stop solutions, allowing enterprises to complete the migration from prototype validation to production deployment within a unified technology stack and avoid the governance fragmentation caused by stitching together multiple platforms.

8. Conclusion

The rapid evolution of open-weight models, combined with the operational relief brought by managed agents, is turning agent development from “a custom engineering effort for a few teams” into “a standard practice most teams can afford.” The real dividing line is no longer who can call a model, but who can do three things solidly: security, auditing, and cost. For developers, the best path to start right now is: understand the evaluation metrics, choose a good managed runtime, and put governance rules first—then all that’s left is to let the agent run in real business scenarios.