Architecting Managed Agent Services and Evaluating Open-Weight Models

I. Background: From Conversational Models to Task-Oriented Agents

Recently, the industry’s focus on Large Language Model (LLM) capabilities has shifted significantly. Conversational benchmarks that merely evaluate whether a model “answers well” are giving way to agentic task evaluations that assess whether a model “gets the job done.” An AI Agent refers to a working mode where the model autonomously completes task planning, tool calling, command execution, and result iteration within a runtime environment to deliver end-to-end results. Scenarios like code repair, IT operations response, investment analysis, and customer service are typical agentic tasks.

This transition brings two practical challenges for technical teams:

1. Model Selection: Facing a large number of open-weight models, how do we scientifically evaluate their agentic capabilities? Static, quiz-style benchmarks suffer from data contamination and task distortion, making it difficult to reflect true production performance.
2. Implementation: An agent cannot run simply with a “model + prompt.” It relies on a complete runtime framework, including task scheduling, tool orchestration, state management, sandbox isolation, and exception retries. Building this system in-house requires significant engineering effort and long-term maintenance costs.

Consequently, two notable technical paths have emerged in the industry. The first involves models providing open weights combined with an evaluation system based on real task sessions, allowing enterprises to deploy privately and validate capabilities against their specific scenarios. The second is managed agent services, which deliver the runtime framework as a cloud service, enabling developers to consume agent runtime environments just like computing resources. Let’s explore both approaches below.

II. Agent Evaluation: Replacing Static Benchmarks with Real Sessions

(I) Why Traditional Benchmarks Fall Short

Traditional benchmarks mostly consist of one-shot Q&A or fixed script tasks, which have three main flaws. First, the questions may have leaked into the training data, inflating the scores. Second, the task formats differ significantly from real production environments—real tasks are often multi-turn, long-running, and require mid-stream user corrections. Third, they lack measurement of “process quality,” focusing only on the final answer. Therefore, a more informative evaluation method involves collecting large-scale, long-running task sessions submitted by real users (typically requiring tens of thousands of samples for statistical confidence), recording the model’s complete behavior during multi-turn interactions, and then statistically analyzing task quality changes across multiple dimensions.

(II) Key Evaluation Dimensions Explained

From an engineering perspective, a comprehensive agent evaluation system typically includes the following dimensions:

1. Net Improvement Rate: Measures the actual gain brought by model version iterations. It compares the net change in task completion quality before and after a session, filtering out interfering factors like “user completed it themselves.” A positive, higher value indicates the new model version contributes more to task success.
2. Command Execution Reliability: The stability of the agent executing commands in a terminal environment, including command accuracy and self-repair capability after errors. This is a foundational capability for programming and ops agents; if command execution is unreliable, upper-level task planning is useless.
3. User Satisfaction Signals: Quantifies the real experience by tracking the ratio of positive feedback to complaints within sessions. It bridges the “experience layer” gap that automated metrics cannot capture, such as overly verbose replies or taking excessive unauthorized actions.
4. Steerability: Measures the model’s ability to adjust its execution direction based on user feedback. In production, users frequently correct the model. Whether the model can “understand and correct properly” directly determines human-machine collaboration efficiency.
5. Confirmation Success Rate: The proportion of task completions confirmed explicitly by the user, serving as a hard metric closely aligned with business closure.

It is worth noting the tension among these dimensions: chasing a high confirmation success rate might push the model toward conservative strategies, while high steerability requires the model to be highly sensitive to feedback. Evaluations should be weighted according to business scenarios rather than relying on a single leaderboard ranking.

(III) How Enterprises Can Build an Internal Evaluation Loop

• Task Replay Set: Extract desensitized real tasks from production logs to form a regression test set. Automatically replay them after changing models or adjusting prompts.
• Blind A/B Testing: Have new and old models process similar tasks in parallel, evaluated blindly by reviewers to avoid brand or version bias.
• Process Metrics Instrumentation: Beyond final results, record process data like tool call counts, retry counts, and user correction frequency to pinpoint experience bottlenecks.

III. Managed Agent Services: Turning the Runtime into a Cloud Capability

(I) The Hidden Costs of Self-Built Runtimes

A production-grade agent runtime must at least handle task queues and scheduling, state persistence and breakpoint recovery for long-running tasks, concurrency and timeout control for tool calls, sandbox isolation and resource quotas, failure retries and idempotency, as well as logging and auditing. These tasks are unrelated to business logic yet consume significant engineering effort. The core value of a managed agent service is abstracting this layer into a standard cloud capability: the platform ensures runtime availability and elasticity, while developers only need to define the agent’s goals, tools, and permissions to get a 24/7 managed environment with native support for long-running tasks and multi-agent collaboration.

Dimension	Self-Built Runtime	Managed Agent Service
Initial Investment	Requires building scheduling, sandbox, toolchain	Out-of-the-box
Ops Burden	Self-responsible for availability and upgrades	Cloud-managed
Security Capabilities	Self-built sandbox and audit	Built-in sandbox, interception, and audit
Elastic Scaling	Self-managed scaling	On-demand configuration

(II) Two Typical Deployment Patterns

1. Internal Integration: Integrating agents into internal systems for business monitoring, IT ops response, and data inspection. This emphasizes stability and permission containment, usually triggered by events.
2. External Embedding: Embedding agents into external-facing products to provide investment research or customer service assistants. This emphasizes response latency, content security, and experience consistency.

(III) Reference Architecture

A typical managed agent service can be broken down into six layers:

Business System / End User
        │  API / SDK / Webhook
        ▼
┌──────────────────────────────────┐
│ Access Layer: Auth, rate limiting, session management       │
├──────────────────────────────────┤
│ Orchestration Layer: Task planning, sub-task decomposition, │
│                      multi-agent collaboration, retry       │
├──────────────────────────────────┤
│ Inference Layer: Model invocation + pre-inference safety    │
├──────────────────────────────────┤
│ Tool Layer: Built-in skills/plugins + custom tool access    │
├──────────────────────────────────┤
│ Sandbox Layer: Isolated execution env, network access ctrl  │
├──────────────────────────────────┤
│ Governance Layer: Audit trails, usage reports, cost track   │
└──────────────────────────────────┘

IV. Security and Compliance: The Lifeline of Agent Deployment

Agents possess real “execution capabilities,” so their security design must be stricter than conversational applications. Production-grade solutions typically include four lines of defense:

1. Independent Sandbox and Network Access Control: Each task execution occurs in an isolated sandbox. External network access uses an allowlist mechanism to define the agent’s activity boundaries at the system level.
2. Operation Boundaries and Session Circuit Breakers: Administrators can define the executable scope for agents. If anomalous behavior occurs, the session can be terminated immediately to prevent loss escalation.
3. End-to-End Audit Trails and Export: The execution process is fully recorded, supporting post-hoc querying and export to meet compliance audit requirements. Usage reports allow administrators to view resource consumption and costs per agent project, achieving cost observability.
4. Pre-Inference Security Checks: Before model inference, task inputs and tool outputs are inspected to intercept privacy and sensitive data based on custom enterprise rules. This capability is usually provided as an independent security component for easy integration with enterprise-built or third-party security services.

A simplified pre-inference interception logic example:

def pre_inference_guard(payload: dict, rules: list) -> dict:
    """Check task input and tool output before model inference"""
    text = payload.get("input", "") + str(payload.get("tool_output", ""))
    for rule in rules:
        if rule.matches(text):
            if rule.action == "BLOCK":
                audit_log(rule, payload, blocked=True)
                raise SensitiveDataBlocked(rule.id)   # Intercept and log
            elif rule.action == "MASK":
                payload = rule.mask(payload)          # Desensitize and allow
    audit_log(rules, payload, blocked=False)
    return payload

V. Tool Ecosystem and Integration Practices

1. Skills and Plugin System: Managed platforms typically offer built-in skills and plugins covering web search, professional data query, office document processing, and content generation. Developers can attach them as needed, avoiding reinventing the wheel.
2. Custom Tools and Data Source Integration: General skills cannot cover internal enterprise logic, so platforms must support custom tools and private data sources. Engineering-wise, it is recommended to adopt a unified tool description standard (name, parameters, return structure, permission declarations) paired with invocation auth and rate limiting.
3. Templatization and Cost Estimation: To lower the barrier to entry, mature platforms provide agent templates and documentation covering architectural design concepts, configuration tutorials, and runtime cost estimates. Developers can clone templates into their project spaces and incrementally replace them with internal business systems and private databases.

A typical agent configuration example:

agent:
  name: ops-insight-agent        # Ops Insight Agent
  runtime:
    hosting: managed             # Managed runtime
    sandbox: isolated            # Isolated sandbox
    long_run: true               # Supports long-running tasks
  permissions:
    network:
      mode: allowlist            # Network access allowlist
      allow: ["Internal Monitoring API", "Knowledge Base Index Service"]
    tools: ["log_query", "alert_notify"]
  security:
    pre_inference_hook: on       # Pre-inference security check
    sensitive_rules: ["Phone Number", "ID Number", "Key Credentials"]
  audit:
    trace: enabled               # End-to-end audit trail
    export: enabled              # Audit export
    usage_report: daily          # Usage report

VI. Implementation Guidelines

1. Evaluate First: Before integrating any model, run an evaluation loop using an internal replay set. Let real task data speak rather than chasing public leaderboard rankings.
2. Start with Low-Risk Scenarios: Prioritize read-only tasks (monitoring, inspection, search summarization). Gradually open up write operations after verifying stability.
3. Shift Security Left: Permission allowlists, sensitive data interception, and audit trails should be completed before launch, not patched afterward.
4. Cost Observability: Establish usage reports and cost allocation from day one to prevent cost overruns as agents scale.

It should be noted that managed services are not the only option. If enterprises require stronger control over data and pipelines, they can also build their own runtimes based on the general capabilities of cloud providers. For instance, they can use container platforms to deploy isolated sandboxes, logging services for audit trails, and monitoring/alerting to ensure long-running task stability. Cloud providers like Baidu AI Cloud offer such general product capabilities. Choosing between the two routes is essentially a trade-off between “build cost” and “granularity of control.”

Conclusion: Agent technology is transitioning from “demo-ready” to “production-reliable.” For developers, understanding the logic behind evaluation systems and mastering the architectural layering and security design of managed runtimes provides more long-term value than fixating on a single model release or leaderboard ranking.