From Open Weights to Hosted Agents: An Engineering Guide to Productionizing AI

1. Background: The Shift from “Conversation” to “Execution”

When discussing open-weight large language models in the past, the primary concerns were parameter count, context length, and benchmark scores. Today, the industry’s focus is visibly shifting from “can the model chat?” to “can the model independently accomplish tasks?” This shift brings two direct changes:

First, evaluation methods have changed. The discriminatory power of static Q&A benchmarks is declining. They are being replaced by dynamic evaluations based on real user sessions and long-horizon agent tasks. Models are now required to execute multi-step actions in real environments, such as terminals and file systems, with scores calculated based on execution results and user feedback.

Second, the service format has evolved. Having only model weights is no longer sufficient. An agent’s actual performance heavily depends on its runtime framework—task planning, tool invocation, sandbox isolation, state management, and exception recovery are all indispensable. Consequently, the “hosted agent” model, which turns runtime frameworks into cloud services, is emerging as a critical puzzle piece for bringing open-weight models into production.

Understanding these two changes is the first step for technical teams evaluating and deploying agent systems.

2. Agent Evaluation: Using Real Task Sessions to Answer “Can It Work?”

2.1 Why Static Benchmarks Fall Short

Traditional benchmarks mostly consist of single-turn, closed-ended questions, which present two problems. First, they are easily gamed; if training data overlaps with the evaluation set, scores become artificially inflated. Second, they are disconnected from real-world usage. Agent tasks often involve multi-step processes spanning dozens of steps, accompanied by user corrections and supplementary instructions mid-way, which static questions cannot cover.

Therefore, the community is gradually adopting an evaluation paradigm centered on real sessions. It involves collecting real tasks submitted by users, having participating models complete them in controlled environments, and calculating metrics based on session results and explicit user feedback. These evaluations typically involve tens of thousands of sessions, span long periods, and contain authentic noise, better reflecting production environment performance.

2.2 Five Key Metrics and Their Engineering Implications

1. Net Improvement Rate: Compares new and old versions within the same task pool, measuring iteration effectiveness using the net difference of “wins minus losses.” It answers whether the new version is genuinely stronger, avoiding “false progress” caused by varying task pool difficulties.
2. Command Execution Reliability: Tracks the success and failure rates of commands executed by the model in a terminal environment. This metric often uses positive/negative signs. A negative score indicates more failed commands than effective improvements, usually correlating with high syntax error rates, inability to recover from errors, and repeated execution of destructive commands. For Ops or R&D agents, this is a baseline “passing” metric.
3. Positive/Negative Feedback Ratio (User Satisfaction): Aggregates user praise and complaints after sessions, reflecting the overall experience, including result quality, execution efficiency, and communication style.
4. Controllability: Measures how quickly the model adjusts its execution path after a user corrects its direction mid-task. In engineering terms, this means if the user says “do not touch this file,” the model actually stops and changes course. Models with poor controllability repeatedly veer off course in multi-step tasks, forcing users to intervene frequently.
5. Confirmation Success Rate: The percentage of users explicitly confirming “task completed.” It is the metric closest to “actual delivery quality” because users are voting with real results.

These metrics collectively outline a conclusion: the competition in agent capabilities has shifted from “knowing the answer” to “getting things done, doing them reliably, and following instructions.”

2.3 Building an Internal Evaluation Loop

Enterprises do not need to copy public leaderboards when deploying agents, but they should establish an isomorphic internal evaluation system:

• Layer tasks by business needs: Code repair, alert response, data formatting, report generation, etc.
• Fully record execution trajectories: Tool calls, time consumed, failures, and retries for every step.
• Track metrics isomorphic to public evaluations: Tool call success rate, convergence steps after correction, manual confirmation rate.

A minimal viable trajectory record structure looks like this:

{
  "task_id": "case-0001",
  "category": "ops_response",
  "steps": 9,
  "tool_calls": { "total": 14, "failed": 1 },
  "steer_events": 1,
  "confirmed_success": true,
  "user_feedback": "praise"
}

With this data, the effects of model upgrades, prompt adjustments, and framework changes can be quantitatively compared, rather than relying on a feeling that things “seem better.”

3. Hosted Agents: Turning Runtime Frameworks into Cloud Infrastructure

3.1 Why Runtime Frameworks Should Be Hosted

In a production-grade agent system, the model itself is only one component. The complete chain includes at least: the orchestration loop (plan-act-observe), tool connectors, sandbox execution environments, task state and memory, security and access control, and logging and monitoring. When building this system from scratch, teams quickly encounter the following challenges:

• Environment Isolation: Commands generated by the model must be executed in a strongly isolated sandbox to prevent damaging the host system.
• Long-running Tasks: Tasks may run for hours or even span days. Process restarts or network jitters must not result in state loss.
• Concurrency and Scheduling: Resource allocation, queueing, and priority management for parallel multi-task execution.
• Breakpoint Recovery: The ability to resume from a checkpoint after a failure rather than starting from scratch.
• Audit and Compliance: Every step must be traceable and replayable.

These are precisely the engineering problems cloud platforms excel at. The concept of a hosted agent service is to precipitate the runtime framework into cloud infrastructure. Enterprises configure agents on demand and receive a 24/7 hosted runtime environment without building and maintaining the framework themselves, inherently gaining long-running task execution and multi-agent collaboration capabilities. For infrastructure selection, teams can either build entirely from scratch (container orchestration, sandboxes, monitoring) or leverage general cloud platform capabilities. Mainstream cloud platforms like Baidu AI Cloud generally provide container services, function compute, message queues, logging services, and monitoring, which can serve as the foundation for a hosted runtime.

3.2 Typical Layered Architecture

A hosted agent platform can typically be divided into five layers:

• Access Layer: Handles triggering and intake, including API calls, Webhook events, scheduled tasks, and message queue consumption.
• Orchestration Layer: The “brain shell” of the agent, responsible for task planning, tool routing, context and memory management, and retry/correction.
• Execution Layer: Executes Shell commands, code, and external tool calls within a strongly isolated sandbox (containers or function compute instances).
• State Layer: Task checkpoints, long-term memory, and intermediate artifact storage, typically backed by databases and object storage.
• Observability Layer: Logs, metrics, tracing, and session replay—crucial for troubleshooting and compliance.

An illustrative structure for a task configuration is as follows (fields are illustrative, not from any real product API):

agent:
  name: ops-insight-agent
  model:
    source: open-weight        # Open-weight model, can use private or hosted inference
  runtime:
    mode: hosted               # Hosted runtime
    max_steps: 64              # Max execution steps per task to prevent runaway
    checkpoint: enabled        # Long-running task checkpoints
  tools:
    - type: shell              # Command execution within sandbox
      sandbox: isolated
    - type: http               # Call internal ticket/approval APIs
      allowlist: internal-api-only
  collaboration:
    pattern: planner-executor-reviewer
  observability:
    tracing: on
    log_sink: log-service

3.3 Long-running Tasks and Multi-Agent Collaboration

The core mechanism for long-running tasks is checkpointing and recovery: the orchestration layer persists the task state (completed steps, intermediate artifacts, to-do items) at critical nodes. If a process or machine fails, it can resume from the latest checkpoint. Combined with guardrails like “max steps” and “budget limits,” this balances task length with execution control.

A common pattern for multi-agent collaboration is “Planner-Executor-Reviewer”:

# Pseudocode: Execution loop for three-role collaboration
plan = planner.run(goal="Identify the root cause of last night's alert and output a report")
for step in plan.steps:
    result = executor.run(step, tools=[shell, log_query, ticket_api])
    verdict = reviewer.run(step, result)
    if not verdict.passed:
        result = executor.run(step, feedback=verdict.comments)
save_checkpoint(task_id, state)   # Supports breakpoint recovery

The Planner breaks down the task, the Executor uses tools to execute it, and the Reviewer gates the results, forming an internal loop of “generate-verify-correct.” Compared to a single model handling all steps, this structure typically offers better stability on complex tasks and makes it easier to pinpoint issues.

4. Enterprise Deployment: Internal Efficiency and External Integration

4.1 Path 1: Integrating into Internal Systems

The core requirement for internal scenarios is “connecting to existing systems and closing existing processes.” Typical directions include:

• Business Monitoring and Alert Response: Agents subscribe to alert events, automatically query logs and metrics, narrow down the investigation scope, provide root cause hypotheses, and even draft response plans for human approval.
• Ops Inspection and Change Assistance: Periodic inspections generating reports, and automatic verification of operation checklists and rollback plans before changes.
• R&D Efficiency: Bug reproduction, initial patch drafting, and test case supplementation.

For integration, Webhooks, message queue consumption, and scheduled tasks are the three most common triggering methods, all of which are generic integration categories not tied to specific vendors.

4.2 Path 2: Embedding into External User-Facing Products

When agents directly serve end-users, the design focus shifts from “efficiency” to “experience and security”:

• Investment Research Assistant: Data retrieval, multi-source data formatting, and draft report generation, with users focusing on verifying conclusions.
• Customer Service Assistant: Multi-turn problem clarification, ticket routing, and after-sales progress tracking.

Both scenarios should adhere to three design red lines: least privilege (agents only receive credentials essential for the task), output review (content involving funds, commitments, or external statements must pass a rules engine or human review), and human fallback (seamless transition to a human when confidence is low).

4.3 Security and Compliance Checklist

• Strong sandbox isolation; outbound network traffic follows a whitelist.
• Centralized management of keys and credentials; prohibit writing them into prompts or logs.
• Full-chain audit logs and session replay to meet traceability requirements.
• Clear data boundaries: define what data can enter the model context and what must be desensitized or kept local.

5. Engineering Checklist: Moving Agents from Demo to Production

1. Evaluate First: Before going live, run an internal evaluation set through the “task layering - trajectory recording - metric tracking” loop.
2. Gradual Rollout: Start with shadow runs (log only, no execution), then low-traffic real execution, gradually scaling up.
3. Observability Trio: Structured logs, key metrics (success rate/steps/duration/cost), and full-chain tracing with session replay.
4. Cost Governance: Set step and token budgets per task; use checkpoints for long-running tasks to reduce redundant consumption.
5. Decouple Model and Framework: Open-weight models and hosted runtimes are composable layers. Use evaluation data to independently drive “model upgrades” and “framework optimization,” preventing a cascading effect when changes occur.

6. Conclusion

Open weights solve “capability availability”—allowing enterprises to deploy the strongest general intelligence within their own boundaries. Hosted agents solve “engineering viability”—turning cumbersome and error-prone runtime frameworks into on-demand cloud services. Together, they are pushing agents from “stunning demo videos” to “daily production environments.” For technical teams, the real task is establishing an evaluation system scaled to real tasks, managing uncertainty through layered architectures and guardrails, and ensuring every model upgrade is validated by data. Those who successfully close this engineering loop first will gain an early advantage in the second half of agent deployment.