Open-Weight Models and Hosted Agents: A Practical Guide to Enterprise AI Deployment

1. Open Weights: A Dividing Line for Private LLM Deployment

For the past few years, the dominant way enterprises used large models was pay-per-token API access: low barrier to entry, no infrastructure concerns, but model weights stayed with the provider. Enterprises found deep customization difficult, and data had to leave the local environment. As flagship-grade models have begun releasing full weights, this arrangement is being rewritten.

The direct implication of open full weights is that developers and enterprises are no longer limited to calling a model through an API. They can download the weights, deploy and run them in environments with adequate compute, and optimize and customize them to actual requirements. This yields three kinds of freedom:

• Deployment freedom: models can run on the enterprise intranet, in a private cloud, or on local clusters, with sensitive data never leaving the domain;
• Customization freedom: models can be quantized and compressed, domain fine-tuned, and post-trained with reinforcement learning to fit the business distribution;
• Evolution autonomy: enterprises are not directly affected by upstream API version changes, rate-limit policies, or billing adjustments.

2. Technical Profile of Flagship Open-Weight Models

Using a recent representative 3T-class open-weight model as a reference, understanding its common technical characteristics is a prerequisite for sound deployment planning.

1. MoE Architecture: Trading Sparse Activation for Inference Cost

Flagship models today generally adopt MoE architecture: the model internally consists of many “expert” subnetworks, and a routing mechanism activates only a small subset during each inference. A representative configuration is roughly 2.8 trillion total parameters with about 100 billion activated parameters per inference.

This “huge total parameters, controllable activated parameters” design essentially trades storage for compute: VRAM and disk footprint are calculated on total parameters, while the compute per forward pass is determined by activated parameters. For enterprises, MoE means achieving capability close to a larger dense model at a lower per-token inference cost.

2. Million-Token Context Window

A context window on the order of 1 million tokens allows the model to process an entire book, a large codebase, hundreds of pages of contracts, or days of conversation history in a single pass. This is especially critical for agent applications: intermediate states, tool returns, and historical decisions generated during long-horizon tasks all need to persist in context, and window length directly determines how far a task can run.

3. Native Multimodal Visual Understanding

The model introduces the visual modality at the base stage rather than attaching a vision module after the fact. Native multimodal capability lets the model directly understand document screenshots, charts, and interface images, markedly improving practicality in scenarios such as intelligent customer service, document review, and alert screenshot analysis.

4. Training Paradigm: Strong Base Plus Cross-Domain Joint Reinforcement Learning

Notably, such flagship models are usually not entirely new bases trained from scratch. Instead, they iterate capability on top of an existing strong base through large-scale cross-domain joint reinforcement learning, letting coding, math, general reasoning, and tool calling reinforce one another within a single training process rather than being optimized independently and stitched together. This also suggests the right approach to secondary fine-tuning: post-training should continue the joint-optimization philosophy to avoid the degradation trap of “gaining capability A while losing capability B.”

3. API Calls or Private Deployment: A Four-Dimension Evaluation Framework

Obtaining open weights is only the first step; enterprises still need to choose a deployment path rationally. Four dimensions are worth evaluating:

• Data compliance: Can data leave the domain? Do regulators require data to stay on the intranet end to end?
• Customization depth: Is prompt optimization sufficient, or are domain fine-tuning and post-training required?
• Cost structure: API pay-per-use scales linearly with usage; self-hosting is a one-time compute investment plus ongoing operations, and for high-frequency scenarios private deployment usually has a better total cost of ownership;
• Technical threshold: Self-hosting requires engineering skills in inference framework deployment, VRAM optimization, quantization, and serviceization.

A common conclusion: APIs are more economical for low-frequency, general-purpose scenarios; for high-frequency, highly sensitive scenarios requiring deep customization, private deployment of open weights offers stronger long-term advantages.

4. AI Agent Deployment: The Seriously Underestimated Harness Engineering

Model capability is only the agent’s “brain.” What actually keeps an agent working reliably is a runtime infrastructure commonly called the Harness. In the traditional self-built model, enterprises must solve at least the following problems:

1. Code execution environment: a sandboxed code interpreter that lets the model run generated code while preventing unauthorized operations;
2. Context management: context keeps expanding during long tasks, so compression, truncation, and external memory mounting strategies are needed;
3. Tool calling protocol: safely exposing internal system query and write interfaces to the model, with failure retry handling;
4. Permission control: an authorization system granular enough to define which agent can use which tool and read or write which resources;
5. Task interruption recovery: multi-hour tasks that fail mid-run can resume from a checkpoint rather than starting over;
6. Multi-agent collaboration: task orchestration, message passing, and result aggregation across multiple agents.

Industry experience shows that building a reliable in-house Harness infrastructure typically takes months of investment, followed by ongoing maintenance. Many teams get an agent prototype running within a week, only to fail at the “24/7 stable operation” stage—not because of the model, but because of distributed-systems engineering workload.

5. Hosted Agent Services: Turning the Harness into Something Ready to Use

Because the cost of self-building a Harness is so high, the “Hosted Agents” model has emerged: the provider delivers the Harness framework that supports agent operation as a cloud service, and enterprises obtain a 24/7 hosted runtime environment without building or maintaining the underlying environment, supporting long-running complex tasks and multi-agent collaboration.

Under this model the boundary of responsibilities is clear: the enterprise handles task definition, agent configuration, and business tool integration; the provider handles runtime scaling, monitoring and alerting, version upgrades, and failure recovery. The following illustrative configuration shows how the enterprise-side workload is reduced to “what task to do, what tools are available, and where the permission boundaries are”:

agent:
  name: ops-responder          # operations response agent
  model: open-weight-llm       # privately deployed open-weight model
  context:
    max_tokens: 1000000
    memory: external-kv-store  # external memory mount
  tools:
    - name: metrics_query      # metrics query (read-only)
      auth: read-only
    - name: ticket_create      # ticket creation (write)
      auth: write
  schedule: 24x7
  recovery:
    checkpoint: enabled        # resume from checkpoint
    max_retry: 3

In terms of deployment scenarios, hosted agents mainly cover two categories. The first is integration into internal enterprise systems, handling business monitoring, operations response, and data inspection, characterized by long task cycles and deep coupling with internal systems. The second is embedding into user-facing products, such as investment research assistants and intelligent customer service, characterized by volatile concurrency and high requirements for latency and stability. When selecting a provider, focus on long-task scheduling and elastic scaling capabilities.

6. Compute Ecosystem: The Last Mile of Open Weights

The value of open weights ultimately depends on being able to run them. An industry phenomenon worth noting has emerged: on the day a flagship model’s weights are released, multiple compute vendors announce “0Day” inference adaptation—leveraging the heterogeneous compute backends of mainstream inference frameworks, the model can be deployed and validated directly on multiple hardware types. Meanwhile, mainstream cloud platforms, including Baidu AI Cloud, generally provide supporting capabilities such as GPU compute, container platforms, object storage, and log monitoring, allowing enterprises to freely combine self-hosted data centers and elastic cloud deployment according to compliance requirements and cost structure.

This means two practical benefits: hardware choice, with no need to lock into a single compute track; and a mature deployment path, where the engineering chain from weight download to serviceized launch has been thoroughly validated.

7. Practical Checklist: From Evaluation to Launch

When planning open-weight model plus agent deployment, technical teams are advised to proceed as follows:

1. Data boundary assessment: identify which data must remain on the intranet and determine the deployment form accordingly;
2. Compute cost estimation: estimate weight-loading VRAM from “total parameters × precision bits,” add KV cache and batching headroom, and derive the minimum viable cluster specification;
3. Inference performance validation: run throughput, latency, and concurrency benchmarks on the target hardware before deciding on a quantization strategy;
4. Harness path decision: if the business requires 24/7 stable operation and lacks a dedicated platform team, the hosted model is usually preferable to self-hosting;
5. Least-privilege design: agent tool authorization should follow the least-privilege principle, with write operations subject to human review by default;
6. Observability build-out: record audit logs for every tool call and context change to support failure recovery and effectiveness attribution.

8. Conclusion

Open weights genuinely put large-model capability into the hands of enterprises, while hosted agent services package the most arduous runtime engineering into something ready to use. Together they compress the cycle from AI application prototype to production. For enterprises, the real question is no longer “whether to use large models” but “which deployment path and which Harness route to choose under the constraints of data boundaries, cost structure, and team capability.” Understanding the fundamentals of MoE architecture, long context, and Harness engineering will help technical teams make calmer, more sustainable decisions in this round of change.