Deploying Ultra-Large Open-Weight Models: Self-Hosted Inference and Managed Agent Practices

I. What Open Weights Mean: A New Phase in LLM Delivery

Over the past two years, the mainstream approach for enterprises to leverage Large Language Models (LLMs) has been through API calls. Models are hosted by providers, and enterprises pay per token for inference capabilities. While this offers quick integration and zero maintenance, its limitations are clear: data must leave the network boundary, deep customization of sampling and deployment strategies is restricted, and users are bound by the provider’s iteration pace.

Recently, a leading model team fully opened the weights of an ultra-large MoE model: 2.8 trillion total parameters, 104 billion activated parameters per inference, a 1-million-token context window, and native multimodal visual understanding. This announcement sparked widespread discussion in the global tech community. The fundamental changes brought by open weights include:

• Deployment Autonomy: Developers are no longer restricted to APIs; they can deploy and run models themselves when compute resources are available.
• Data Sovereignty: Local inference keeps data within the local domain, meeting the strict compliance requirements of finance, healthcare, and government sectors.
• Customizability: Secondary fine-tuning based on industry corpora enables the model to “understand the business.”
• Deep Controllability: Developers can intervene in underlying components like logits analysis, sampling strategies, and caching for deep optimization.

A notable technical detail is that this model is not a new base pre-trained from scratch, but rather an existing base iterated through large-scale cross-domain joint reinforcement learning (RL). The underlying industry trend: base model reuse + post-training (especially RL) is becoming the primary path for capability leaps. This indicates that future differentiated models will share the same base but feature different post-training. Enterprises should focus more on post-training quality rather than simply comparing parameter scales.

II. Reading the Model Resume: MoE, Activated Parameters, and Ultra-Long Context

Before deciding on self-hosting, it is crucial to understand the technical profile of such ultra-large MoE models.

1. MoE: Compute Depends on Activated Parameters, VRAM on Total Parameters

The Mixture of Experts (MoE) architecture splits traditional feedforward layers into multiple parallel expert networks, paired with a router that selects a few experts for each token’s computation. This creates a key asymmetry:

• Inference Compute Cost is mainly determined by activated parameters—104 billion activated parameters yield a compute volume similar to a 100-billion-scale dense model.
• VRAM Cost is dictated by total parameters—2.8 trillion weights must constantly reside in VRAM (or rely on high-speed interconnects/offloading).

In other words, a model with “small activation, large total” is compute-friendly but VRAM-unfriendly. This is the first hurdle for self-hosting: you must calculate the VRAM budget clearly.

2. Million-Token Context: Capability and Cost Are Two Sides of the Same Coin

The value of ultra-long context is intuitive: injecting an entire codebase, hundred-page contracts, or complete conversation and tool-call histories at once. But the cost lies in the KV cache. It can be estimated using the following general formula:

def kv_cache_gib(layers, kv_heads, head_dim, dtype_bytes, seq_len):
    """Estimate KV cache VRAM usage (illustrative, not specific to any model)"""
    bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes
    return bytes_per_token * seq_len / (1 << 30)
# Assume a hypothetical model: 64 layers, 8 KV heads, head_dim=128, KV stored in FP16
# kv_cache_gib(64, 8, 128, 2, 1_000_000) ≈ 256 GiB

The above calculation shows that for a million-token context, the KV cache alone can reach hundreds of GiB. Engineering implementations must include a series of optimizations:

• Paged KV Cache: Reduces VRAM fragmentation and improves concurrent throughput.
• Prefix Caching: Common prefixes like system prompts and tool descriptions are reused across multiple requests.
• KV Quantization: Compresses the KV cache from FP16 to FP8/INT8, halving VRAM usage or more.
• Sparse/Hierarchical Attention: Reduces the attention computation complexity for ultra-long sequences.

3. Native Multimodality: The Significance of Visual Understanding for Agents

Native visual understanding means agents can directly “see”: monitoring dashboards, financial report charts, scanned documents, and software interfaces. For scenarios like IT operations automation (reading monitoring panels) and investment research analysis (reading chart data), this is a critical capability that avoids the integration costs of external OCR/VLM pipelines. It is also the foundation for agents transitioning from “text chat” to “working with visuals.”

4. Cross-Domain Joint Reinforcement Learning

Joint RL involves optimizing capabilities across multiple domains—such as code, math, tool usage, and multi-turn conversations—within the same training flow, using unified reward signals to mitigate the “seesaw effect” between capabilities. The implication for users is that when evaluating these models, focus on their performance in cross-capability tasks (e.g., the “read chart -> query tool -> write code -> give conclusion” pipeline) rather than single-task benchmark scores.

III. Self-Hosted Deployment: Calculate the VRAM Budget, Choose the Right Framework

1. The VRAM Budget: Multi-Node Multi-GPU is the Baseline

A rough estimate for a 2.8-trillion-parameter model (magnitude illustration only):

• FP8/INT8 quantized weights: ~2.8TB
• INT4 aggressive quantization: ~1.4TB
• Plus KV cache, activation values, and framework overhead.

The conclusion is clear: even with aggressive quantization, such models require a multi-node, multi-GPU cluster relying on high-bandwidth interconnects. Single-node deployment is not realistic. Enterprises should factor cluster size, interconnect bandwidth, and operations personnel into the Total Cost of Ownership (TCO) during project planning.

2. Inference Framework and Hardware Ecosystem Adaptation

The three practical challenges of self-hosting are: the inference framework’s support for ultra-large MoEs (expert parallelism, communication optimization), its adaptation maturity to target hardware (especially alternative accelerators), and its optimization capabilities for ultra-long contexts. Concurrent with this weight release, several compute providers announced same-day compatibility, enabling deployment validation via mainstream open-source inference frameworks, mitigating VRAM pressure through KV caching and quantization—the ecosystem’s collaborative pace is visibly accelerating.

Checklist for selecting an inference framework:

• Support for hybrid strategies of Expert Parallelism (EP) and Tensor Parallelism (TP);
• Implementation of paged KV cache and prefix caching;
• Support for continuous batching to improve throughput;
• Mature toolchains for weight and KV quantization;
• Depth of adaptation for target hardware backends and community activity.

3. Deployment Models

• Self-Hosted Data Center: Data never leaves the domain; high CapEx; suitable for strict compliance and ultra-large stable workloads.
• Cloud GPU Container Instances: For example, leveraging general capabilities like GPU containers, object storage, and logging/monitoring provided by platforms such as Baidu AI Cloud to host self-built inference services and agent runtimes, balancing elasticity and compliance.
• Hybrid Model: Self-hosted inference combined with managed agent runtimes, decoupling both to choose the optimal delivery method.

IV. Managed Agents: Turning the Harness into a Cloud Service

Simultaneous with the open weights, the team launched a “managed agent” service—providing the Harness framework that supports agent operations as a cloud service for enterprises.

1. Why the Harness is the Hidden Bulk of Agent Implementation

The model only handles “thinking”; for an agent to “work,” it requires a complete runtime infrastructure: code execution sandboxes (secure isolation), context management (compression, retrieval, memory), tool calling protocols and authentication, access control, task interruption/resumption, and multi-agent orchestration and communication. Building this reliable harness from scratch takes mature teams months, with continuous long-tail maintenance costs post-launch—this is the core reason many agents remain stuck at the Demo stage.

2. Value Division in the Managed Model

Once managed, the division of labor between the enterprise and the platform becomes clear: enterprises only need to define tasks, configure agents, and connect business tools; the platform handles the maintenance and upgrades of the underlying runtime environment, providing 24/7 managed execution capabilities, supporting long-duration complex tasks and multi-agent collaboration. An illustrative agent configuration is as follows:

agent:
  name: ops-oncall-agent
  model:
    mode: hosted-inference        # Can also point to self-hosted inference endpoints
    max_context_tokens: 1000000
  runtime:
    execution: sandboxed
    timeout_minutes: 120
    retry: { policy: exponential, max_attempts: 3 }
    resume: checkpoint
  tools:
    - name: query_metrics
      auth: service_account
    - name: run_playbook
      scope: allowlist
  policies:
    max_tool_calls_per_task: 50
    data_egress: deny

3. Two Typical Scenarios and Engineering Essentials

• Internal Scenarios: Business monitoring, ops response—emphasizing long execution times, scheduled tasks, and alert integration; high requirements for permission boundaries and auditing.
• External Scenarios: Investment research assistants, intelligent customer service—emphasizing high concurrency, SLAs, and content compliance; high requirements for isolation and rate limiting.

For either scenario, long-running tasks must design checkpoints and idempotent retries; tool calls should have rate and budget limits to prevent runaway tasks; permissions should follow the principle of least privilege while retaining full-chain audit logs.

V. Open Weights ≠ Traditional Open Source: Licensing Compliance Essentials

It must be clarified that what was opened this time are model weights; the training code and original datasets were not released, and commercial use must follow the model provider’s custom license agreement. This fundamentally differs from the standard open-source definition. Legal and technical teams should jointly review:

• Authorization Scope: Whether commercial use is allowed and if revenue threshold clauses exist;
• Derivative Model Obligations: Whether fine-tuned models must adopt the equivalent license;
• Usage Restrictions: Some licenses impose constraints on specific industries or competitive use.

Despite this, the practical value of the open-weight model remains significant: local deployment, data localization, and industry-specific fine-tuning are all achievable. However, for an ultra-large MoE in the 2.8-trillion-parameter class, hardware investment is an unavoidable cost. Evaluations must consider the total cost rather than just “free download.”

VI. Enterprise Decision Framework: API, Self-Hosted, or Managed

It is recommended to establish a decision matrix across four dimensions: Data Sensitivity (strict compliance favors self-hosting), Usage Volume (stable high traffic yields better self-hosted TCO, fluctuating traffic favors APIs), Latency and Controllability Requirements, and Team Inference Engineering Capabilities. For cost, use an approximate model: API Cost ≈ token unit price × usage; Self-Hosted Cost ≈ (cluster depreciation/rent + electricity + ops personnel) ÷ effective utilization. Empirically, if utilization remains below 30% long-term, self-hosting struggles to break even.

Recommended phased path: Use APIs and managed agents for rapid PoC validation; transition core scenarios to self-hosted weights + a mature Harness during scale-up; implement hybrid scheduling at full scale—long-tail traffic via API, core and peak traffic via the proprietary cluster.

Implementation Checklist: Define data compliance boundaries; complete VRAM and throughput estimates; confirm license terms; select inference framework and run benchmarks; design failure recovery and permission models; establish monitoring and cost dashboards.

VII. Conclusion

Open weights lower the barrier to acquiring LLM capabilities but do not lower the engineering barrier. Ultra-large MoEs and million-token contexts shift the challenge from the algorithm layer to the system layer: VRAM, interconnects, inference frameworks, and agent runtimes all require hardcore engineering. The emergence of managed agent services abstracts the heaviest engineering parts into cloud services. For enterprises, true competitiveness lies not in “possessing the weights” but in the ability to integrate the model, runtime, and business data—defining business and compliance boundaries before deciding on the delivery model is the prudent approach to implementation.