Deploying Open-Weight LLMs: Self-Hosted Inference and Agent Hosting in Practice

1. The Shift of Open Weights: From “Renting Capabilities” to “Holding Assets”

For a long time, the mainstream approach for enterprises to utilize large language models has been via API calls: pay-per-token, linear scaling, and low entry barriers, but with model control remaining in the hands of the service provider. Releasing complete model weights offers an alternative—developers and enterprises can directly download weights, deploy them in environments with sufficient compute capacity, and optimize or customize them according to actual needs.

The core differences between the two models include:

• Data Compliance: In API mode, sensitive data must leave the local domain; private deployment keeps data entirely within the local domain, meeting the stringent compliance requirements of finance, government, and healthcare.
• Customization Depth: APIs only allow adjustments to prompts and sampling parameters; holding the weights enables fine-tuning, quantization, and even structural modifications to inference strategies.
• Cost Structure: APIs charge linearly based on call volume; private deployment requires a one-time compute investment with decreasing marginal costs, making it more economical for high-frequency invocation scenarios.
• Version Control: APIs are uniformly upgraded by the provider; self-deployment allows version locking, ensuring stable and reproducible online behavior.

Recently, open-weight models featuring a total parameter scale of 2.8 trillion, approximately 100 billion activated parameters per inference, a context window of one million tokens, and native multimodal visual understanding have emerged. Their release quickly sparked intense discussions within the global tech community. This signifies that the open-weight route has crossed the “usable” threshold and entered an “ultra-large scale + high controllability” phase, turning “how to effectively utilize open-weight models” into an engineering problem enterprises must answer.

2. Understanding the Technical Profile: What a Trillion-Level MoE Means

1. MoE Architecture: Decoupling Capacity from Cost

The core idea of MoE (Mixture of Experts) is “on-demand activation”: the model consists of numerous expert networks, and the router selects only a few experts most relevant to the current token to participate in computation during each inference. This creates two key metrics—total parameters determine knowledge capacity, activated parameters determine single-inference compute cost. Taking the “2.8 trillion total parameters, ~100 billion activated parameters” scale as an example: the model can store vast amounts of knowledge, while the actual compute consumption per inference is equivalent only to a 100-billion-level dense model. Capacity and cost are decoupled at the engineering level.

The direct impact on the deployment side is: VRAM must be large enough to hold all experts, and compute must be sufficient to run the activated parts. VRAM usage is calculated based on total parameters, while compute consumption is based on activated parameters. This asymmetric characteristic is the premise for all compute planning.

2. Million-Token Context: From “Conversation” to “Task”

The significance of a million-token window goes far beyond “reading longer documents”; it enables three categories of applications:

• Repository-level understanding: loading an entire project at once for cross-file defect analysis and refactoring suggestions.
• Long-chain tasks: agents retaining complete task trajectories and intermediate artifacts during hours-long executions without “amnesia.”
• Multi-document cross-referencing: joint reading and comparison of batched materials like contracts, financial reports, and research papers.

The trade-off is that KV Cache memory grows linearly with context length, constituting one of the greatest sources of VRAM pressure on the deployment side.

3. Native Multimodality and Capability Iteration Path

These models generally support visual understanding natively, directly receiving mixed image-text inputs without external vision modules. For capability building, the industry’s common practice is to iterate capabilities through large-scale, cross-domain joint reinforcement learning on top of existing large base models, rather than training a new base from scratch each time. This path also holds reference value for enterprises: prioritize domain post-training and alignment on mature base models rather than blindly developing proprietary ones from scratch.

3. Four Engineering Challenges of Self-Deployment

1. VRAM and Compute Estimation

Before deployment, these questions must be answered: How much VRAM do the weights occupy? How much additional VRAM is needed during inference? The following sample code can be used for rough estimation:

# Rough estimation (unit: GB). Actual deployment requires additional buffer for activations and framework overhead
def weights_mem_gb(total_params_b, bytes_per_param=2):
    # 1 billion params x 2 bytes (FP16) ≈ 2GB
    return total_params_b * bytes_per_param
def kv_cache_mem_gb(num_layers, kv_heads, head_dim, seq_len, batch=1, bytes_per_elem=2):
    # The factor 2 corresponds to two caches: K and V
    elems = 2 * num_layers * kv_heads * head_dim * seq_len * batch
    return elems * bytes_per_elem / 1e9

Two reminders: First, MoE models must keep all experts resident in VRAM. Even estimating with FP16, the weight VRAM for 2.8 trillion parameters is astronomical. Actual deployment will almost certainly rely on multi-node distributed schemes like tensor parallelism and expert parallelism across multiple GPUs. Second, under a million-token context, the KV Cache can reach tens of gigabytes; planning must never look at weights alone.

2. Inference Framework and Compute Adaptation

Mainstream open-source inference frameworks generally support MoE routing, continuous batching, and prefill/decode disaggregation, serving as the de facto standard carriers for self-deployment. A positive signal from the ecosystem is that multiple compute vendors can complete inference adaptation concurrently with model weight releases (often termed “0Day adaptation” in the industry). Relying on mainstream inference frameworks, models can be directly deployed and verified on various AI accelerators. The combination of open weights and compatible hardware forms a deployment closed loop for enterprises with localization requirements.

3. VRAM Optimization for Long Contexts

To address the VRAM pressure brought by ultra-long contexts and massive MoE models, common industry optimization combinations include:

• Quantization: Reducing weights and activations from FP16 to lower precision formats like INT8/INT4/FP8. Weight VRAM can be compressed to 1/2 or even 1/4 of the original.
• KV Cache Management: Paged memory management to reduce fragmentation, prefix caching to reuse common prefixes like system prompts, and KV quantization to further compress cache size.
• Parallelism Strategies: Tensor parallelism to split single-layer computations, expert parallelism to distribute different experts across different cards, and pipeline parallelism to split layers.

4. Servitization and Operations

Getting the model running is only the first step. Production environments also require SLO monitoring for throughput and latency, version canary releases and rollbacks, retry and degradation for failed requests, and comprehensive logging with distributed tracing. The engineering effort “beyond the model” is often more persistent than the deployment itself.

4. The Hidden Threshold of Agents: The Runtime Framework (Harness)

The complete form of an AI Agent = Model + Tools + Runtime. The industry refers to the runtime layer supporting stable agent operations as the Harness. Typical responsibilities include:

• Task loop scheduling: driving the continuous cycle of “perception → reasoning → decision → execution.”
• Context management: context compression, memory sedimentation, and pruning for ultra-long tasks.
• Tool invocation: tool registration, parameter validation, timeout control, and failure retries.
• Permission control: tool-level authorization and operation auditing under the principle of least privilege.
• State persistence and interrupt recovery: checkpoint mechanisms ensuring long tasks can resume after interruptions.
• Multi-agent collaboration: task decomposition, result aggregation, and conflict arbitration.
• Observability: end-to-end logging, metrics, and alerting.

Its core loop can be expressed in pseudocode:

while not task.is_done():
    obs      = env.observe()              # Perceive environment and task state
    decision = llm.reason(obs, memory)    # Model reasoning and decision
    action   = normalize(decision)        # Parse and validate action
    result   = tools.invoke(action, acl)  # Controlled tool invocation
    memory.append(obs, action, result)    # Update memory and checkpoints

This loop seems simple, but the difficulty lies in the details: How to recover from a checkpoint if the process crashes mid-task? How to converge tool permissions to the minimal set? How to compress context almost losslessly when it exceeds limits? Building a reliable Harness infrastructure in-house typically requires months of engineering effort, plus continuous maintenance costs post-launch—this is precisely why many agent projects stall at the demo stage.

5. Managed Agent Services: Turning Runtime into Cloud Capabilities

Encapsulating the Harness as a cloud-managed service is an emerging solution. Enterprises no longer need to build and maintain the underlying runtime framework themselves. It requires only three steps: define the task, configure the agent, and connect business tools. The maintenance and upgrading of the runtime environment are handled by the service provider, offering 24/7 managed execution capabilities that support long-duration complex tasks and multi-agent collaboration.

The boundary of responsibilities is roughly as follows:

Responsibility Dimension	Enterprise Side	Platform Side
Task & Goal	Define task goals and boundaries	—
Agent Configuration	Select model, configure role and toolset	Provide configuration interface
Business Tools	Implement and connect internal systems	Invocation, auth, retry hosting
Runtime Environment	—	24/7 hosting, elastic scaling, upgrade & maintenance
Fault Handling	Define business-level degradation strategies	Process guarding, checkpoint recovery
Observability	Consume logs and metrics	Logging, monitoring & alerting, tracing

Two typical implementation scenarios:

1. Integration into internal enterprise systems: Handling business monitoring, ops response, and alert duty, such as round-the-clock metric anomaly monitoring and automatic execution of contingency plans.
2. Embedding into user-facing products: Investment research assistants, smart customer service, etc., where managed services ensure concurrency and stability, allowing teams to focus on the business experience itself.

The core value lies in significantly shortening the agent’s cycle from “prototype validation to actual deployment.” When evaluating any managed solution, it is recommended to verify five key points: whether data leaves the domain and private network access capabilities; whether the tool integration protocol is open and compatible with existing auth systems; whether interrupt recovery and idempotency mechanisms are verified; whether the permission model supports least privilege and auditing; and the upper limit of multi-agent orchestration capabilities.

6. Enterprise Implementation Path Recommendations

1. Requirement Tiering: Tier data sensitivity, customization depth, and invocation volume to determine a combined strategy of “primarily private deployment, supplemented by managed services.”
2. Model Evaluation: Build an evaluation set using proprietary business data to assess general capabilities, domain capabilities, and inference costs, avoiding relying solely on public benchmark scores.
3. Infrastructure Planning: Estimate VRAM and compute using the formulas above to determine parallelism strategies. For the cloud foundation, use public or private clouds as needed. For example, leverage general product capabilities like container platforms, GPU compute scheduling, object storage, logging, and monitoring provided by cloud vendors like Baidu AI Cloud to build the foundational environment for inference services and agent execution.
4. Agent Transformation: Start with single-tool assistant scenarios, validate the value, and then expand to multi-tool and multi-agent orchestration.
5. Governance and Ops: Establish mechanisms for least-privilege permissions, operation auditing, monitoring, alerting, and version rollbacks to integrate agents into existing R&D management systems.

7. Conclusion

Open weights transform large models from “rented services” to “held assets,” while managed agents shift the infrastructure needed to “keep models working continuously” from self-built to ready-to-use. The combination of the two is significantly lowering the barrier for enterprises to deeply apply large models: the former solves “capability controllability,” and the latter solves “operational reliability.” For enterprises, the real question is no longer “whether to use large models,” but “which scenarios should be privatized, which should be managed, and how to combine them for the optimal implementation path.” Understanding the boundaries and cooperation of these two technical routes means understanding the next phase of LLM engineering implementation.