From Open Weights to Managed Agents: Deploying Large Models in the Enterprise

1. Introduction: Enterprise LLM Adoption Enters the “Open Weights” Deep Water

For the past two years, the dominant way enterprises used large models was paying per token through an API — zero operations overhead, pay-as-you-go, but inherently limited in three ways: data must leave the enterprise boundary, deep customization against industry corpora is difficult, and the pace of capability upgrades is controlled by the provider. A new trend has recently emerged: leading vendors have begun opening the weights of flagship models with total parameter counts in the trillions, while simultaneously launching “managed agent” services that deliver model capabilities together with an agent runtime framework. This means competition in large models is extending from “model capability” to “engineering ecosystem,” and for the first time enterprises can obtain capabilities close to flagship closed-source models under the premise that data never leaves their domain and deep customization is possible. This article breaks down the key issues behind this trend from a technical perspective.

2. What Open Weights Actually Change

For developers and enterprises, opening full weights means the way a model is acquired shifts from “renting” to “owning.” The differences between the two modes can be summarized as follows:

Dimension	API Access	Self-Hosted Weights
Data flow	Must leave the enterprise boundary	Data never leaves the domain
Customization depth	Prompts, RAG	Secondary fine-tuning to inject industry knowledge
Operations cost	Nearly zero	Requires building your own inference cluster
Version cadence	Decided by the provider	Controlled by the enterprise

Three typical value scenarios:

1. Heavily regulated industries: In finance, healthcare, and government scenarios, sensitive data never leaves the domain throughout the entire process, satisfying regulatory and audit requirements;
2. Deep industry customization: On top of open weights, continued pre-training or instruction fine-tuning with domain corpora lets the model “speak the industry’s language”;
3. Autonomous capability cadence: Version upgrades and canary releases are decided by the enterprise itself, avoiding the impact of external interface changes on production business.

3. Technical Profile of Ultra-Large MoE Models

Take the recently released 3T-class model as an example — roughly 2.8 trillion total parameters, on the order of one hundred billion activated parameters per inference pass, a one-million-token context window, and native multimodal visual understanding. Behind each of these numbers lies a layer of engineering meaning.

3.1 MoE Architecture: Decoupling Parameter Scale from Inference Cost

The MoE (Mixture of Experts) architecture uses a routing network to assign each token to a small number of expert sub-networks for processing. Of the 2.8 trillion total parameters, only about one hundred billion are activated per inference pass, producing an intuitive result: the model’s knowledge capacity approaches that of a dense trillion-parameter model, while per-token computation is equivalent only to a dense model in the hundred-billion class. Training capacity and inference cost are thereby decoupled, which is also the precondition that makes ultra-large-scale openness possible.

3.2 Memory Estimation: The First Arithmetic Problem Before Deployment

Regardless of architecture, weights must reside permanently in GPU memory. A rough estimate can be made as follows:

# Rough MoE model memory estimate (illustrative logic, not precise values)
total_params = 2.8e12          # total parameter count
bytes_per_param = 1            # FP8 quantization is about 1 byte per parameter
weight_mem_tb = total_params * bytes_per_param / 2**40
# Conclusion: even with aggressive quantization, the weights alone require
# multiple TB of GPU memory; once KV cache is added, multi-machine,
# multi-GPU deployment is the standard.

The conclusion is clear: even with FP8 or more aggressive quantization, the weight portion still requires several terabytes of GPU memory, and multi-machine deployment is unavoidable. This explains why “whether the hardware ecosystem can complete adaptation in tandem” is a prerequisite for on-premises deployment.

3.3 Million-Token Context: Capability and Cost Go Hand in Hand

• On the capability side: book-length document analysis, one-shot understanding of large code repositories, and cross-session memory consolidation — many scenarios that previously required chunked retrieval can be simplified to “put it directly into the context”;
• On the cost side: the KV cache expands linearly with context length; at one million tokens, a single sequence’s cache alone can reach tens of GB. Production environments typically require prefix cache reuse, tiered context residency, and optimized attention computation working together, otherwise throughput becomes unacceptable.

3.4 Training Paradigm: Base Model Iteration Rather Than Training from Scratch

It is worth noting that such models are typically not brand-new bases trained from scratch, but rather capability leaps completed on top of an existing base through large-scale cross-domain joint reinforcement learning. This paradigm is more data-efficient and inherits capabilities strongly, but its capability ceiling is also constrained by the base — when evaluating a model, pay attention to its iteration path rather than only the parameter scale.

4. Four Engineering Problems of On-Premises Deployment

4.1 Inference Frameworks and Hardware Adaptation

Ultra-large MoE places extremely high demands on expert parallelism and communication optimization in inference frameworks. Industry experience shows that whether mainstream open-source inference frameworks and domestic compute hardware can be adapted quickly after a weight release directly determines the accessibility of on-premises deployment. Adaptation work usually revolves around three main threads:

• Expert parallelism scheduling: reasonably partitioning expert sub-networks across multiple GPUs and machines to balance load;
• Cache reuse: using prefix caching to reduce the computation and memory overhead of repeated context;
• Weight quantization: compressing memory footprint while keeping precision loss controllable.

4.2 A Combined Approach to Memory and Communication Optimization

Quantization (FP8/INT4), KV cache compression, and on-demand expert loading and offloading often need to be used in combination. A single optimization can hardly compress a multi-terabyte requirement into an acceptable range; in engineering terms it is more like a set of parameter tuning problems.

4.3 Deployment Paths on the Cloud

Enterprises without the ability to build their own data centers can build a deployment pipeline on general-purpose public cloud capabilities such as GPU compute clusters, container platforms, object storage, logging services, and monitoring and alerting. Cloud providers offer these general product categories, which can turn weeks of cluster operations into on-demand provisioning. For enterprises with stricter compliance requirements, a hybrid deployment with dedicated network access is also an option, balancing data boundaries against elastic compute.

4.4 Serviceification and Observability

Deployment is only the starting point. Productionization also requires rate limiting and quotas, canary releases, call-chain tracing, token cost accounting, and content safety auditing — items often underestimated by teams doing their first self-deployment.

5. The Agent Harness Problem: The Runtime Is Harder Than the Model

The model is only the agent’s “brain.” For an agent to run reliably 24/7 in a real business, a whole外围 engineering system is needed, known in the industry as the Harness. Its core elements include:

• Context management: memory compression and state maintenance during long tasks;
• Tool invocation: function registration, parameter validation, timeout and retry policies;
• Permission control: code sandbox execution, manual approval for sensitive operations;
• Task interruption and recovery: checkpoint persistence, idempotent replay;
• Multi-agent collaboration: task distribution, result aggregation, conflict resolution.

Practical experience shows that building a reliable Harness in-house often requires months of engineering investment, and after launch it still demands ongoing maintenance cost. For most business teams, this is a severely underestimated hidden cost item.

6. Managed Agent Services: Turning the Harness into a Cloud Service

The “managed agent” model that has recently emerged is precisely a response to the above pain points: providing the Harness framework that supports agent operation as a cloud service, so enterprises can obtain a 24/7 managed runtime without building and maintaining the underlying runtime framework, supporting long-running complex task execution and multi-agent collaboration.

Two typical deployment scenarios:

1. Integrated into internal enterprise systems: undertaking on-duty tasks such as business monitoring and operations response, long-running and frequently triggered;
2. Embedded in user-facing products: service-oriented tasks such as research assistants and intelligent customer service, emphasizing multi-turn interaction and tool orchestration.

In this model, the work on the enterprise side is abstracted into three things: define the task, configure the agent, and connect business tools; maintenance and upgrades of the underlying runtime are the provider’s responsibility. An illustrative configuration is as follows:

agent:
  name: ops-responder
  model: 3t-open-weight
  context_window: 1_000_000
  tools: [metrics_query, log_search, ticket_create]
  permissions:
    allow: [metrics_query, log_search]
    require_approval: [ticket_create]
runtime:
  mode: hosted        # hosted mode, underlying runtime maintained by the provider
  retry:
    max_attempts: 3
  checkpoint: true    # supports checkpoint recovery for long tasks

Value judgment: the managed model compresses the cycle from prototype to production from months to days, at the cost of runtime dependency on an external service. Enterprises with extremely sensitive data still need to weigh efficiency against boundaries.

7. Open Weights ≠ Open Source: Compliance Points

A common misconception must be clarified: open weights are not equivalent to open source in the traditional sense.

• What is not disclosed: training code, raw training datasets, and data cleaning pipelines are typically not provided;
• Commercial constraints: use must follow the vendor’s custom license agreement, with attention to attribution requirements, commercial thresholds, and whether derivative models must be similarly open;
• Recommended actions: complete a legal review of the license before launch; clarify the boundaries of redistribution and fine-tuned models; and establish a content compliance review mechanism for model outputs.

8. Selection Advice: A Three-Path Decision Framework

Based on the above analysis, enterprises can decide along four dimensions: data sensitivity, compute budget, customization depth, and time to launch.

• Highly sensitive data + sufficient owned compute → self-hosted weights, combined with quantization and cache optimization;
• Rapid validation, small to mid-size teams → API first, then migrate after validating business value;
• Agent scenarios + weak platform engineering capability → managed agent services;
• Large organizations → hybrid path: non-sensitive business over API, core data domains self-hosted.

A practical suggestion: use the API first to complete business prototype validation, then migrate to self-hosting for scenarios that genuinely have value and compliance necessity, avoiding the burden of multi-terabyte GPU memory operations from day one.

9. Conclusion

The combination of open weights and managed agents essentially splits large model capability into a “model layer” and an “engineering layer” that are opened in layers: the model layer addresses “whether the capability exists,” and the engineering layer addresses “how fast it lands.” For enterprise developers, the real homework is not chasing parameter numbers but thinking clearly about three questions — where the data is, where the compute is, and where the engineering team is. Answer those three questions well, and the technical path emerges naturally.