Trillion-Parameter MoE Open Weights: From Architecture to Deployment

1. Background: Open Weights Enter the Trillion-Parameter Era

The open-source strategy for large language models is undergoing a structural shift. Previously, models with hundreds of billions or more parameters were primarily accessible via online APIs, with their weights treated as proprietary core assets. Recently, however, a provider publicly committed to open weights upon releasing their new generation model and fulfilled this promise weeks later by hosting a Mixture of Experts (MoE) model with approximately 2.8 trillion total parameters in the open-source community, making it available for any developer to download and research.

For the industry, this signal is significant: the competitive focus of open weights has shifted from “whether to open” to “whether the open model is sufficiently powerful and truly usable.” For developers, the value of open weights manifests in at least three areas:

• Auditability: Direct inspection of model architecture, configuration files, and tokenizer implementations to assess safety, compliance, and licensing constraints.
• Customizability: The ability to perform continued pre-training, domain-specific fine-tuning, or knowledge distillation based on the weights, pushing beyond the capability boundaries of online services.
• Transferability: Support for private deployment to meet compliance requirements such as data residency and network isolation.

However, it is crucial to note that a gap exists between “downloadable” and “locally runnable.” Section 5 provides a quantitative analysis of this challenge.

2. Architecture Analysis: How MoE Achieves “Large but Affordable”

2.1 Sparse Activation: Decoupling Parameter Scale from Inference Cost

The model employs a Mixture of Experts (MoE) architecture with roughly 2.8 trillion total parameters, yet only activates about 104 billion parameters per inference pass—an activation ratio of less than 4%. The fundamental principle involves replicating the feed-forward networks in the Transformer into multiple parallel “experts.” A learnable router dynamically selects a small subset of experts for each token to participate in the computation:

# Pseudo-code for MoE forward pass
def moe_forward(x, experts, router, top_k=2):
    gates = softmax(router(x))              # Calculate expert weights for each token
    topk_gate, topk_idx = topk(gates, k=top_k)
    y = 0
    for g, i in zip(topk_gate, topk_idx):
        y = y + g * experts[i](x)           # Only compute the selected experts
    return y

This design introduces a critical decoupling: computation scales with activated parameters, while storage scales with total parameters. As long as VRAM or storage capacity is sufficient, model capacity can be expanded by stacking more experts without increasing the per-token inference computation. This is how “2.8T total parameters and 104B activated parameters” coexist—trading sparsity for the separation of capacity and cost.

Naturally, MoE introduces new challenges: routing collapse (uneven expert load), inter-expert communication overhead, and total VRAM expansion. These require supporting technologies like load-balancing loss and expert parallelism to mitigate, which is why the engineering complexity of training and inference for MoE models is significantly higher than for dense models.

2.2 Gated Sparse Attention: Supporting Million-Token Contexts

The model supports a context window of approximately 1 million tokens. In standard full attention, computation and intermediate states grow quadratically with sequence length. At the million-token scale, the attention matrix and KV cache become overwhelming sources of overhead.

The model’s approach is to base its foundation model and replace some full attention layers with gated sparse attention layers. A learnable gate filters a small number of the most relevant Key-Value pairs for each query token to participate in the attention computation, reducing the layer’s overhead from O(n²) to approximately linear. Simultaneously, a portion of full attention layers is retained to maintain the ability to model global dependencies.

This is a common technical route in current long-context modeling. Similar approaches include sliding window attention, block attention, and retrieval-augmented sparsification. Their shared philosophy is: making “what to look at” a learnable decision, rather than computing equally for every token.

3. Post-Training: Large-Scale Joint RL Shaping Agentic Capabilities

Base parameters are merely “potential.” Agentic capabilities—multi-turn tool calling, long-horizon task planning, and error correction based on environmental feedback—are primarily shaped during post-training. Using the previous generation model as a base, this model underwent large-scale joint reinforcement learning (RL) covering hundreds of downstream domains.

Why do agentic scenarios rely heavily on RL rather than just Supervised Fine-Tuning (SFT)? This can be understood from three perspectives:

1. Insufficient coverage of demonstration trajectories: SFT can only mimic human-demonstrated trajectories. In real environments, tool return values and error messages vary widely. Once reality deviates from the demonstration, the model tends to persist down the wrong path.
2. More objective reward signals: Agentic tasks inherently carry feedback: did the command execute successfully, did the test pass, did the user confirm completion? RL can directly use these outcome signals to optimize policies, teaching the model the closed loop of “attempt—observe—correct.”
3. Positive transfer from multi-domain joint training: Joint training across hundreds of domains prevents overfitting in a single domain. Capabilities such as coding and tool usage often mutually reinforce each other—improvements in coding ability typically enhance the format stability and structured output quality of tool calls simultaneously.

This training orientation also explains the significant improvement in the “steerability” metric seen in subsequent evaluations: the model is explicitly trained to actively respond to human correction signals rather than stubbornly executing its original plan.

4. Evaluation Innovation: From Pairwise Blind Tests to Session-Level Causal Tracking

4.1 Traditional Blind Tests Fail for Agentic Tasks

Mainstream model leaderboards have long utilized pairwise blind testing: having users compare responses from two anonymous models and vote. This method suits single or few-turn Q&A scenarios but has a structural blind spot for agentic tasks—whether a task truly succeeds is often only determinable by the user after multiple execution turns or even after a period of real use. A single, rapid vote cannot capture these delayed success or failure signals.

4.2 Session Signal Extraction and Causal Attribution

The evaluation approach for the new generation of agents is fundamentally different. Instead of asking users to vote, it automatically extracts process signals from real user agent sessions, including:

• Explicit Confirmation: The user explicitly states the task is complete and the result is usable.
• Praise and Complaints: Positive or negative feedback statements appearing in the session.
• Correction Commands: The user interrupts execution and demands a change in direction.
• Execution Errors: Environmental failure signals such as shell command errors.

Using randomized inference causal attribution methods—where different models are randomly assigned to handle real traffic, naturally providing comparability—the net improvement rate of each model relative to the average level is estimated. This model’s ranking is based on over 17,500 real long-cycle agent sessions from global users, achieving a net improvement rate of +5.1%, a 4.2 percentage point increase from the previous generation’s +0.9%. It ranks first among open-weight models and 13th among all evaluated models on a third-party agent leaderboard, securing the top spot for open-weight models in the Work, Coding, and Chatting categories.

4.3 Sub-Metrics and Their Engineering Implications

Sub-metrics carry more information than total scores:

Metric	Value Change	Engineering Implication
Shell Command Reliability	-4.5% → +7.9%	Stability of the tool execution chain, directly determining the success rate of automated tasks
Praise/Complaint Ratio	+1.3% → +7.7%	Proxy metric for user subjective satisfaction
Steerability	+6.5%	Ability to adjust execution direction based on feedback, crucial for multi-turn tasks
Confirmation Success Rate	+3.9%	Proportion of explicit user confirmations of completion, closest to “real task success rate”

The shift of Shell Command Reliability from negative to positive is particularly noteworthy. A negative value meant that the model’s error rate when handling real terminal tasks was higher than the average level. This is a classic source of “high benchmark scores but failing in actual use,” indicating that agent selection cannot rely solely on Q&A benchmarks; process metrics correlate more strongly with real success rates.

For reference, on a public terminal agent benchmark, the model scored 65.7%, less than 1 percentage point behind the strongest closed-source flagship model (66.4%). Open-weight models now possess the strength to compete head-to-head with top-tier closed-source models in the agentic track.

5. Deployment Barriers: What 5.6TB of Weights Means

5.1 A Simple Calculation

The most easily overlooked barrier of open weights is their sheer size. Stored in BF16 precision, 2.8 trillion parameters require:

total_params = 2.8e12
for name, bpp in [("BF16", 2), ("INT8", 1), ("INT4", 0.5)]:
    print(f"{name}: {total_params * bpp / 1e12:.1f} TB")
# BF16: 5.6 TB / INT8: 2.8 TB / INT4: 1.4 TB

This means approximately 5.6TB for BF16, and even when compressed to INT4, it still requires about 1.4TB. For context, current consumer graphics cards typically have 8 to 24GB of VRAM. Even a data-center-grade 8-GPU high-end server only has about half a TB of total VRAM—more than an order of magnitude short of the 1.4TB quantized weights, making single-machine deployment virtually impossible. Furthermore, quantization itself can lead to capability degradation; post-compression, regression testing with real business tasks is mandatory.

5.2 Viable Deployment Paths

Faced with this scale, realistic options generally fall into three categories:

1. Directly call cloud model services: Billed per token, zero operations required, suitable for the vast majority of businesses in the validation phase.
2. Self-hosted inference on cloud clusters: Build a multi-node inference cluster leveraging general cloud capabilities like GPU cloud servers, container platforms, load balancing, and monitoring. This is suitable for data-sensitive, high-volume scenarios and requires engineering optimizations such as expert parallelism, tiered parameter loading, and KV cache management.
3. Distill small models: Use the large model as a teacher to distill a student model capable of single-machine deployment, finding a compromise between cost and performance.

For most teams, the more robust sequence is to “first validate business value with a managed service, then decide whether to bear the cost of privatization.” For instance, teams can first complete proof of concept and stress testing on managed model services and GPU compute on cloud platforms (e.g., Baidu AI Cloud), and then evaluate whether migrating to a self-built cluster is necessary—avoiding heavy capital investment in 5.6TB weights from the start.

6. Conclusion: A Rational View on “Large, Powerful, and Open”

This case study reflects three trends in the LLM industry: MoE sparsification decoupling parameter scale from inference cost, RL post-training becoming the primary source of agentic capabilities, and session-level process evaluations replacing single-turn blind tests. For developers, understanding the methodology behind the metrics and conducting regression validations with real business tasks is more important than chasing leaderboard rankings. Ultimately, open weights only provide a possibility; whether they can be transformed into productivity depends on the combined forces of architectural understanding, evaluation methodology, and deployment strategy.