Open-Sourcing a Trillion-Parameter MoE Model: Architecture and Practice

1. Introduction: Open-Source LLMs Enter the Trillion-Parameter Era

In recent years, the parameter scale of large language models has continued to climb, and both the open-source community and industry have been exploring the balance between model capability and openness. A domestic AI research team recently released the full weights of a Mixture of Experts (MoE) model with 2.8 trillion total parameters, drawing widespread industry attention. The core technical significance of this event is twofold: an open-source model has, for the first time, significantly outperformed previous open-source benchmarks on comprehensive intelligence evaluations, while a highly permissive license lowers the barrier to commercial adoption for enterprises.

For developers and technical decision-makers, understanding the architecture design, training strategy, and licensing constraints behind such a model is a key prerequisite for evaluating whether and how to adopt it. This article provides an in-depth analysis across four dimensions: architectural evolution, attention mechanism innovation, evaluation performance, and commercial licensing.

2. MoE Architecture: Leveraging Trillion-Scale Parameters Through Sparse Activation

2.1 Basic Principles

The core idea of a Mixture of Experts model is “large total capacity, small per-inference usage.” The model holds a massive number of parameters (approximately 2.8 trillion in the model discussed here), but during each inference, a gating network dynamically selects a small number of “expert” subnetworks to participate in computation. This model activates only about 104 billion parameters per inference, which means:

• Training cost: Although total parameters are enormous, each forward/backward pass only updates the weights of activated experts, so the computational load is far lower than that of a dense model of equivalent scale.
• Inference efficiency: Under the same hardware conditions, an MoE model can complete inference with lower compute consumption, significantly reducing per-request latency and energy use.

The routing logic can be illustrated with pseudocode:

def moe_forward(input_tokens, experts, gating_net):
    # gating_net outputs the selection probability of each token for each expert
    router_logits = gating_net(input_tokens)          # [batch, seq_len, num_experts]
    top_k_probs, top_k_indices = top_k(router_logits, k=2)
    # Dispatch tokens only to the selected experts for computation
    expert_outputs = []
    for expert_id in range(num_experts):
        mask = (top_k_indices == expert_id)
        selected_tokens = input_tokens[mask]
        if selected_tokens.numel() > 0:
            expert_outputs.append(experts[expert_id](selected_tokens))
    # Weighted aggregation
    final_output = weighted_sum(expert_outputs, top_k_probs)
    return final_output

2.2 Comparison with Dense Models

In a traditional Transformer model, all parameters in each layer participate in the computation for every token. MoE achieves a leap in parameter utilization through “expert division of labor”: a model with 2.8 trillion total parameters has an actual inference compute load close to that of a 104-billion-parameter dense model, yet gains knowledge capacity and expressive power far beyond that scale. Given current constraints on GPU memory and compute, this design is one of the mainstream technical routes for realizing ultra-large-scale models.

3. Attention Mechanism Innovation: From Full Attention to Sparse Attention

3.1 The Bottleneck of Full Attention

The computational complexity of the standard self-attention mechanism is O(n²), where n is the sequence length. When the context window reaches the scale of 1 million tokens, the memory and compute overhead of full attention becomes unsustainable. Earlier models (such as the version open-sourced by this team in July) used a 24-layer full-attention structure, which limited efficiency in long-text scenarios.

3.2 The Core Idea of Sparse Attention

The new version replaces attention layers with a sparse attention scheme: each token no longer computes attention scores with all other tokens, but only with the 2048 most relevant tokens. Key technical points of this design include:

• Locality preservation: Through Top-K selection, the model can still capture long-range dependencies, because the most relevant tokens are not necessarily nearby ones.
• Reduced complexity: Computation drops from O(n²) to O(n·k), where k=2048 is a constant, yielding enormous savings in million-token scenarios.
• Hardware friendliness: Sparse computation patterns can be implemented efficiently on existing GPU architectures through engineering techniques such as blocking and indexing.

This improvement directly supports the practical use of the model’s 1-million-token context window, enabling it to process full-length novels, large-scale code repositories, or multi-frame analysis of long videos.

4. Evaluation Performance and Cost Efficiency

4.1 Comprehensive Intelligence Index

On the latest comprehensive intelligence index of a third-party independent evaluation platform, the model ranks first among open-source models with a score of 46.5, roughly 14 points higher than the previous highest-scoring comparable open-source model. This gap represents a significant lead in AI evaluation systems, reflecting the model’s overall strength across reasoning, coding, knowledge QA, and other multidimensional tasks.

4.2 Terminal Operation Capability Test

In another independent test of “AI capability to execute tasks in a computer terminal environment,” the model achieved 63.64% accuracy, ranking among the top of nearly 60 evaluated models and second only to a few closed-source commercial models. This means the model has strong practical value in high-frequency developer scenarios such as code generation, command execution, and file operations.

4.3 Per-Inference Cost

Notably, while maintaining high capability, the model’s average per-inference cost is approximately $4.57 (estimated according to the test standard), whereas some contemporaneous closed-source flagship models cost as much as $13.20 per inference, about three times higher. For enterprise applications requiring large-scale API calls, this cost difference has important economic significance.

5. Open-Source License: Permissive but with Boundaries

5.1 MIT License as the Main Body

The license for the model weights follows the general wording of the MIT license, granting users the following rights:

• Free use, copying, modification, and redistribution;
• Commercial use permitted, including embedding the model into proprietary products or services.

This means the vast majority of enterprises can use the model for internal systems, SaaS products, or private deployments with zero barrier.

5.2 Two Additional Restrictions

To prevent unfair competition in extreme scenarios, the publisher set two additional clauses:

1. API provider constraint: If a company or its affiliates engage in the business of selling model APIs and the group’s total revenue over 12 consecutive months exceeds $20 million, it must sign a separate agreement with the publisher before commercial use.
2. Super-app attribution: If a product built on the model exceeds 100 million monthly active users or $20 million in monthly revenue, the product interface must prominently attribute the model source.

For internal enterprise use, access through official channels, or use via certified inference services, the above restrictions do not apply. This “permissive entry, strict exit” strategy is relatively rare in the open-source community and aims to balance openness with ecosystem health.

5.3 Practical Impact on Developers

• Small and medium teams: Essentially unrestricted; free commercial use.
• Large platform companies: Need to assess whether their business touches the two red lines; if API sales or ultra-large-scale applications are involved, legal review should be conducted in advance.
• Cloud service providers: If providing hosted inference services based on the model, they need to confirm whether they fall under the category of “certified inference service providers.”

6. Enterprise Adoption Recommendations and Best Practices

6.1 Deployment Architecture Selection

For enterprises planning to deploy models at this scale, the following architecture patterns are recommended:

• Inference acceleration: Use distributed inference frameworks such as tensor parallelism and pipeline parallelism to fully leverage the sparse computation advantages of the MoE architecture.
• Memory planning: Although only 104 billion parameters are activated, the 2.8 trillion total parameters need to be sharded across multiple GPUs or nodes; high-bandwidth interconnect (such as NVLink or equivalent high-speed networks) is recommended.
• Long-context optimization: Combined with the sparse attention mechanism, adopt chunked loading and KV cache management strategies to avoid memory overflow in million-token scenarios.

6.2 Compliance Review Checklist

Before formal commercial use, enterprises should verify each item:

Check Item	Description
Business type	Whether it involves selling model APIs externally
Revenue scale	Whether group revenue over 12 months exceeds $20 million
Product scale	Whether MAU or monthly revenue triggers the super-app clause
Usage method	Whether access is through official or certified channels

6.3 Collaboration Perspective with Baidu AI Cloud

Baidu AI Cloud provides complete infrastructure capabilities covering compute, storage, networking, and AI development platforms. When deploying large-scale open-source models, enterprises can leverage the cloud platform’s general capabilities—such as elastic compute scheduling, object storage for model weight management, and container service for inference cluster orchestration—to reduce the operational complexity of self-built infrastructure. Through the cloud platform’s monitoring, alerting, and logging services, enterprises can also observe large-scale inference requests in real time and control costs.

7. Conclusion

The official open-sourcing of a trillion-parameter MoE model marks a new capability tier for the open-source LLM ecosystem. The combination of sparse attention and MoE architecture makes “large parameters, low activation, long context, and commercial usability” possible. For developers, the key lies in thoroughly understanding architecture details, carefully evaluating licensing constraints, and reasonably planning deployment solutions. As more teams join the open-source ranks, the democratization of large model technology will further accelerate, and enterprise adoption will move from “whether it can be used” to “how to use it well.”