Trillion-Parameter MoE Model Open-Sourced: How Sparse Attention and Permissive Licensing Reshape the Tech Landscape

1. Background: Accelerating Open-Source Release of Ultra-Large Models

The AI community recently witnessed a significant open-source release: a Mixture-of-Experts (MoE) model with a total parameter count of 2.8 trillion has officially had its complete weights published. From the initial official announcement to the weight upload, the process took merely two weeks, fulfilling the commitment to release “within two weeks at the latest.” This pace signals that top-tier research institutions are increasing their execution speed and certainty in the open-source domain.

For developers, this means the boundary of accessible cutting-edge model capabilities has expanded once again. More importantly, this release adopts an extremely permissive licensing framework, allowing direct and free use in the vast majority of commercial scenarios. The ceiling for open-source models has been pushed higher once again.

2. Architecture Analysis: MoE + Sparse Attention + Native Multimodal

2.1 Mixture-of-Experts Architecture: 2.8 Trillion Total Parameters, 104 Billion Activated

The model utilizes a Mixture-of-Experts (MoE) architecture. The core concept of MoE is to divide the model into multiple “expert” subnetworks. During each forward inference pass, a routing mechanism activates only a small fraction of these experts. While the total parameter count reaches 2.8 trillion, only approximately 104 billion parameters are activated per inference.

The engineering significance of this design lies in decoupling model capacity from inference costs. The total parameter count determines the model’s upper limit for knowledge storage, while the activated parameter count dictates the computational overhead of a single inference. Developers can achieve knowledge coverage far exceeding that of dense models without significantly increasing inference latency.

From a deployment perspective, MoE still demands high VRAM capacity since all expert weights must be loaded or rapidly swapped in. However, through strategies like expert parallelism and hierarchical loading, viable inference can be achieved on limited hardware resources.

2.2 Sparse Attention: From Full Attention to On-Demand Computation

Compared to the previous generation of architectures, a key technical shift in this release is the replacement of full attention mechanisms with sparse attention in certain layers.

In traditional full attention, each token must compute attention weights with every other token in the sequence, resulting in quadratic complexity growth relative to the sequence length. When the context window scales to the million-token level, computational overhead surges dramatically.

The sparse attention approach dictates that each token only selects a specific number of the most relevant tokens for computation. In this implementation, each token selects only the 2,048 most relevant tokens for attention calculation. This yields two direct benefits:

• More efficient compute for long contexts: The longer the sequence, the more significant the computational savings generated by sparsification.
• More controllable inference costs: It avoids full computation while preserving the ability to capture long-range dependencies.

Various implementation paths for sparse attention exist in the industry, but the core principle involves using routing or selection mechanisms to transition attention computation from “fully connected” to “sparsely connected.”

2.3 Million-Token Context and Native Multimodality

The model supports a context window of 1 million tokens and natively supports image understanding. This implies:

• Long documents, codebases, and extensive multi-turn conversation histories can be ingested into the context all at once.
• Mixed image-text inputs do not require additional splicing adaptation layers; multimodal capabilities are fused during the pre-training phase.

For developers, typical scenarios unlocked by million-token contexts include full-repository code retrieval and comprehension, long-video subtitle analysis, large-scale log troubleshooting, and cross-document reasoning.

3. Performance: Dual Validation of General Intelligence and Terminal Tasks

3.1 General Intelligence Index: A New High for Open-Source

On a third-party evaluation organization’s general intelligence index, the model scored 46.5. For comparison, a previous high-scoring open-source model version scored 32.6, representing an improvement of approximately 14 points.

It is worth noting that general intelligence indexes typically aggregate multiple task types, including language understanding, reasoning, mathematics, and coding. A 14-point improvement is a substantial leap in open-source model iteration, indicating that the model has made tangible progress across multiple capability dimensions.

3.2 Terminal Task Benchmark: Approaching Real-World Work Scenarios

Another evaluation agency conducted independent testing using the Terminal-Bench 4.0, assessing the AI’s ability to execute practical operations within a computer terminal environment, such as file operations, command execution, and environment configuration.

The model scored 63.64%, ranking third among 59 tested models. The two models ranked ahead of it are closed-source commercial models, while another well-known closed-source model ranked fifth. This demonstrates that in the dimension of “letting AI do hands-on work,” open-source models now possess the strength to compete directly with top-tier closed-source models.

3.3 Cost Advantage: $4.57 vs. $13.20 per Task

Even more noteworthy is the cost data. In the terminal task benchmark, the model incurred an average cost of $4.57 per task, whereas the top-ranked closed-source model required $13.20—nearly three times as much.

This comparison reveals a crucial trend: open-source models are not only closing the capability gap with closed-source models but are also beginning to forge an advantage in inference economics. For enterprise scenarios requiring large-scale, high-frequency model invocations, cost differences translate directly into commercial viability.

4. Licensing Design: Primarily Permissive with Two Targeted Restrictions

4.1 MIT-Based Framework, Commercially Friendly

The main body of this open-source license follows the MIT license framework. This means:

• Individuals and companies can use it for free.
• It can be modified and redistributed.
• It can be used for commercial purposes.

For the vast majority of developers and small-to-medium enterprises, this equates to barrier-free usage. The MIT license is one of the most permissive in the industry, carrying low legal risk and minimal compliance costs.

4.2 Two Additional Restrictions: API Resale and Super-App Attribution

The releasing institution added two specific restrictions on top of the MIT framework:

Restriction 1: Large-scale API resale requires a separate agreement.
If a company or its affiliates sell the model’s API, and the entire group’s total revenue exceeds $20 million in any consecutive 12-month period, they must contact the official team to sign a separate agreement before commercial use. This primarily targets large cloud providers or API service providers whose core business is reselling model capabilities.

Restriction 2: Super-apps require attribution.
If a product built using this model exceeds 100 million monthly active users (MAU) or $20 million in monthly revenue, the model’s name must be prominently displayed in the user interface. This is similar to common “attribution requirements” in the open-source community, but the trigger threshold is exceptionally high, meaning most products will never reach it.

4.3 Unrestricted Scenarios

The following scenarios are not subject to the two restrictions above:

• Internal company use.
• Usage through official products or certified inference service providers.

Essentially, only vendors selling APIs at scale need to negotiate separate agreements, and super-apps only need to provide attribution. For average developers and enterprise users, the license poses virtually no additional burden.

5. Practical Recommendations for Developers

5.1 Model Selection Considerations

When deciding whether to adopt this model, evaluate the following dimensions:

• Task type: Determine if capabilities like long context, multimodality, and terminal operations are required.
• Inference cost: Compare the cost per token or per task against existing solutions.
• Deployment conditions: Assess whether sufficient VRAM or accessible inference services are available.
• Compliance requirements: Evaluate if your use case involves API reselling or a massive user scale.

5.2 Deployment Paths

For most teams, it is recommended to invoke the model via certified inference service providers first to avoid the engineering complexity of self-hosting an ultra-large parameter model. If data isolation or custom fine-tuning is required, private deployment can be considered, provided the hardware investment and operational costs are thoroughly evaluated.

5.3 Long Context Usage Tips

A million-token context is not a “free lunch.” In practical use, it is recommended to:

• Place the most critical information at the beginning or end of the context to avoid the “middle dropout” effect.
• Segment and summarize ultra-long inputs before concatenation to reduce noise.
• Integrate retrieval mechanisms (like RAG) to send only relevant fragments into the model.

6. Conclusion

The open-sourcing of this trillion-parameter MoE model marks simultaneous progress for the open-source community across three dimensions: model capability, inference economics, and license friendliness. Sparse attention makes long-context processing more viable, the permissive license grants more freedom for commercial applications, and the cost advantage makes large-scale deployment more imaginable.

For developers, the key question is no longer “is there an available model?” but rather “how to choose and utilize the right model.” Understanding architectural principles, benchmark data, and licensing boundaries is the prerequisite for making sound technical decisions. The ceiling for open-source models has been refreshed; the next phase is the time for engineering implementation.