A Trillion-Parameter Open-Source Sparse Model: Engineering Practices and Deployment Lessons from the 2.8T Era

1. The ceiling for open-source models has been raised again

The open-source large model ecosystem is undergoing a clear generational shift. Over the past two years, most active open-source models have ranged from tens of billions to hundreds of billions of parameters. As sparse model architectures have matured, leading research teams have begun releasing full weights at the trillion-parameter scale while keeping inference costs relatively manageable.

A recent milestone came when a research team officially released the weights of a mixture-of-experts model with a total of 2.8 trillion parameters. The team had committed to releasing the full weights by a specific date and ultimately uploaded them on schedule to a public model hosting platform. This move drew attention from multiple researchers overseas, reflecting a broader trend toward more predictable and verifiable release schedules for open-source models.

Public benchmark results show that the model achieves a composite intelligence score of 46.5, significantly higher than the previous best open-source score of 32.6, a gap of roughly 14 points. This indicates that the capability gap between open-source and leading closed-source models is narrowing further.

2. Mixture-of-experts architecture: trading sparsity for inference efficiency

The model adopts a mixture-of-experts architecture, commonly known as MoE. Its total parameter count is 2.8 trillion, but each inference pass activates only about 104 billion parameters. This design effectively uses a larger parameter capacity to improve the model’s expressive power while controlling computational cost through sparse activation.

Compared with the team’s previous-generation model, this release introduces a key architectural change: the network, originally consisting of 24 layers with full attention, has been updated to use a sparse attention mechanism proposed by the research community. Specifically, when computing attention for each token, the model selects only the 2048 most relevant tokens from the context instead of attending to all historical tokens.

This change is particularly important for long-context tasks. When the context window reaches 1 million tokens, the memory and computational overhead of full attention grows rapidly with sequence length. By limiting the attention range of each token, sparse attention can significantly reduce resource consumption during long-sequence inference while preserving the ability to recall key information.

In engineering terms, sparse attention typically requires additional support in the inference framework. For example:

Input sequence length: 1,000,000 tokens
Attention range per token: 2,048 tokens
Approximate attention computation complexity: O(n x k)
where n is the sequence length and k is the sparse window size

Compared with the O(n²) complexity of traditional full attention, sparse attention offers clear scalability advantages for extremely long text scenarios.

3. Multimodal and long-context capabilities: native image input and a million-token window

The model is not designed solely for text tasks. Native multimodal support means it can directly accept image input without relying on an external vision encoder for post-hoc integration. This native design typically processes text and image tokens together during pretraining, giving the model more consistent representations across multimodal understanding tasks.

The 1-million-token context window also opens up new possibilities for enterprise applications, such as:

• Parsing extremely long documents: reading an entire book, a collection of financial reports, or a technical manual in a single pass.
• Repository-level code understanding: using the main source files of an entire codebase as context for cross-file reasoning.
• Long multi-turn tasks: maintaining effective references to early information across extended sessions.

It is important to note that long-context capability cannot be judged by window size alone. Position awareness, information recall, and reasoning consistency across long sequences also matter. The introduction of sparse attention is precisely intended to strike a more practical balance between long-context capability and computational cost.

4. Benchmark performance: general capability and terminal execution ability

In a third-party composite intelligence evaluation, the model ranks first among open-source models with a score of 46.5. The score is derived from weighted performance across multiple task categories, including knowledge-based QA, logical reasoning, mathematical ability, and code generation. Compared with the previous best open-source score of 32.6, the lead is substantial.

More notable is its performance in terminal execution benchmarks. In a benchmark measuring AI agents performing real operations on a computer desktop, the model ranked third out of 59 evaluated models, with a score of 63.64%. Only two closed-source frontier models ranked above it, while another mainstream closed-source model ranked only fifth.

At the same time, the model averaged only $4.57 per task, while the top-ranked closed-source model required $13.20, roughly three times the cost. This suggests that the model offers a favorable cost-benefit ratio in practical task completion.

This combination of high scores and low cost has direct relevance for developers and enterprise users. It implies that in certain scenarios such as desktop automation, command execution, and environment manipulation, open-source models may already be viable candidates for production validation.

5. License design: a compromise between openness and commercial protection

The license for this release largely follows a permissive agreement common in the industry, allowing individuals and companies to use, modify, and redistribute the model free of charge, including for commercial purposes. However, compared with fully permissive licenses, it adds two restrictions.

The first restriction targets API resellers. If a company or its affiliates sell API access to this model, and the entire group’s total revenue exceeds $20 million in any consecutive 12-month period, the company must sign a separate agreement with the model publisher before commercial use.

The second restriction targets super-scale applications. If a product built with this model exceeds 100 million monthly active users or $20 million in monthly revenue, the product interface must display the model name prominently.

The following usage scenarios are not subject to the above restrictions:

• Internal enterprise deployment and use.
• Access through official products or certified inference service providers.
• Standard application development that does not reach the user scale or revenue thresholds mentioned above.

This license design essentially seeks a middle ground between permissive open source and commercial interest protection. For the vast majority of small and medium-sized developers and enterprises, the barrier to entry is very low. For a small number of large cloud service providers or super-scale applications, additional negotiation or attribution obligations apply.

6. What developers should consider when evaluating trillion-scale sparse models

Teams considering adopting such a model should evaluate it from the following dimensions.

Inference cost. Although the model’s total parameter count reaches 2.8 trillion, each inference pass activates only 104 billion parameters. This means the per-inference computational load is closer to a hundred-billion-parameter dense model than a trillion-parameter model. Teams should calculate the actual unit cost based on self-hosted inference or inference service pricing.

Hardware requirements. For self-hosted inference, memory capacity and bandwidth are key concerns. Even with sparse activation, the full weights typically still need to be loaded into GPU memory or high-speed storage during inference. Multi-GPU parallel deployment is a common choice.

Long-context task adaptation. If the business scenario involves extremely long document processing, it is advisable to first test recall and accuracy on target data. While sparse attention reduces computational overhead, information recall may vary across tasks.

Commercialization path. If the team plans to offer model API services externally, it should first determine whether its revenue scale would trigger the additional agreement threshold. For most startups, this restriction is unlikely to apply in the near term.

Alternatives and backups. The advantage of open-source models lies in self-hosting, fine-tuning, and redistribution. However, teams should also monitor subsequent version iterations and community maintenance to avoid over-reliance on a single model version for critical business operations.

7. Conclusion: trillion-parameter open-source models are entering the usable range

The 2.8-trillion-parameter open-source model is no longer a proof of concept confined to the lab. It is becoming a practical tool with real deployment potential. The combination of sparse attention and a mixture-of-experts architecture allows the model to maintain large-scale parameter capacity while controlling inference cost. A permissive license with carefully defined commercial restrictions provides different paths for individual developers, small teams, and large enterprises.

For technical teams, the most important implication of this generation of models is that open-source models can now approach or even surpass some closed-source models in selected long-context, multimodal, and desktop automation tasks. Whether to adopt them, when to adopt them, and how to deploy them will depend on specific business scenarios and cost assessment rather than on whether the model capabilities themselves are sufficient.