Open-Source 2.8T-Parameter Sparse Attention MoE Model Released: Architecture, Benchmarks, and License Analysis

Background: A Weight Release Delivered as Promised

A mixture-of-experts (MoE) model with 2.8 trillion total parameters has recently had its full weights made publicly available on a model hosting platform. Unlike earlier releases that often combined a technical report with a vague commitment to open weights at some later point, this release included an explicit timeline from day one—and the weights were uploaded on the promised date.

For the open-source community, the significance goes beyond having another massive model available for download. It reinforces a broader shift: leading research organizations are moving from ambiguous statements about whether a model will be open-sourced, toward concrete commitments about when it will be released, under which license, and with which commercial restrictions. For enterprise technical decision-makers, this predictability is valuable in itself—it means internal evaluation, prototype validation, and compliance assessments can be built around a stable timeline.

Model Architecture: MoE Combined with Sparse Attention

Based on publicly available technical information, the model uses a mixture-of-experts architecture. Total parameters stand at 2.8 trillion, but only approximately 104 billion are activated during each inference pass. The design intent is to decouple model capacity from per-inference compute: larger total parameter counts increase the knowledge capacity and expressive power of the model, while smaller activated parameter counts keep floating-point operations per inference manageable, reducing pressure on memory bandwidth and compute units during deployment.

Compared with the previous model in the same series, the attention mechanism has undergone a notable change. The earlier version used standard full attention throughout the model. The new version replaces 24 layers with sparse attention: for each token, the model selects only the 2,048 most relevant tokens from the context to participate in attention computation, rather than interacting with all tokens in the context.

This change brings clear benefits in long-context scenarios. If the context length is N, standard attention scales quadratically with N. When N reaches hundreds of thousands or even millions of tokens, full attention quickly becomes an inference bottleneck. Sparse attention limits the number of key-value pairs each query token attends to, keeping computational complexity linear in N. For long documents, repository-scale code contexts, or extended conversation logs, this significantly reduces per-request latency and cost.

Context Window and Multimodal Input

The model supports a context window of up to 1 million tokens. A long context window is not just a marketing number—it determines how much information the model can observe in a single inference pass. In code generation scenarios, 1 million tokens is enough to hold the complete codebase of a medium-to-large project. In document processing, it can intake several thousand pages of technical documents or contracts at once, without relying on external retrieval modules to chunk and recall relevant sections.

The model also has native multimodal capabilities, directly accepting image inputs. “Native” here means that visual and textual representations were unified within the same parameter set during training, rather than implemented by attaching a vision encoder and a text adaptation layer on top of an existing model. The typical advantage of this approach is more stable cross-modal performance, particularly in inference tasks involving mixed image-text inputs, where insufficient modality alignment can otherwise cause information loss.

Third-Party Evaluation Results

In a third-party evaluation’s composite intelligence index, the model scored 46.5. For reference, the best-performing open-source model previously scored 32.6 in the same benchmark. A gap of roughly 14 points is a substantial improvement in current LLM evaluation terms. That said, the composite index aggregates scores across multiple sub-tasks, and a single number does not fully capture how the model will perform in a specific business context. When selecting a model, it is still advisable to run targeted evaluations on your own task types—such as text extraction, code generation, logical reasoning, or document summarization.

Another noteworthy benchmark examined agent performance in terminal environments. The model achieved a 63.64% completion rate on a set of terminal tasks, ranking third among the 59 models evaluated. The two models ranked above it are closed-source commercial models, with significantly higher per-question costs. Public data shows this model costs approximately $4.57 per question on average, while the top-ranked closed-source model costs $13.20—roughly three times higher.

Cost comparisons of this kind should be read carefully. Terminal agent task costs depend not only on the model’s inference pricing but also on the number of tool calls, context length, and interaction rounds with the environment. Still, based on evaluation data, this model demonstrates favorable cost efficiency while remaining competitive with leading closed-source models. That is relevant for teams building automated agents on constrained budgets.

License Analysis: Permissive with Two Specific Clauses

The license terms deserve close attention from both legal and technical leadership. The main license text largely follows the MIT License wording, meaning individuals and companies may freely use, modify, redistribute, and commercially deploy the model weights. Two additional restrictions are attached.

The first restriction targets resellers of model APIs. If a company or its affiliates sell model APIs, and the group’s total revenue exceeds $20 million over any continuous 12-month period, a separate agreement with the model publisher is required before commercial use. The logic is straightforward: it allows smaller service providers to integrate and resell inference services under the revenue threshold without additional negotiation, but once a company reaches a certain commercial scale, it must renegotiate revenue sharing with the model publisher. For the vast majority of enterprises using the model internally, this restriction will not apply.

The second restriction addresses super-apps. If a product built on this model exceeds 100 million monthly active users or $20 million in monthly revenue, the product interface must visibly display the model name. This is essentially an attribution requirement—it does not involve additional fees or license negotiation. For almost all current applications, 100 million monthly active users is a very high threshold, and $20 million in monthly revenue is also beyond what early-stage products typically approach.

It is important to note: if a company downloads the model weights and deploys them on its own infrastructure, using the model only for internal systems without offering external API services, neither of the two restrictions applies. Likewise, calling the model through the publisher’s official product or through certified inference service providers is not subject to these clauses. For most enterprise users, the commercial freedom of this release is therefore very close to a standard MIT license.

Deployment and Integration Considerations

Weights being open is one thing; getting from “downloaded weights” to “production-ready” is quite another. Teams considering self-deployment should assess the following areas first:

Memory and distributed deployment. Even after quantization, the full weight files for 2.8 trillion total parameters are extremely large. Realistic deployment requires multi-GPU distributed inference, typically with tensor parallelism or pipeline parallelism. Even with only 104 billion activated parameters per inference, KV cache memory usage grows quickly with concurrent requests in long-context scenarios. Memory budgets and request queue management should be planned at the architecture design stage.

Inference framework compatibility. Sparse attention layers require the inference framework to support custom attention masks or sparse operators. If the framework you are using has not been optimized for this structure, actual inference speed may fall short of theoretical performance. Before integrating, run benchmarks on hardware similar to your production environment, rather than relying on the performance figures published by the model developer.

Engineering adaptation for long contexts. A 1-million-token context window imposes new requirements on upstream systems. Text chunking strategies, input construction methods, and timeout settings all need to be redesigned. For instance, conventional input pipelines that split documents into fixed-length chunks and concatenate them may not take full advantage of a long window. Conversely, feeding an entire document collection into the model in a single pass requires weighing inference time against request blocking.

Cost and SLAs. Self-hosted large models behave differently from traditional microservices in terms of cold start time, failure recovery, and version upgrades. If your team does not have experience operating large-scale GPU clusters, a more practical path is to host the model on a cloud inference platform that supports this weight, letting container orchestration and inference service frameworks handle resource scheduling. In this arrangement, end users still benefit from the controllability and data compliance advantages of open weights, while avoiding the underlying infrastructure complexity.

Suitable Use Cases for This Model

Based on its capabilities and license conditions, the model is best suited to three types of business scenarios:

Knowledge-intensive applications requiring deep understanding of long documents. With its very large context window and sparse attention reducing the cost of long inputs, it fits use cases such as contract summarization, research report analysis, and cross-document comparison. Compared with traditional retrieval-augmented pipelines, feeding long contexts directly can reduce information loss caused by retrieval omissions.

Internal development tools and code assistance. Tasks such as large-scale codebase understanding, cross-file refactoring suggestions, and test case generation demand substantial context and structural comprehension. Using this model internally within a development workflow, without exposing an external API, generally does not trigger the additional license restrictions.

Cost-sensitive agent prototypes. Based on terminal task benchmarks, the model shows a clear advantage in per-task cost. For teams validating agent feasibility, building prototypes and running workflow tests with this model allows cost-controlled evaluation of task quality, with the option to switch or keep the model based on observed results.

Conclusion

This release sends a clear signal: ultra-large-scale sparse models are transitioning from laboratory demonstrations to enterprise-accessible infrastructure. A 2.8-trillion-parameter model with 104 billion activated parameters, a 1-million-token context window, and a near-MIT license offers developers and businesses a new set of options.

But before adoption, the engineering realities remain: evaluate inference cost, deployment complexity, framework compatibility, and how the license applies at your own business scale. Open weights are only the starting point. What ultimately determines business value is how well these capabilities are embedded into specific workflows and how reliably they run within a controlled cost boundary.