Inside the 2.8T-Parameter Open-Source MoE Model: Sparse Attention, 1M Context, and What the License Allows

One Release, Two Milestones: Scale and Openness

In the open-source large model ecosystem, every major weight release reflects a combination of engineering capability, compute investment, and attitude toward openness. The newly released ultra-large model uses a mixture-of-experts (MoE) architecture with 2.8 trillion total parameters, of which roughly 104 billion are activated per inference. This design preserves very high model capacity while significantly reducing inference compute costs.

The weights have been uploaded to a public model repository, so developers can obtain and use them directly. The main body of the license follows the widely accepted permissive open-source style: individuals and enterprises may freely use, modify, redistribute, and commercialize the model. Supplemental clauses impose additional requirements only in a small number of large-scale commercial scenarios, so the overall openness remains high.

Unlike the “demo first, weights later” pattern common to previous large model releases, the release timing matched earlier commitments. Delivering weights on schedule helps developers plan projects with confidence and reduces the decision cost of waiting and watching.

Mixture of Experts: Trading Sparse Activation for Inference Efficiency

The architectural core is a mixture-of-experts design. Unlike dense models, which activate all parameters on every inference pass, an MoE model splits the network into multiple expert networks, and each input activates only a subset of them.

For this model, 2.8 trillion total parameters with about 104 billion active per inference means an activation ratio of roughly 3.7%. Theoretical compute requirements are therefore far below a dense model of the same scale. For enterprises, this yields two concrete benefits:

First, significantly lower per-inference cost. Although total capacity is huge, the floating-point operations in each forward pass are dominated by the activated parameters. At the same throughput target, an MoE architecture can serve more concurrent requests with fewer compute resources.

Second, engineering benefits from separating training and inference. During training, large clusters can fully train all experts; during inference, experts are scheduled on demand. This model suits cloud-native deployment, where expert networks can be distributed across heterogeneous compute nodes.

That said, MoE adds engineering complexity. Routing strategy, expert load balancing, and communication overhead all affect real-world performance. Teams planning private deployment should pay close attention to how well their inference framework optimizes MoE execution plans.

Long Context and Sparse Attention: Cutting the Cost of Long Sequences

Context window length is one of the key competitive metrics for large models. This model supports a 1M-token context window, placing it firmly in the long-context category. To control compute costs for long-sequence inference, some layers use sparse attention.

In standard full attention, every token attends to all tokens in the sequence, so compute grows quadratically with sequence length. Once the context extends to hundreds of thousands or millions of tokens, the compute and memory pressure of full attention becomes severe.

Sparse attention prunes the attention matrix: each token computes attention against only its most relevant tokens, reducing complexity from quadratic to roughly linear or lower. In this model, each token selects a small set of the most relevant tokens to participate in the attention computation, which saves substantial compute on long-context workloads.

This design matters for practical scenarios:

• Ultra-long document analysis, such as full manuals, audit reports, or legal texts;
• Multi-turn conversations that require long-term context retention;
• Codebase-level understanding, where an entire repository fits into the context window for retrieval and modification.

One caveat: while sparse attention lowers compute costs, it may provide insufficient information coverage on some dense reasoning tasks. When evaluating the model, enterprises should test long-document QA and cross-paragraph reasoning against their own task types to confirm the results meet expectations.

Native Multimodal Input: Beyond Text

The model natively supports image input, meaning multimodal data was introduced during training rather than bolted on through an external vision encoder after the fact. In natively multimodal models, text and visual features can interact at deeper levels early in the network, improving performance on visual question answering, chart understanding, and screenshot reproduction.

For enterprise applications, multimodality extends to scenarios such as:

• Customer service, where users upload product screenshots and ask questions;
• Operations, where monitoring charts are fed in and the model generates written analysis;
• Office workflows, where scanned contracts, invoices, and technical drawings need to be read.

Note that multimodal inference typically demands more compute and memory. At the deployment level, consider splitting image preprocessing, vision encoding, and text generation into independently scalable pipeline stages so resources can flex with business load.

Benchmark Results: Composite Intelligence and Terminal Task Performance

On a composite intelligence index, the model scores notably higher than the previous leaders in the open-source camp. Scoring systems differ across evaluation organizations, but a lead of roughly 14 points suggests a relative advantage across a broad set of tasks.

More instructive is the terminal task benchmark, which measures a model’s ability to complete real tasks in a computer terminal environment, including command execution, file operations, tool calls, and result verification. The model ranks near the top among dozens of evaluated models, and its average cost per completed task is far below that of some closed-source flagships.

The results point to two trends:

First, model capability is evolving from dialogue generation toward closed-loop task execution. Fluency alone no longer defines value; the ability to complete multi-step operations in constrained environments is becoming the more important metric.

Second, cost control is now a key selection dimension. Even among models with similar capability, per-task costs can differ several-fold. For businesses that call models frequently to automate tasks, cost-effectiveness often matters more than peak capability.

License Breakdown: Permissive by Default, with Two Restrictions

The license deserves close attention in this release. Its main terms follow a permissive open-source license: enterprises may use the model commercially, modify it, and redistribute it free of charge. However, additional constraints apply in two specific scenarios.

The first covers model API service providers. If a company or its affiliates sell the model’s API and the group’s total revenue exceeds $20 million in any consecutive 12 months, a separate agreement with the model provider is required. This clause primarily targets large cloud providers and model platforms and is intended to regulate resale of the model.

The second covers super apps. If a product built on the model exceeds 100 million monthly active users or $20 million in monthly revenue, the product must prominently display the model’s name in its interface. This clause does not prohibit commercial use; it requires attribution.

For the vast majority of enterprise users, the following remain unrestricted:

• Internal enterprise systems;
• Access through the official product or certified inference providers;
• Commercial products below the MAU and revenue thresholds above.

From a compliance standpoint, enterprises should complete a license review before adopting the model and confirm whether their business model approaches either threshold. For projects that may grow into large platform-scale products over time, it is advisable to assess attribution and additional-agreement risks early.

Deployment Guidance: Key Steps from Selection to Production

For teams planning to run this model in production, the following path is a useful reference.

1. Inference architecture selection. Given the parameter scale, a distributed inference setup is recommended. Experts can be sharded across accelerator cards or nodes, combined with tensor parallelism and pipeline parallelism. For long-context workloads, also evaluate how well the target inference framework supports sparse attention.

2. Memory and VRAM planning. 2.8 trillion parameters imply an enormous full weight footprint. Calculate memory requirements based on your precision strategy before deployment. Common approaches include low-bit quantization, on-demand expert weight loading, and offloading some layers to system memory. Each option trades precision against throughput, so choose based on task-level results.

3. Multimodal pipeline design. If your business involves image input, design the preprocessing pipeline in advance. Image size, resolution, and compression quality all affect final quality and inference latency. Compare preprocessing options against a standard evaluation set.

4. Monitoring and cost tracking. Because MoE activation patterns depend on input content, actual compute costs fluctuate with traffic. Log the number of activated experts, inference time, and resource consumption per request to support later capacity planning and cost optimization.

5. Compliance and license review. As noted above, assess whether your actual business scale triggers the supplemental license clauses. Platforms planning to offer API services should complete a license assessment before commercial launch to avoid compliance issues as they scale.

A Pragmatic View on Technology Selection

Ultra-large open-source models give the industry more options, but parameter count should never be the sole selection criterion. Enterprises should weigh task fit, inference cost, deployment complexity, license boundaries, and their teams’ engineering capabilities.

For small teams that want to validate the model quickly, start with the official inference API or certified inference services. Once the use case is clear and call volumes are stable, evaluate whether private deployment is warranted. For data-sensitive, compliance-heavy scenarios, plan for private deployment from the start.

Looking further ahead, the pace of model openness is accelerating and architectural evolution is far from over. Understanding the technical core — mixture of experts, sparse attention, long context, and multimodality — matters more than chasing any single version. Only by building transferable evaluation and deployment capabilities can teams keep the technical initiative as models continue to upgrade.