A 2.8-Trillion-Parameter Open Source MoE Model: Architecture, Capabilities, and License Boundaries

The ceiling for open source large language models has been raised again. A release that originally promised time-limited weight availability has now delivered on schedule: a mixture-of-experts (MoE) model with 2.8 trillion total parameters has been uploaded, under a license that largely follows permissive open source conventions and allows free commercial use for most organizations. This is not just another parameter-scale milestone. It is also a practical sample of three things at once: whether open source commitments are credible, whether extremely large models are engineering-feasible, and where commercial license boundaries can be drawn.

This article examines the release from four angles: architecture, capabilities, cost, and compliance.

Why 2.8 Trillion Parameters Deserves Attention

Parameter count alone is not the goal. What it determines is threefold: the upper bound of model capacity, the engineering complexity of training and inference, and the range of viable deployment strategies.

Under a dense architecture, 2.8 trillion parameters would mean every forward pass has to traverse all weights. Memory consumption and compute cost would be almost prohibitive. At this scale, the practical path is almost inevitably MoE: parameters are partitioned into multiple expert sub-networks, and each token is routed to only a small number of them. This decouples total parameter count from per-inference activated parameter count.

The key figures for this release are:

• Total parameters: 2.8 trillion
• Activated parameters per inference: approximately 104 billion
• Architecture: MoE
• Context window: 1 million tokens
• Modality support: native multimodal with image input

What these numbers mean is that the model has an extremely large knowledge capacity, while the actual compute cost of a single inference step is controlled at the level of a hundred-billion-parameter model. In other words, it attempts to balance capacity and usability, which is currently the central engineering challenge for very large models.

It is worth emphasizing that the activated parameter ratio is only about 3.7% (104 billion out of 2.8 trillion). This level of sparsity places demanding requirements on routing algorithms, load balancing, and expert-parallel communication. If routing becomes uneven, some experts become overloaded while others sit idle, and real-world throughput can fall far below theoretical expectations. For that reason, evaluating this kind of model requires looking beyond parameter count to whether the routing strategy and parallel deployment plan are mature.

Sparse Attention: The Economics of Long Context

Compared with the previous open source model in the same family, the most substantial architectural change is in the attention layer: 24 layers that originally used full attention have been replaced with sparse attention.

Full attention complexity grows quadratically with sequence length. At a context scale of one million tokens, attention computation becomes the dominant bottleneck, and KV cache memory consumption expands rapidly. The idea behind sparse attention is that each token no longer attends to the entire history. Instead, it attends only to a small subset of the most relevant tokens.

The configuration described for this implementation is that each token selects only the 2048 most relevant tokens for attention computation. The engineering value shows up in three ways:

1. Compute savings: Attention cost is tied to the number of selected tokens rather than total sequence length, significantly reducing marginal cost in long-context scenarios.
2. Controllable memory: The KV cache only needs to retain the selected subset, reducing memory pressure for long-context deployment and increasing the number of concurrent sessions a single machine can support.
3. A trade-off on long-range dependencies: Fixed-window sparse selection sacrifices some very long-range precise dependencies. However, for mainstream long-context tasks such as document Q&A, codebase understanding, and long multi-turn conversations, it usually delivers order-of-magnitude efficiency gains at a relatively small accuracy cost.

From an engineering perspective, this kind of change is not a free lunch. Index selection for sparse attention itself introduces compute overhead, and different tasks define relevance differently. A fixed selection strategy may miss important content in certain retrieval-intensive tasks. In real deployments, teams should evaluate against their own workloads rather than relying only on general benchmark scores.

A simplified illustration of the sparse attention selection logic can be expressed as follows:

def sparse_attention(query, key, value, top_k=2048):
    # Compute relevance scores between the query and each key
    scores = relevance_score(query, key)      # [seq_len]
    # Keep only the top_k most relevant positions
    indices = top_k_indices(scores, k=top_k)  # [top_k]
    selected_k = key[indices]
    selected_v = value[indices]
    # Apply standard attention over the sparse subset
    return standard_attention(query, selected_k, selected_v)

In actual implementations, index selection usually operates on blocks rather than individual tokens to match GPU memory access patterns and improve throughput. The snippet above is illustrative only and does not represent any specific product interface or parameter.

Benchmark Results: Reading Beyond the Scores

A third-party comprehensive intelligence evaluation gave the model a score of 46.5, roughly 14 points higher than the previous highest-scoring open source model at 32.6. In an independent terminal-operation benchmark that tests a model’s ability to complete multi-step tasks in a command-line environment, the model scored 63.64%, placing third among 59 evaluated models, behind two leading closed-source models and ahead of another closed-source model.

These numbers should be interpreted with three cautions:

• Validity of horizontal comparison: Composite index benchmarks usually weight multiple task categories, and the weighting scheme can significantly affect rankings. A higher score indicates strong overall capability in the open source tier, but it does not mean the model leads in every sub-task.
• Differences across task types: Terminal-operation benchmarks are closer to agentic execution scenarios. A high score suggests strong capability in multi-step tool calling, state tracking, and error recovery. These skills are highly valuable for automation, code repair, and similar use cases.
• Reproducibility: Open weights mean the scores can be independently reproduced and verified, an advantage closed-source models do not provide. Enterprises should still run small-scale evaluations on their own business data rather than accepting public leaderboards at face value.

Inference Cost: A More Important Metric Than Scores

Public data shows that the average per-task cost for this model on terminal-operation tasks was about 4.57 US dollars, compared with roughly 13.20 US dollars for a leading closed-source model. The former is about one third of the latter.

This comparison reveals an often overlooked fact: the economics of a very large model depend not only on per-token price, but also on the total number of tokens required to complete a task. If a model can finish a task with fewer interaction turns, shorter reasoning chains, and fewer retries, total task cost can still be lower even if its unit price is not.

For enterprise cost evaluation, a useful framework includes:

1. Per-task cost = average input tokens × input unit price + average output tokens × output unit price + expected retry and failure cost.
2. Concurrency capacity: On a given hardware cluster, the number of requests processed per second directly determines cost per unit of time.
3. Marginal cost in long-context scenarios: The efficiency benefit of sparse attention becomes more pronounced in long-document tasks and should be measured separately.
4. Self-hosting vs. managed services: Open weights allow self-hosted inference. For sustained high-load scenarios, self-hosting is usually more economical. For low-frequency or bursty loads, managed services are more flexible.

It should be noted that the dollar figures above come from specific benchmarks and specific configurations. They cannot be directly extrapolated into general pricing. Real-world cost is affected by batch size, quantization strategy, hardware type, KV cache management, and many other factors.

License Breakdown: Permissive Core with Two Commercial Boundaries

The license design is one of the most important parts of this release for legal and architecture teams to study together.

The main body follows permissive open source conventions: individuals and companies may use, modify, redistribute, and use the model commercially for free. On top of that, two additional restrictions are defined.

Restriction 1: Large-scale model API sellers.
If a company or its affiliates provide model API services, and the corporate group’s total revenue in any consecutive 12-month period exceeds 20 million US dollars, the company must sign a separate agreement with the releasing party before commercial use.

Restriction 2: Super-scale applications.
If a product built with this model exceeds 100 million monthly active users, or generates more than 20 million US dollars in monthly revenue, the product interface must display the model name prominently.

The license also defines exemptions: internal company use, or use through official products and certified inference service providers, is not subject to the two restrictions above.

From an engineering and compliance perspective, this design can be interpreted as follows:

• Fully friendly to most small and medium-sized organizations and developers: The revenue and user thresholds are far above the scale of most teams. Ordinary commercial use, fine-tuning, and redistribution carry essentially no additional obligations.
• Precisely targeted at two high-value scenarios: Large-scale API resale involves direct commercial competition, while super-scale applications involve brand exposure. These conditions are a typical open-source-plus-commercial-boundary design.
• Compliance recommendations: Before adoption, organizations should complete three steps: confirm the revenue scope of themselves and their affiliates, confirm whether product MAU or monthly revenue reaches the thresholds, and confirm whether the usage is internal or certified service use versus external API selling. The license text should be included in the legal review process, and the compliance conclusion should be recorded in architecture documentation.

Engineering Deployment Recommendations

For teams preparing to evaluate or deploy this model in production, the following practical steps are recommended:

1. Build a capability baseline first: Create a 50–200 item evaluation set from your own business data, covering core task types. Compare against the current solution with A/B testing, and avoid being misled by public leaderboards.
2. Validate sparse attention task fit: Focus on long-document retrieval, cross-paragraph reasoning, and similar scenarios to confirm whether the sparse selection strategy causes recall loss.
3. Measure real throughput: Test throughput and memory usage on the target hardware under different batch sizes and context lengths, then estimate per-task cost accordingly.
4. Design fallback and routing strategies: A very large model should not carry all traffic. Route simple tasks to smaller models and complex tasks to the large model, keeping overall cost under control.
5. Front-load compliance: Document the key license points and a compliance checklist in the code repository, avoiding retroactive compliance issues as the business grows.

Conclusion

The core significance of this open source release is not that it refreshed a parameter number. It is that it provides three verifiable answers at once: very large MoE models can be trained and deployed through practical engineering; sparse attention makes million-token context economically feasible; and an open source license can remain broadly permissive while still setting clear, predictable boundaries for commercial use. For developers and enterprises, the real work is to re-measure these capabilities inside their own business coordinate system. Public scores belong to others. Cost and compliance belong to you.