A 2.8T-Parameter Open-Source MoE Model: Architecture, Benchmarks, and License Design

1. The Event Itself: A Weight Release That Landed on Schedule

Whether an open-source commitment is actually honored has long been an implicit measure of a model team’s engineering organization. Releasing model weights involves aligning training framework versions, defining tensor-parallel sharding strategies, ensuring inference-side operator compatibility, finalizing license text, and preparing model cards and evaluation reproduction scripts. A delay in any one of these links causes a slip. So when a team publicly commits to “releasing the full weights by a certain date” and actually uploads them to a public model hosting platform within that window, the signal this sends matters more than the parameter count—it shows the team’s training-to-release pipeline has reached repeatable, industrial-grade maturity.

The core facts of this release can be summarized in four points:

• Scale: Mixture-of-Experts (MoE) architecture, 2.8 trillion total parameters, with only about 104 billion activated per inference pass;
• Capabilities: 1 million token context window, native multimodal image input support;
• Architectural evolution: Compared with the previous generation of the same model family, 24 layers that previously used full attention were replaced with a sparse attention scheme;
• License: An MIT-style permissive core, with two additional restrictive clauses.

Below, we break down the technical meaning and engineering implications of these facts layer by layer.

2. MoE Architecture: What 2.8 Trillion Total Parameters Actually Means

2.1 Separating Total Parameters from Activated Parameters

For readers unfamiliar with MoE, the most common misunderstanding is: “Doesn’t 2.8 trillion parameters mean inference costs are unbearable?” The answer is no, and the key lies in distinguishing total parameters from activated parameters.

An MoE model consists of a number of expert subnetworks plus a router (gating) network. Each token is routed to only a few experts per layer, so the floating-point operations in a single forward pass are determined mainly by the activated parameters, not the total. This release has 2.8 trillion total parameters and roughly 104 billion activated, an activation ratio of about 3.7%. This means:

• Memory footprint is still dominated by total parameters. Even if only a subset of experts is activated, all expert weights must reside in memory or be swapped in efficiently. The 2.8 trillion parameters impose a very real demand on VRAM and host memory capacity;
• Compute cost is close to that of a hundred-billion-scale dense model. Compute budgets can be estimated at the activated-parameter scale rather than the total-parameter scale;
• Routing quality becomes a critical variable. The more experts there are, the harder it is to train routing stably, and problems such as load-balancing loss and expert collapse become more prominent.

2.2 Introducing Sparse Attention

Compared with the previous generation, the most substantive architectural change here is replacing 24 full-attention layers with sparse attention. The compute and memory overhead of full attention grows quadratically with sequence length, and at a 1 million token context this overhead quickly becomes a bottleneck.

The basic idea of sparse attention is that each token no longer computes attention against all tokens in the sequence, but only against a small, most relevant subset. In this scheme, each token selects only about the 2,048 most relevant tokens for computation. The direct benefit is a significant reduction in compute and memory in long-context scenarios; the cost is that it places demands on the accuracy of the “relevance selection” algorithm—if the selection strategy fails to retain genuinely important long-distance dependencies, quality on long-context tasks will degrade.

In engineering practice, this kind of sparse attention usually requires a matching kernel implementation to realize its performance gains; otherwise, the extra overhead of sparse indexing can offset the compute savings. When evaluating such models, then, it is worth looking beyond the architecture description to whether the inference framework provides corresponding optimized operators.

3. Benchmark Performance: Composite Intelligence Index vs. Terminal Task Capability

3.1 Composite Intelligence Index

On a third-party evaluation organization’s composite intelligence index, this model scored 46.5, while the highest-scoring comparable open-source model previously scored 32.6—a gap of about 14 points. This number should be viewed rationally:

• A composite index is a weighted aggregate, and different organizations use different weightings, so scores cannot be compared directly across organizations;
• A 14-point gap on an aggregate index falls in the significant range, usually indicating improvements across multiple sub-items rather than an incidental lead on a single dimension;
• But an aggregate score cannot substitute for validation on specific tasks, especially tasks closely tied to business scenarios.

3.2 Terminal Task Benchmark

Another evaluation organization independently tested a set of models on a terminal task benchmark, assessing their ability to complete real work in a command-line terminal environment. This model scored 63.64%, ranking third among 59 evaluated models, with the top two both being closed-source models and another closed-source model ranking fifth.

The terminal task benchmark is characterized by being close to real engineering scenarios: the model must execute commands in a shell environment, read output, and adjust strategy based on feedback, testing multi-turn tool calling, error recovery, and state management. A high score on this kind of benchmark is often more valuable to developers than pure text question-answering benchmarks.

3.3 The Cost Dimension

More noteworthy is the cost data: this model costs about $4.57 per question on average, while a top-ranked closed-source model requires about $13.20—nearly three times as much. This indicates that, on a per-task cost basis, an MoE architecture with roughly 104 billion activated parameters holds a structural advantage over large dense models.

Note, however, that per-question cost depends heavily on task type, number of retries, and context length. The figures above should be understood as averages under a specific benchmark and should not be extrapolated directly to your own business. To assess real costs, consider a small-scale estimate along these lines:

# Illustrative cost-estimation framework (not a real API; shows the calculation logic only)
def estimate_cost(avg_input_tokens, avg_output_tokens, n_calls,
                  price_per_input_token, price_per_output_token):
    input_cost = avg_input_tokens * n_calls * price_per_input_token
    output_cost = avg_output_tokens * n_calls * price_per_output_token
    return input_cost + output_cost
# Key variable: in long-context scenarios, input_tokens may grow by one to two orders of magnitude
# The benefit of sparse attention shows up mainly in this term

4. License Design: A Permissive Core Plus Two Targeted Restrictions

This is the part of the release that corporate legal and technical decision-makers should read most carefully.

4.1 MIT-Style Core

The license core follows MIT-style permissive terms: individuals and companies may use, modify, and redistribute it freely, including for commercial purposes. This means the way most small and mid-sized teams use it carries no additional constraints—internal deployment, embedding into proprietary products, and modified closed-source distribution are all permitted under the core terms.

4.2 Two Additional Restrictions

Restriction one: aimed at large-scale API resellers. If a company or its affiliates provide a model API service externally and the group’s total revenue over any consecutive 12-month period exceeds $20 million, a separate agreement with the model provider is required before commercial use.

The target of this restriction is clear: it constrains entities that “wrap the model as an API and sell it externally” and that have substantial revenue scale. The design logic is that the model provider does not oppose commercialization, but wants to preserve room for separate negotiation in cases of resale where model capability is the core commodity and the scale is large enough.

Restriction two: an attribution requirement aimed at super apps. If a product built on this model exceeds 100 million monthly active users, or $20 million in monthly revenue, the model name must be prominently displayed in the product interface.

This restriction involves no fee, only attribution—essentially a brand-visibility requirement. For the vast majority of products, reaching 100 million MAU or $20 million in monthly revenue is itself rare, so the practical trigger probability is low.

4.3 Exempted Scenarios

Two categories of scenarios are not subject to the above restrictions:

• Internal company use: including internal tools and internal process automation, with no external distribution;
• Use through the model provider’s official products and its certified inference providers: that is, access through official channels.

Taken together, the only parties that genuinely need a separate agreement are essentially large vendors selling API access at scale; super-app scenarios generally only require interface attribution. For most developers and small-to-mid-sized businesses, the practical binding force of this license is close to zero.

4.4 A Checklist for Enterprise Adoption

Before adding such a model to your stack, it is advisable to perform compliance and engineering checks in the following order:

1. Determine the usage form: internal use, embedding in a proprietary product, or providing an external API? The three forms fall under different clauses;
2. Revenue and affiliate scope: if external API service is involved, total revenue must be calculated on a “company and its affiliates, consecutive 12 months” basis, not just a single legal entity;
3. Product-scale projection: assess whether monthly active users and monthly revenue might reach the attribution threshold, and reserve a place for labeling in the product design in advance;
4. Check redistribution terms: when distributing modified versions, the license and copyright notice must be retained, a general requirement of MIT-style agreements;
5. Confirm the inference channel: if going through a certified inference provider, confirm whether its certification status remains valid.

5. Practical Deployment Guidance

5.1 Memory and Deployment

2.8 trillion total parameters mean this is not a model that fits on a single GPU. Real deployment requires multi-machine, multi-GPU sharding, the communication overhead of expert parallelism, and KV cache management under long contexts. If sparse attention is used, you also need to confirm whether the inference framework provides a matching operator implementation.

5.2 The Real Benefit of Long Context

A 1 million token window has clear value in document analysis, codebase comprehension, and long-conversation memory, but note:

• Window length does not equal effective usable length; validate performance on “needle-in-the-middle” style retrieval tasks with your own data;
• Cost grows non-linearly with long context; sparse attention mitigates this but cannot eliminate it;
• Run separate latency and cost benchmarks for long-context tasks rather than relying only on short-task averages.

5.3 Reproducing Evaluations

Third-party aggregate scores should serve only as an initial screen. Before adoption, build a small evaluation set from your own business samples covering four scenario types—tool calling, multi-turn error correction, long-document question answering, and image input—and cross-check against the model’s public benchmark performance. If the gap is large, determine whether it is a prompt-adaptation issue or a capability-boundary issue.

6. Summary

The core message of this release can be reduced to three points. First, the combination of very large-scale MoE with sparse attention pushes the capability ceiling of open-source models into a new range. Second, the separation of activated and total parameters gives it a structural advantage in per-task cost relative to large dense models. Third, the “permissive core plus two targeted restrictions” license design keeps commercial friendliness while reserving room for negotiation and attribution in large-scale API resale and super-app scenarios.

For developers, the real task is not to be persuaded by parameter counts, but to follow the sequence of “determine usage form → compliance check → small-scale evaluation → cost estimation” and use your own data to verify whether it actually solves your problem. Parameters are the threshold, benchmarks are a reference, and cost and licensing are constraints—all three together are needed to make a responsible adoption decision.