Trillion-Parameter MoE Model Open-Source Practice: Architecture, Evaluation, and License Design

1. Background: A New Phase for Ultra-Large Open-Source Models

Over the past two years, the capability frontier of open-source large models has been pushed steadily upward. From early hundred-billion-parameter dense models to hundred-billion-scale Mixture of Experts (MoE) architectures, and now to ultra-large models with total parameters approaching the three-trillion mark, the open-source ecosystem is undergoing a shift from “usable” to “genuinely useful,” and from “catching up” to “running in parallel.”

Three key drivers underlie this shift:

• Improved architectural efficiency: MoE and sparse attention allow a large parameter count to no longer imply high inference cost. A model can retain enormous knowledge capacity while activating only a small fraction of parameters per computation.
• Mature engineering capabilities: Distributed training, weight sharding, and quantized inference have matured enough to make open-sourcing and deploying trillion-parameter models feasible.
• Evolution of licensing strategy: More teams are adopting licenses close to permissive open-source terms, while using a small number of additional clauses to protect commercial interests and lower compliance barriers for enterprise users.

This article focuses on an open-source MoE model with 2.8 trillion total parameters, systematically covering the engineering practice points of such ultra-large models across architecture design, capability evaluation, and license terms.

2. Architecture: MoE + Sparse Attention

2.1 The Basic Logic of Mixture of Experts

The core idea of Mixture of Experts (MoE) is “divide and conquer”: splitting the feed-forward network into multiple expert sub-networks, with each token routed to only a few experts during inference. As a result:

• Total parameters can be very large, carrying broad knowledge;
• Activated parameters stay relatively low, controlling the compute cost of a single inference pass.

This model has 2.8 trillion total parameters but activates only about 104 billion per inference. Its compute consumption is roughly equivalent to a hundred-billion-scale dense model, while its knowledge capacity is far greater. For enterprises, this “high capacity, low activation” property directly determines unit inference cost.

2.2 From Full Attention to Sparse Attention

Compared with the previous generation in the same series, the most notable architectural change is the replacement of the attention layer: the “full attention” used in the previous 24 layers is changed to “sparse attention.”

The computational complexity of full attention grows quadratically with sequence length, so overhead rises sharply in long-context scenarios. Sparse attention works by having each token select only the most relevant tokens to attend to. In this model, each token selects the 2,048 most relevant tokens.

The benefits of this design appear in two areas:

1. More compute-efficient long-context processing: The longer the sequence, the more pronounced the advantage of sparse attention over full attention;
2. Context window expandable to 1 million tokens: The model supports a context window of up to one million tokens and natively supports multimodal input, including image understanding.

It is worth emphasizing that sparse attention does not simply “discard information”; it retains key dependencies through relevance filtering. In engineering practice, the quality of such mechanisms depends heavily on the design of routing and selection strategies, which is also a key variable in long-text task performance during evaluation.

2.3 Architecture at a Glance

Dimension	Configuration
Architecture type	Mixture of Experts (MoE)
Total parameters	2.8 trillion
Activated parameters per inference	~104 billion
Context window	1 million tokens
Multimodal	Native support for image understanding
Attention mechanism	Sparse attention, 2,048 relevant tokens selected per token

3. Capability Evaluation: General Intelligence and Terminal Tasks

3.1 General Intelligence Index

On a third-party evaluation organization’s general intelligence index, the model scores 46.5. For comparison, the highest-scoring comparable open-source model previously scored 32.6, roughly 14 points lower.

A 14-point gap is uncommon on aggregate leaderboards like the general intelligence index, typically indicating simultaneous progress across multiple capability dimensions rather than a single-point breakthrough. For developers, the reference value of such leaderboards lies in “horizontal positioning”—they help judge roughly which capability tier a model occupies, but cannot replace self-built evaluations for specific business scenarios.

3.2 Terminal Task Capability

Another evaluation organization independently tested a batch of models using the Terminal-Bench 4.0 benchmark, which examines an AI’s ability to execute tasks in a computer terminal environment. This model scored 63.64%, ranking third among 59 models, behind two closed-source models, with another closed-source model ranking fifth.

Terminal task tests emphasize “hands-on” operation: the model must understand instructions, plan steps, execute operations, and handle errors in a real command-line environment. This capability is closely related to scenarios such as automated operations, script generation, and CI/CD assistance, making it directly relevant to developers.

3.3 Cost Efficiency

Beyond scores, cost is another key dimension. This model costs about $4.57 per question on average, while the top-ranked closed-source model requires about $13.20, nearly three times as much.

Looking at “score” and “cost” together yields a more complete conclusion: on the terminal task dimension, this model reaches a level close to leading closed-source models at about one-third of the cost. For teams that need to call models at scale for automated tasks, this cost difference translates directly into monthly bills.

Practical recommendation: When selecting a model, build a small evaluation set from real tasks in your own business and record both “pass rate” and “cost per task,” rather than relying solely on public leaderboard rankings.

4. License Design: Permissive Core with Two Additional Restrictions

4.1 Permissive Core Terms

The main body of the model’s license follows permissive open-source terms: individuals and companies may use, modify, and redistribute it freely, including for commercial purposes. For the vast majority of enterprises, this means it can be brought into production without legal approval.

4.2 Two Additional Restrictions

Beyond the permissive core, the license adds two restrictions:

Restriction 1: Large vendors selling model APIs

If a company or its affiliates are selling model APIs and the group’s total revenue exceeds $20 million in any consecutive 12-month period, a separate agreement with the model provider is required before commercial use.

Restriction 2: Super apps

If a product built on this model exceeds 100 million monthly active users or $20 million in monthly revenue, the model name must be prominently displayed in the interface.

4.3 Practical Scope of the Restrictions

The coverage of these two restrictions is actually quite limited:

• Internal enterprise use: Not restricted;
• Use through official products: Not restricted;
• Use through certified inference service providers: Not restricted;
• Requiring a separate agreement: Essentially only large vendors selling APIs;
• Super apps: Only need to display attribution in the interface.

From a compliance perspective, this “permissive core + targeted additional clauses” design essentially seeks a balance between open source and commercial protection: it lets small and mid-sized developers and enterprises adopt the model with zero barrier, while preserving room for negotiation with the largest commercial users.

5. Engineering Deployment Considerations

5.1 Choosing a Deployment Form

For trillion-parameter MoE models, there are three common deployment paths:

1. Self-hosted inference clusters: Suitable for enterprises with strong infrastructure capabilities and hard requirements for data isolation. Key considerations include memory capacity, expert parallelism strategy, and communication overhead.
2. Managed inference services: Suitable for teams that want rapid validation without bearing operational costs. Confirm whether the provider is an officially certified channel to ensure license compliance.
3. Hybrid deployment: Route sensitive tasks to self-hosted infrastructure and elastic demand to managed services, balancing compliance and cost.

5.2 Inference Cost Optimization

Even with activated parameters already controlled at the hundred-billion level, there is still room to optimize inference cost:

• Quantization: Use low-bit quantization within acceptable accuracy ranges to reduce memory usage;
• Batching: Merge requests to improve throughput and spread the per-call cost;
• Cache reuse: For long-context scenarios, reuse already-computed key-value caches;
• Routing tuning: Monitor load balancing across expert routing to avoid hot experts slowing overall throughput.

5.3 Adapting to Long Context and Multimodal Input

A million-token context window opens new possibilities for document analysis, codebase understanding, and similar scenarios, but also introduces new engineering challenges:

• Input preprocessing requires more refined chunking and retrieval strategies to avoid “filling the context while drowning the key information”;
• Multimodal input requires the data pipeline to support image encoding and alignment;
• During evaluation, distinguish between “can fit” and “can use well”—the latter depends on prompt design and task decomposition.

6. Selection Reference: How to Evaluate Such Models

Overall, when evaluating an ultra-large open-source model, consider the following dimensions:

• Capability fit: Public leaderboards are for reference only; the core is a self-built business evaluation set;
• Cost structure: Look not only at per-call price but also at actual overhead in long-context, multimodal, and other scenarios;
• License compliance: Confirm whether your usage form falls within the scope of the additional clauses;
• Deployment feasibility: Assess the cost, timeline, and operational complexity of self-hosting versus managed services;
• Ecosystem maturity: Support for inference frameworks, quantization tools, and fine-tuning approaches.

7. Conclusion

The open-sourcing of trillion-parameter MoE models marks a step forward for the open-source ecosystem across three dimensions simultaneously: parameter scale, architectural efficiency, and license design. Sparse attention makes long context more affordable, MoE allows large capacity and low activation to coexist, and the “permissive core + targeted additions” license provides clearer boundaries for commercial use.

For developers, what truly matters is not the parameter count itself, but how these design choices translate into usable capability, controllable cost, and a predictable compliance path. When adopting such models, validate results with small-scale business evaluations first, then make the final decision based on deployment form and cost model.