Open-Source Milestone: 2.8-Trillion-Parameter Mixture-of-Experts Model Released with Open Weights

Section 1: A Delivered Promise—Trust and Technical Execution in Open Source

In the open-source AI community, honoring a announced weight-release date often tests a lab’s engineering discipline and ecosystem credibility just as much as the model’s benchmark scores. Recently, a leading AI lab met its public commitment: within eleven days of the official model announcement, the complete weight files were uploaded to a major open-source model hosting platform. The symbolic and technical significance of this move is equally important—developers worldwide can now instantly download, deploy, fine-tune, and integrate the model into production environments.

As a prominent overseas AI researcher noted on social media, the punctuality of Chinese open-source labs in releasing model weights has become a stable expectation in the industry. This “follow-through” engineering discipline, especially during rapid model iteration cycles, is fundamental to building developer trust.

Section 2: Architecture Analysis—The Sparse Activation Revolution of 2.8 Trillion Parameters

2.1 Scaling MoE to the Extreme: Total vs. Activated Parameters

This open-source model uses a Mixture-of-Experts (MoE) architecture with 2.8 trillion total parameters. However, during a single forward pass, the system activates only 104 billion parameters. This design philosophy decouples the model’s “knowledge capacity” from its “computational cost” via sparse activation.

• Knowledge Capacity Dimension: 2.8 trillion parameters represent a vast potential combination space of patterns, facts, and reasoning paths learned from massive corpora. In theory, this supports broader coverage of long-tail knowledge, more specific domain terminology, and complex cross-domain reasoning chains.
• Computational Efficiency Dimension: By activating only 104 billion parameters per inference and leveraging MoE’s routing mechanisms, the model maintains high capacity while keeping single-inference FLOPs within acceptable bounds. Compared to dense models of similar total parameter size, inference costs can be reduced by more than an order of magnitude.

2.2 Attention Mechanism Upgrade: From Full Attention to Sparse Attention

Compared to its predecessor, the most significant architectural change is replacing the 24 Transformer layers that used full attention with implementations based on sparse attention. This approach draws directly from published sparse attention research, with the core logic as follows:

• Token Selection Mechanism: During each attention computation step, the model no longer calculates attention weights for all tokens in the sequence. Instead, a learnable routing strategy selects only the 2048 most relevant tokens for each query token. This threshold is carefully tuned to balance information preservation and computational savings.
• Long-Context Advantage: For a one-million-token context window, full attention scales quadratically in complexity. Sparse attention reduces this to near-linear scaling. This means lower latency and reduced memory usage when parsing long documents, analyzing multi-turn conversation histories, or processing repository-scale code inputs.

2.3 Native Multimodal Support and Context Window

The model natively supports image inputs, enabling joint vision-language understanding. Architecturally, feature sequences from the visual encoder are directly concatenated with text token sequences before entering the Transformer backbone, requiring no additional adaptation modules. The context window spans 1 million tokens, enough to hold the full content of several thick books or tens of thousands of lines of code. This configuration provides a practical foundation for applications such as long-document Q&A, complex codebase analysis, and multi-document comparative summarization.

Section 3: Performance Evidence—Open-Source Models Take the Lead

3.1 Composite Intelligence Index: A Generational Gap

In the latest version of the Artificial Analysis Intelligence Index from a third-party evaluation body, this model scored 46.5 points. For reference, the highest score among previously open-source models was 32.6 points. This increase of roughly 14 points represents a relative improvement of nearly 44%. This leap is not confined to a single dimension but reflects a weighted average across multiple metrics, including knowledge Q&A, logical reasoning, mathematical problem-solving, code generation, and instruction following.

3.2 Terminal Agent Capability: Ranked in the Top Three

Another evaluation body, Vals AI, independently tested the model’s ability to perform real-world tasks in a computer terminal environment using the Terminal-Bench 4.0 benchmark. Tasks included file operations, command-line invocation, script writing, and software configuration. Among 59 tested models, the model ranked third with a score of 63.64%, trailing only two closed-source flagship agent models. It outperformed a dedicated agent version of a well-known closed-source LLM. This result indicates that on key agentic capabilities—tool calling, state tracking, and multi-step task planning—the model is competitive with leading closed-source systems.

3.3 Unit Cost Efficiency: Pareto Improvement in Performance and Cost

In the Terminal-Bench 4.0 evaluation, the average economic cost per task for this model was only USD 4.57 (calculated using standard market API pricing). In comparison, the top-ranked closed-source agent model on the same leaderboard incurred a per-task cost of USD 13.20. This means achieving a similar task completion rate at roughly one-third of the cost. For high-frequency or batch-processing scenarios, this cost difference translates directly into significant total cost of ownership advantages.

Section 4: A Refined Open-Source License—Balancing Permissiveness and Constraints

Beyond the technical specifications, the license terms of this release deserve close examination. The main license framework follows the widely recognized MIT License, allowing individuals and companies to use, modify, redistribute, and commercially use the model free of charge. On top of this open base, the copyright holder adds only two constraints targeting specific scenarios.

4.1 Revenue Threshold for API-Reselling Enterprises

The first restriction targets companies whose primary business model is reselling the model’s API. Specifically, if a company or its affiliates are selling API services based on this model and the group’s total revenue exceeds USD 20 million in any consecutive 12-month period, a separate commercial agreement with the copyright holder is required before commercial use.

This clause is designed with a degree of precision:

• Exempts SMEs: API service providers with annual revenue below USD 20 million are unaffected, ensuring early-stage startups are not hindered.
• Exempts Internal Use: Using the model internally for product development, data analysis, or process automation does not trigger this clause.
• Clear Commercial Boundary: Only enterprises earning substantial revenue directly from reselling model capabilities need to enter a commercial negotiation. This effectively creates a sustainable “open-source core + commercial license” ecosystem model.

4.2 Brand Attribution for Super-Apps

The second restriction applies to end-user applications: if a product built on this model reaches more than 100 million monthly active users or generates over USD 20 million in monthly revenue, the application interface must display a clearly visible attribution stating that it is built on this open-source model.

This clause is based on the following rationale:

• When a product reaches super-app scale, the foundational contribution of the model deserves brand-level visibility.
• This attribution obligation does not affect functionality or involve revenue sharing—it is purely an acknowledgment of contribution.

4.3 Exemptions and Practical Paths

The following scenarios are completely unaffected by the two restrictions above:

• Internal use within an enterprise for internal systems, employee tools, or private data analysis.
• Accessing the model through the official API or certified inference service providers.
• Academic research, teaching demonstrations, and personal projects.

In practice, only two categories truly require a separate agreement: “scale API resellers” and “products reaching super-app scale.” Overall, the license achieves a relatively delicate balance between openness and commercial protection. It provides a reference model for how future open-source licenses might be structured.

Section 5: Developer Implementation Guide—From Weights to Production

5.1 Model Acquisition and Deployment Options

Weight files are hosted on a mainstream open-source platform and support multiple deployment modes:

• Local Deployment: Use inference frameworks such as vLLM or SGLang to fully exploit the acceleration benefits of sparse attention for long contexts. A GPU cluster with at least 4x A100/H100-class accelerators is recommended, with tensor parallelism and pipeline parallelism enabled.
• Cloud API Access: Integrate quickly into existing business systems by using the official API or interfaces provided by certified inference service providers.

5.2 High-Priority Application Scenarios

Based on the model’s capabilities, the following scenarios are well positioned for early adoption:

# Example: Long-document analysis using sparse attention features
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("path/to/your/weights")
tokenizer = AutoTokenizer.from_pretrained("path/to/your/weights")
# Input a complete annual financial report (~200K tokens)
long_report = load_pdf_text("annual_report.pdf")
# Direct cross-chapter information extraction and logical reasoning
prompt = f"Based on this report, analyze the compound annual growth rate of revenue over the past three years and assess whether the drivers are sustainable.\n\nReport text:\n{long_report}"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=1024)

5.3 Fine-Tuning and Adaptation Strategies

For vertical domains such as legal documents, medical records, or financial research, parameter-efficient fine-tuning methods such as LoRA or QLoRA are recommended:

• Data Preparation: Build 50K–500K domain-specific instruction pairs, truncating inputs to within 32K tokens.
• Training Configuration: Using 8x A100 GPUs, set LoRA rank to 64, learning rate to 2e-4, and train for 3 epochs.
• Evaluation Metrics: Use a domain-specific held-out test set to monitor instruction following accuracy and hallucination rates.

Section 6: Summary and Outlook

This release of a 2.8-trillion-parameter MoE model achieves three things at once. Technically, it demonstrates the scalable replacement of full attention with sparse attention. In terms of performance, it sets a new record on the composite Intelligence Index among open-source models. Ecologically, its carefully designed license provides a reusable legal framework for commercial deployment of open-source models. For developers and enterprises, now is an opportune moment to evaluate integrating this model into their technology stack—its performance is sufficient for demanding agentic tasks and ultra-long context processing, while its cost structure offers clear advantages at this capability level.

As the open-source ecosystem continues to mature, we can expect that releasing model weights will no longer be merely a “publishing event.” Instead, it becomes a comprehensively engineered system—covering architectural innovation, performance validation, license strategy, and deployment tooling in a complete delivery package. This may well be a defining sign that open-source AI is coming of age.