Open-Source LLM Sets New Benchmark: 2.8T-Parameter MoE Architecture Powers Long-Horizon Agent Applications

Introduction: A Paradigm Shift from Model Scale to Task Depth

As the competition among large language models gradually shifts from “parameter stacking” to “capability deepening,” a notable trend has emerged: the pursuit of model size alone is no longer the sole objective. How to achieve efficient inference, long-horizon planning, and complex tool invocation at extremely large parameter scales has become the core challenge for next-generation models. Recently, a model with a total parameter count reaching an astonishing 2.8T level officially released its weights. Its MoE (Mixture of Experts) architecture and breakthrough attention mechanism design set a new technical benchmark for handling long-horizon Agent tasks.

1. Core Architecture Analysis: The Ultimate Balance Between Sparsity and Efficiency

1.1 The “Total vs. Active” Parameter Trade-off Under MoE Architecture

The model adopts a cutting-edge MoE architecture with a total parameter count of 2.8T, yet only 104B activated parameters. This means that in each forward pass, only about 3.7% of parameters participate in computation. The design philosophy is: storing knowledge through massive “expert” subnetworks while dynamically selecting the most suitable expert set for the current input through a gating mechanism, thereby maintaining model capacity while keeping inference compute requirements within the range of feasible physical devices. For developers, this means that even with the capabilities of a trillion-parameter-class model, it can still be deployed on a limited GPU cluster through high-throughput inference frameworks.

1.2 Sparse Attention: The Synergistic Innovation of Gated DSA and Delta Attention

The optimization of the attention mechanism is another highlight of this model. Compared to the previous generation, the model replaces some standard attention layers with Gated DSA (Gated Sparse Attention) layers, combined with 69 layers of Delta Attention. The core logic of Gated DSA is: before attention computation, a learnable gating unit dynamically selects the most critical 2,048 tokens from the historical token sequence to participate in the current computation. This mechanism directly breaks the linear complexity bottleneck of traditional attention, which must scan the entire historical context.

In long-horizon Agent scenarios, this design advantage is significant: when an Agent needs to trace back tool call results or code definitions from tens of thousands of tokens earlier, traditional models often lose critical information due to attention dispersion. Gated DSA, through its gating mechanism, forces the model to focus on core decision nodes, greatly improving information recall precision over long sequences. Complementing the Delta Attention layers that retain the original full context information, this design ensures both deep mining of critical information and maintenance of global context awareness.

2. Training Methodology: Joint Reinforcement Learning and Multi-Dimensional Alignment

2.1 Cross-Domain Joint RL Training Strategy

The model’s training process is not a simple pretraining plus fine-tuning pipeline, but rather a joint reinforcement learning (RL) approach covering hundreds of downstream domains. This method is no longer limited to optimizing a single task benchmark, but instead places heterogeneous tasks such as code generation, web retrieval, mathematical reasoning, and visual understanding into the same optimization space through a unified reward model design.

Training Pipeline Overview:

flowchart LR
    A[Pretrained Base] --> B[Supervised Fine-Tuning]
    B --> C[Joint Reinforcement Learning]
    C --> D[Multi-Task Generalization]
    subgraph C [Joint RL Stage]
        C1[Code Execution Feedback]
        C2[Tool Call Results]
        C3[Human Preference Ranking]
    end
    C1 --> E[Policy Network Update]
    C2 --> E
    C3 --> E
    E --> F[RM Reward Model]

The core value of this strategy lies in breaking the dilemma of “gradient conflicts between tasks” in traditional RL training. By sharing the policy network, the logical chain reasoning ability learned from code tasks can positively transfer to multi-step reasoning tasks in financial research. Meanwhile, MXFP4-format weight quantization is introduced as a training aid, reducing memory footprint without significantly sacrificing precision, making larger-batch RL training possible.

2.2 Dynamic Reasoning Budget Adjustment Mechanism

To accommodate tasks of varying complexity, the model provides three levels of thinking budget adjustment during inference: “low,” “high,” and “maximum.” The essence of this mechanism is controlling the length of the model’s “chain of thought” before generating the final answer:

• Low budget: Suitable for simple knowledge Q&A or format conversion, with extremely fast response times;
• High budget: Suitable for code repair and multi-step logical reasoning, allowing the model to generate longer intermediate reasoning steps;
• Maximum budget: Designed for long-horizon Agent tasks, where the model collaboratively invokes retrieval tools, executes code, and performs multiple rounds of reflection based on results.

This flexible design enables developers to find the optimal balance between API call costs and task precision, without paying maximum compute costs for every request.

3. Capability Boundaries and Application Scenario Deep Dive

3.1 Long-Horizon Code Agent and Software Engineering

Supported by the aforementioned technologies, the model demonstrates strong task decomposition and execution capabilities in software engineering. It scored 65.7 on the Terminal-Bench benchmark and 51.7 on the SWE Marathon benchmark. This is not merely an improvement in code generation ability, but more importantly reflects the model’s autonomous decision-making capability in the closed loop of “understanding repository structure → locating problem modules → generating patches → executing test verification.”

In 3D design scenarios, the model supports native image-text input, can understand annotation information in CAD drawings, and generates corresponding parametric modeling instructions. Its capability is not limited to outputting code, but extends to completing the transformation from design sketches to physical models by combining tool sets.

3.2 Knowledge Work and Agent Collaboration

In office automation and financial research domains, the model natively supports a 1-million-token context window, capable of digesting hundreds of pages of financial report PDFs or historical transaction records in a single pass. Its unique sparse attention mechanism handles such ultra-long texts with ease: despite the extremely long context, the model can precisely locate key financial indicators through gated sparse attention.

Scoring 92.1 on the BrowseComp web retrieval benchmark indicates that the model has developed a mature “retrieve-read-summarize” pipeline—it can autonomously determine when to search, how to construct search keywords, and how to distill the most credible conclusions from conflicting information across multiple sources. This capability is crucial for building automated investment research assistants or intelligent sentiment monitoring systems.

4. Deployment Practices and Toolchain Ecosystem

4.1 Open-Source Weights and Inference Framework Adaptation

The model has released quantized weights based on the MXFP4 format, which can reduce weight storage space by approximately 4x while maintaining equivalent model precision. Official adaptations for multiple mainstream inference engines (such as vLLM and SGLang) are available, along with OpenAI- and Anthropic-compatible API interfaces, greatly reducing migration costs for developers.

Key Deployment Reminders:

• Multi-turn tool call preservation: When performing inference in Agent scenarios, the complete “thinking-tool call-observation” message sequence must be preserved. If only the last user message is retained, the model will lose preceding decision context, causing the tool call chain to break.
• Quantization trade-offs for memory optimization: While MXFP4 weights offer compression advantages, the computational overhead during dequantization must be considered in high-concurrency scenarios. It is recommended to combine KV Cache quantization techniques to achieve end-to-end memory footprint reduction.

4.2 License Compliance Considerations

It is important to note that the model’s weights follow a proprietary license rather than permissive licenses such as Apache 2.0 or MIT. This means developers must carefully review the license terms before commercial use or model redistribution. Enterprise users are advised to introduce legal review processes during the internal testing phase to ensure downstream applications comply with the authorized scope.

5. Technical Comparison and Future Outlook

From benchmark data, the model performs excellently in evaluations oriented toward “real-world task completion,” such as scoring 64.4 on OfficeQA Pro (office document processing) and 92.7 on BenchCAD (CAD generation with tools). This indicates that the value of its reinforcement learning strategy has moved beyond traditional academic benchmarks and is closer to the demands of actual production environments.

As sparse attention mechanisms and joint RL training continue to mature, we have reason to anticipate that large models will evolve in the following directions:

1. Precision in context memory: Gated sparse attention will push models from “remembering everything” to “remembering what matters,” which is expected to reduce inference costs while improving long-horizon task success rates.
2. Multimodal Agent integration: The current model already supports native image-text input. The next stage may incorporate audio and video stream processing into the same framework, enabling more comprehensive environmental perception.
3. Possibilities for edge deployment: Although the 2.8T total parameter scale is difficult to directly port to edge devices, knowledge distillation techniques based on the MoE architecture may give rise to a batch of smaller-parameter models whose capabilities approach the original in specific vertical domains.

Conclusion:
The release of this open-source model is essentially a path exploration of the question “how can large models evolve from chat tools into productivity tools.” It solves the compute challenge of long-horizon dependencies through sparse attention mechanisms, and bridges the gap between knowledge memory and task execution through joint reinforcement learning. For developers, immediately experimenting with its long-horizon Agent performance in their own business scenarios will be the key action to capture this wave of technical dividends.