Open-Sourcing Ultra-Large Sparse Models: A New Foundation for Long-Horizon AI Agents

From Scale to Efficiency: The Paradigm Shift of Sparse Architectures

In the evolution of large language models, the tension between parameter scale and inference cost remains a core challenge. When traditional dense models scale up their total parameters, every forward pass requires the activation of all parameters, leading to a linear increase in inference overhead alongside model size. However, a recently open-sourced model boasts a total parameter scale of 2.8 trillion but only requires 1.04 billion active parameters for a single inference pass. This striking ratio is underpinned by its Mixture-of-Experts (MoE) architecture.

Structurally, the model does not simply stack expert modules; instead, it deeply integrates MoE with sparse attention mechanisms. Specifically, its internal architecture comprises 24 transformer layers utilizing Gated DSA (Dynamic Sparse Attention) and 69 transformer layers employing Delta Attention. The former does not treat all historical tokens equally during attention computation; rather, it dynamically filters and selects the 2,048 most relevant historical tokens for the current generation step. This design mirrors the human strategy of reading long documents: instead of reviewing the entire text word by word, we quickly locate the key paragraphs.

From a computational complexity perspective, traditional full attention mechanisms operate at an O(n²) time complexity, where n is the sequence length. When the context window expands to 1 million tokens, this complexity generates astronomical computational volumes. By constraining the focus to a fixed-size subset, sparse attention reduces the computational complexity from being quadratically dependent on sequence length to approximately linear. This allows the model to maintain long-context capabilities while keeping inference latency under control.

Notably, this sparsification is not a “compromise” that sacrifices model performance. Conversely, in long-horizon code agent tasks—where the model must locate defects across thousands of lines of code, understand inter-module dependencies, and execute cross-function modifications—precise attention focusing is often more critical than “equal-treatment” full attention. The technical insight behind this architectural choice is that for long-sequence scenarios, selective forgetting is just as crucial as selective attention.

Large-Scale Joint Reinforcement Learning: From Generation to Decision-Making

Following the pre-training phase, the model underwent a highly distinctive stage: joint reinforcement learning covering hundreds of downstream task domains. This training strategy fundamentally differs from conventional Supervised Fine-Tuning (SFT).

The goal of standard SFT is to have the model mimic the answers in the training data, meaning its learning ceiling is constrained by the quality of the annotated data. Reinforcement Learning (RL), however, optimizes decision-making policies through trial and error and reward signal feedback. Joint RL extends this mechanism to hundreds of different tasks simultaneously. This means that during each parameter update round, the model must balance reinforcement signals across diverse domains such as code repair, web retrieval, CAD design, and office document processing.

This training approach directly yields a significant improvement in model generalization. Consider a practical scenario: when the model is tasked with “fixing a concurrency security issue in an open-source project,” it must first identify the problematic code, design a fix, generate a patch, and verify its correctness. This sequence involves multiple foundational capabilities, including code understanding, pattern matching, logical reasoning, and code generation. If a model is only fine-tuned via RL for a single task, it often exhibits “imbalances” when facing composite tasks—performing exceptionally well in one phase but failing to maintain a smooth overall workflow. Joint RL bridges this gap, endowing the model with greater robustness in cross-task collaboration.

At the execution level, the model offers three tiers of thinking budgets: low, high, and max. This design reflects a pragmatic consideration of the balance between inference cost and task complexity. Simple Q&A scenarios can be met with the low budget, whereas complex long-horizon agent tasks, such as drafting financial research reports, require the max thinking budget, granting the model sufficient computational resources for multi-step reasoning and tool invocation.

Long-Horizon Agent Capabilities: From Single-Turn Dialogue to Multi-Step Task Closure

Traditional large language models are primarily applied in a “single-turn Q&A” format—a user makes a request, the model generates a response, and the interaction concludes. The “long-horizon agent capability” emphasized by this new model represents a fundamental breakthrough in this paradigm.

The core characteristics of a long-horizon agent involve multi-step decision-making and tool usage. Take terminal operation tasks as an example: the model must understand user intent, plan operational steps, execute commands, observe results, adjust strategies based on those results, and continue until the goal is achieved. Official evaluation data shows that this model scored 65.7 on the Terminal-Bench 4.0 benchmark and 51.7 on the SWE Marathon v1.1 for repository-level tasks. These scores indicate that the model can not only generate isolated code snippets but also independently locate issues, implement fixes, and pass test verifications within complex code repository environments.

The foundation of this capability lies in the model’s effective management of “context accumulation.” In multi-turn tool invocation scenarios, each interaction generates new contextual information—operation results, error messages, intermediate files, etc. Amidst the continuous injection of this information, the model must maintain consistent tracking of the global goal to avoid losing its direction as steps progress. To achieve this, official guidelines recommend retaining the complete thought process and tool invocation messages in actual deployments, rather than merely keeping a dialogue summary. This engineering detail highlights the strict requirement for historical information fidelity in long-horizon agent tasks.

The model also demonstrates differentiated capabilities in knowledge work scenarios. Faced with a financial research task, the model must not only retrieve web information but also cross-verify multi-source data, assess information credibility, and generate a structured analysis report. A score of 92.1 on the BrowseComp benchmark indicates that the model possesses a high proficiency in complex information retrieval and synthesis. Furthermore, a score of 64.4 on OfficeQA Pro v2 validates its ability to understand and manipulate office documents (such as spreadsheets and presentations).

Engineering Deployment Essentials: Quantization, Inference Frameworks, and Licensing

For developers, a model’s practical usability depends not only on its theoretical capabilities but also on deployment convenience and integration costs. This model features several engineering decisions worth noting.

First is the MXFP4 weight quantization technology. This technology compresses model weights from traditional FP16/BF16 formats into a 4-bit floating-point format, significantly reducing the model file size. For an ultra-large model with 2.8T total parameters, weight quantization is a critical measure for lowering deployment barriers—it makes running the model on memory-constrained GPU clusters feasible. However, quantization inevitably introduces some precision loss. Developers must balance model capacity against inference quality, typically determined through A/B testing in specific business scenarios.

Second is inference framework compatibility. The model supports current mainstream open-source inference frameworks as well as OpenAI/Anthropic-compatible APIs. This means developers can use their existing calling interfaces and simply redirect the model endpoint to the new deployment to achieve a smooth tech stack transition. When implementing multi-turn tool invocations, developers must ensure that the API request format includes the complete interaction history, rather than just passing the current turn’s user message.

Finally, there are licensing considerations. The model is released under a proprietary license, rather than mainstream ones like Apache 2.0 or MIT. This implies that there may be specific restrictions regarding commercial applications, model modifications, or secondary distribution. Developers are advised to carefully review the original license text during the project planning phase. For scenarios integrating the model’s capabilities into proprietary commercial products, licensing compliance assessments should be prioritized.

Future Outlook: The Evolution Path of Ultra-Large Sparse Models

The release of this model reveals several important trends in large language model development. First, “large but sparse” has become a mainstream direction for breaking through performance bottlenecks—using MoE architectures to unify massive parameter scales with acceptable inference costs. Second, the weight of reinforcement learning in general model training is increasing, evolving from an “auxiliary optimization tool” in the post-training phase to a “core methodology” that shapes model capabilities. Third, long-horizon agent capability has emerged as a new standard for measuring a model’s practical value. In real-world business scenarios, a model’s value lies not just in “knowing the answer,” but in “being able to execute the task.”

Naturally, this technical route faces challenges. The dynamic token selection mechanism of sparse attention may lead to information omission in tasks requiring global information perception (such as a comprehensive compliance review of a legal contract). Distributed inference deployment for ultra-large parameter models imposes higher demands on infrastructure bandwidth and scheduling capabilities. The potential “load imbalance” among different expert modules also requires continuous training strategy optimization to mitigate.

For teams focused on AI infrastructure, the architectural approach and training methodology of this model offer significant reference value. It proves the technical feasibility of replacing “computational brute-forcing” with “perceptual efficiency”—investing computational resources in the wrong places is far less effective than executing precise computations at critical junctures. This concept applies not only to large model R&D but also provides a reference for broader AI system design. With the open-sourcing of this model, community developers will be able to fine-tune, distill, and innovate upon it, collectively driving long-horizon agent technology toward greater practicality.