Technical Analysis of Open-Source Long-Horizon Agent Models: 2.8T Sparse Architecture and Reinforcement Training Practices

1. Background: New Architecture Requirements for Long-Horizon Agent Tasks

As AI agents transition from single-turn Q&A to multi-step execution, models must handle continuously evolving contexts rather than isolated prompts. Typical scenarios include terminal operations, code repair, 3D design generation, web retrieval, financial research, and office document processing. These tasks share three characteristics: long time spans, frequent tool calls, and complex intermediate states.

In long-horizon tasks, models must remember hundreds or even thousands of rounds of historical information while continuously making decisions. The computational complexity of traditional dense attention mechanisms grows quadratically with sequence length, quickly hitting memory and latency bottlenecks in large-scale models. Therefore, the new generation of agent-oriented LLMs is shifting toward a combination of sparse attention and Mixture-of-Experts (MoE) architectures, aiming to maintain long-context capabilities while controlling inference costs.

Additionally, long-horizon agent tasks typically involve tool calls. The model must not only generate natural language but also output structured actions or dynamically adjust strategies based on tool results. This raises the bar for reinforcement learning in the post-training phase: reward signals must cover cross-step trajectories rather than just evaluating single-turn response quality.

2. Overall Architecture: MoE Backbone and Activated Parameter Design

This open-source model utilizes a Mixture-of-Experts (MoE) architecture with approximately 2.8T total parameters and 104B activated parameters. The core idea of MoE is to split the feedforward network into multiple expert modules, activating only a subset during each inference. This expands model capacity without significantly increasing computational overhead.

Regarding routing strategy, the model dynamically assigns input tokens to different expert combinations. For long-horizon agent tasks, this design offers two main benefits: first, different subtasks can activate distinct expert paths (e.g., code generation and document analysis may rely on different parameter subspaces); second, during multi-turn interactions, the model can adjust internal routing based on context type changes, improving adaptability to heterogeneous information.

Notably, having 104B activated parameters means the active computation per forward pass is much smaller than the total parameter count. This allows deployments to choose higher-precision activation representations while storing the majority of inactive parameters in low-precision formats, striking a balance between inference accuracy and memory footprint.

3. Sparse Attention Mechanism: Evolution from Gated MLA to Gated DSA

The model’s primary attention mechanism change replaces some original Gated MLA layers with Gated DSA (Dynamic Sparse Attention) layers, combined with other sparse attention layers. The core feature of DSA is dynamically selecting a fixed number of tokens (2048) from historical tokens to participate in attention computation, rather than computing against the entire historical sequence.

The advantages of this dynamic sparse selection include:

1. Computational Controllability: Regardless of how long the context extends, the number of tokens involved in a single attention computation remains stable. Inference latency does not linearly degrade with history length.
2. Information Filtering: The selection process in DSA is a learnable policy. During training, the model learns to judge which historical tokens are most critical to the current decision, mimicking human “selective recall.”
3. Long-Horizon Memory Support: By continuously selecting key historical fragments across multi-turn interactions, the model maintains effective utilization of early information within a 1-million-token context, preventing attention dilution caused by overly long sequences.

From an engineering perspective, replacing Gated MLA with Gated DSA is not a simple layer-type switch. It requires redesigning positional encoding, cache structures, and training objectives for the selection strategy. For developers, understanding this mechanism is crucial for planning KV cache memory layouts and retrieval strategies when designing custom inference frameworks.

4. Joint Reinforcement Learning Post-Training: Covering Hundreds of Domains

The model’s post-training employs a joint reinforcement learning strategy covering hundreds of downstream domains. This design goal ensures the model generates consistent decision logic across different types of long-horizon tasks, avoiding overfitting to a single domain.

The implementation of joint reinforcement learning typically includes the following key components:

• Reward Modeling: Defining quantifiable success criteria for different tasks. For instance, terminal operation tasks reward completing specific steps; code repair tasks use test pass rates as the core metric; 3D design tasks combine geometric validity with design constraint satisfaction.
• Trajectory-Level Optimization: Unlike single-step supervised fine-tuning, joint RL optimizes complete interaction trajectories. The model must not only produce correct single outputs but also learn to self-correct when tool calls fail and maintain action consistency across multi-step dependencies.
• Inter-Domain Balancing: Joint training across hundreds of domains requires addressing data distribution imbalance. A common practice introduces dynamic weights during the sampling phase to maintain model sensitivity to low-frequency, high-value tasks.
• Tiered Thinking Budgets: The inference phase offers low, medium, and high thinking budgets, allowing users to adjust the model’s internal reasoning depth based on task complexity. This is typically achieved by controlling the number or length of intermediate reasoning steps, providing flexible options for scenarios with different latency sensitivities.

5. Weight Quantization and Deployment: Practical Significance of MXFP4

Releasing model weights in MXFP4 format is a noteworthy technical choice. MXFP4 is a microscaling format that organizes floating-point numbers into blocks, with each block sharing a single scaling factor, thereby retaining a wider dynamic range at 4-bit precision.

Compared to common integer quantization, microscaling formats are more robust to outliers. In MoE models, weight distributions across different expert layers can vary significantly. Fixed-range integer quantization can easily introduce large errors in certain layers. By using block-level scaling, MXFP4 allows each parameter group to adjust its representation range based on its own distribution, reducing performance loss from quantization.

Regarding inference engine adaptation, the model supports mainstream high-performance inference frameworks and provides standardized API interfaces for easy integration with existing agent development toolchains. Enterprise users should focus on the following factors when deploying long-horizon agent models:

Deployment Considerations Checklist:
1. KV Cache Management: Sparse attention makes cache growth controllable, but historical token retention strategies must be set reasonably.
2. Multi-Turn Tool Calls: Complete thinking history and tool call messages must be retained to avoid context breaks caused by truncation.
3. Quantization Precision Validation: MXFP4 weight performance on long-horizon tasks requires regression testing specific to business scenarios.
4. Thinking Budget Adaptation: Latency-sensitive scenarios should prioritize low budgets; complex reasoning scenarios should switch to high budgets.

6. Performance Evaluation: Capability Boundaries from Benchmarks

Officially released benchmark scores reflect the model’s capability boundaries from different perspectives. Scoring 65.7 in terminal operation evaluations indicates the model can perform multi-step operations in command-line environments, though there is still significant room for improvement. A score of 51.7 in the software engineering marathon suggests that continuous planning and self-correction capabilities in long-cycle code tasks need strengthening.

The model achieved high scores in deep code understanding, web retrieval, and tool-assisted 3D design generation tasks, indicating strong performance when supported by specific tools. Scoring 64.4 in office document Q&A reflects practical utility in handling structured business documents, though complex format understanding remains a challenge.

It should be emphasized that these scores were achieved using the highest thinking tier by default, and some evaluations utilized external agent frameworks as execution carriers. Therefore, actual business performance will be influenced by toolchain quality, prompt design, and thinking budget settings. Enterprises are advised to build internal evaluation sets using their own business data for independent validation during model selection.

7. Licensing and Compliance Reminders

This weight release uses a proprietary license rather than a common permissive open-source protocol. This means users must carefully review the license terms regarding commercial deployment, secondary distribution, and model modifications to ensure compliance for their specific use cases.

Enterprise technical teams need to establish clear compliance review processes when selecting open-source models. Key verification points should include: whether commercial use is allowed, whether derivative works must be open-sourced, whether model outputs are restricted to specific domains, and whether there are geographical or industry limitations. These terms directly impact whether the model can enter production environments.

Overall, the architectural design, training strategies, and engineering deployment practices of this model provide a valuable technical reference for long-horizon agent tasks. When drawing insights from its approach, developers can explore the combination of sparse attention and MoE, as well as the organization of joint RL across cross-domain tasks. More importantly, combining these insights with their own business scenarios to establish a comprehensive evaluation system—spanning model selection, performance validation, and compliance review—is essential for translating large-scale model capabilities into stable production value.