Deep Dive into a 2.8T MoE Architecture: Long-Horizon Agent Capabilities Explained

1. Background: What Do Long-Horizon Agents Demand from Base Models?

Over the past year, AI Agents have transitioned from demo-oriented single-turn Q&A to deliverable engineering forms: terminal operations, repository-level bug fixes, multi-step web retrieval, and document pipeline processing. The common denominator of these tasks is not “high single-shot generation quality” but rather “not collapsing during long-horizon interactions.” A complete code repair task might involve dozens of tool calls, hundreds of file reads, and command executions. Within an extremely long context, the model must continuously maintain goal alignment, state tracking capabilities, and error recovery capabilities.

This introduces three hard constraints: context length (must fit the entire trajectory), inference cost (long contexts mean an explosion in attention computation), and decision quality (tool selection and parameter construction at each step must be reliable). A recent new version released by an open-source model directly addresses these three points through engineering trade-offs. Rather than reiterating release notes, this article deconstructs the technical logic of such “long-horizon Agent base models” across four dimensions—architecture, training, evaluation, and deployment—to serve as a reference for developers during model selection and when building custom inference services.

2. Architecture: Dual Sparsification with MoE Capacity and Sparse Attention

1. MoE: Trading Total Parameters for Capacity, Controlling Cost via Activated Parameters

This version employs a Mixture-of-Experts (MoE) architecture with a total parameter scale of 2.8T and activated parameters of approximately 104B per forward pass. The implications of these numbers must be broken down:

• Total parameters determine “knowledge capacity.” A larger expert pool allows the model to accommodate more domain-specialized sub-networks, reducing mutual interference across heterogeneous tasks like code, math, multilingual, and scientific reasoning.
• Activated parameters determine “per-token cost.” During inference, each token is routed to only a few experts. The actual computational load and memory bandwidth pressure are primarily dictated by the activated portion, not the total parameters.

For engineering teams, this means deployment costs cannot be estimated based on total parameters. While the weight files for 2.8T parameters are massive, the computational overhead is closer to that of a 100B-scale dense model. The real bottlenecks lie in memory capacity and the communication overhead of expert parallelism, rather than FLOPs.

2. Sparse Attention: Compressing O(n²) to an Acceptable Range

If a 1-million-token context uses standard full attention, both computation and memory footprint will grow quadratically with sequence length, making it engineering-infeasible. This version tackles this by mixing two types of attention layers:

• Gated Sparse Attention: Replaces some of the Gated Multi-Head Latent Attention layers from previous versions. The core concept is that each query position no longer computes attention against all historical tokens. Instead, it dynamically selects about 2048 historical tokens for computation. The selection process itself is handled by lightweight routing, avoiding full-scale scoring.
• Delta Attention (Linear Complexity): Stacked alternately with sparse layers to further reduce the average computational density over long sequences.

The combined effect is: locally precise, globally reachable. Sparse layers ensure that critical historical information (like early task goals or key file paths) is not diluted, while linear layers maintain a “coarse-grained perception” of the overall trajectory at a low cost. This design perfectly fits Agent scenarios—in tool call trajectories, the nodes that require precise backtracking are often just a few critical ones, not all intermediate outputs.

A noteworthy engineering detail is that the historical token selection strategy in sparse attention significantly impacts long-context recall rates. When performing long-document Q&A or analyzing ultra-long codebases, developers should verify through actual tasks “whether critical information falls within the selected 2048 tokens,” rather than blindly trusting the nominal context length.

3. Training: Joint Reinforcement Learning and Tiered Thinking Budgets

1. Joint Reinforcement Learning Across Hundreds of Downstream Domains

This version utilizes joint reinforcement learning (RL) covering hundreds of downstream domains during the post-training phase. The term “joint” offers two main values: first, avoiding capability bias, meaning strengthening coding capabilities should not come at the expense of document understanding or numerical reasoning; second, promoting cross-domain transfer, where information extraction capabilities in web retrieval can transfer to office document processing.

The takeaway for users is that these models typically excel at “multi-step, verifiable” tasks. Tasks with clear success signals (test passes, command return codes, CAD geometric validity) benefit the most from RL. In contrast, the subjective quality of open-ended creative generation still relies heavily on base capabilities and prompt design.

2. Three-Tier Thinking Budgets: Making Inference Cost a Tunable Parameter

On the inference side, three thinking budgets are provided: low, high, and max. This is a highly practical engineering interface:

• low: Suitable for high-frequency, low-complexity calls, such as intent classification, simple field extraction, and formatting conversion. Latency-sensitive services should use this tier by default.
• high: Suitable for medium-complexity, multi-step tasks, such as single-file code modifications and routine retrieval Q&A.
• max: Suitable for long-horizon Agent main loops, complex defect localization, and planning tasks requiring multi-path exploration. Official benchmark data defaults to this tier.

In practice, a tiered routing approach is recommended: dynamically switch tiers based on the task phase within the Agent framework. For example, using max for planning, high for execution, and low for result validation strikes a good balance between quality and cost. It is worth noting that increasing the thinking budget does not yield linear returns. Beyond a certain threshold, it may lead to “over-thinking,” causing verbose trajectories. Threshold calibration should be performed on proprietary datasets.

4. Capabilities and Benchmark Evaluation

The officially released evaluations cover three categories of scenarios, defaulting to the max tier, with some tasks completed in conjunction with Agent frameworks like intelligent programming tools:

Evaluation Dimension	Representative Task	Score
Terminal Operation	Command-line multi-step tasks	65.7
Long-Horizon Software Engineering	Repository-level bug fixes	51.7
Code Repair	Defect localization and patch generation	75.5
Web Retrieval	Complex information lookup and integration	92.1
CAD Generation (with tools)	Parametric modeling tasks	92.7
Office Document Q&A	Table and document understanding	64.4

Three key takeaways when interpreting this data:

First, there is a massive difference between using tools and not using tools. CAD tasks scored 92.7 when combined with tools, indicating that the model’s value lies more in “driving tools” than “directly producing the final artifact.” During selection, emphasis should be placed on evaluating the model’s tool call accuracy and parameter construction correctness, rather than just raw generation quality.

Second, the long-horizon software engineering score is relatively modest (51.7). This relates to the inherent difficulty of the task—repository-level fixes require cross-file understanding, dependency analysis, and regression verification, representing the current ceiling of Agent capabilities. For enterprise deployment, it is recommended to start with “single-file, test-covered” tasks and gradually expand the scope.

Third, benchmark scores cannot be directly extrapolated to private scenarios. A model’s performance on public benchmarks is significantly influenced by prompt templates, tool frameworks, and retry strategies. It is advisable to conduct small-sample validations on proprietary task sets, focusing on failure modes rather than average scores.

5. Deployment and Integration: Quantized Weights and Multi-Turn Tool Call Pitfalls

1. MXFP4 Quantization and Inference Engine Adaptation

The weights are released in MXFP4 format, a 4-bit microscaling floating-point quantization scheme that drastically reduces memory footprint while keeping precision loss controlled. Mainstream inference engines already provide support and expose interfaces compatible with common API specifications, facilitating integration into existing Agent frameworks.

Key considerations during deployment:

• Quantization is not cost-free. 4-bit quantization may experience numerical stability issues in long-context and high-concurrency scenarios. It is recommended to enable corresponding precision validation on engines like vLLM and compare the accuracy degradation against your own evaluation sets before and after quantization.
• Communication costs of expert parallelism. In multi-GPU deployments, the MoE structure generates All-to-All communication for expert routing. If deployed across nodes, network bandwidth can become a bottleneck. It is recommended to prioritize expert partitioning within a single node utilizing high-bandwidth interconnects.

2. Multi-Turn Tool Calls Must Retain Complete History

This is a detail that is easy to get wrong: in multi-turn tool call scenarios, the complete history of thinking content and tool call messages must be retained. The reason is that the model’s routing and attention selection depend on the reasoning trajectory within the context. If a framework truncates intermediate thinking steps to save tokens, the model loses the context of “why this tool was called,” leading to subsequent step drift.

Corresponding engineering practices:

# Example: Retain full trajectory when constructing multi-turn messages
messages = []
messages.append({"role": "user", "content": task})
while not done:
    resp = model.chat(messages, thinking_budget="max")
    # Crucial: Append thinking content and tool calls without truncation
    messages.append(resp.full_message)  # Contains reasoning + tool_calls
    if resp.tool_calls:
        results = execute_tools(resp.tool_calls)
        messages.append({"role": "tool", "content": results})

Simultaneously, a trajectory length limit and compression strategy should be set: when the context approaches the window limit, prioritize compressing the raw content returned by tools (such as large files or long web pages), rather than compressing the model’s own reasoning trajectory.

3. Licenses Require Separate Evaluation

This version’s weights are released under a specialized license, rather than a permissive general open-source license. Enterprise users must have their legal teams confirm the licensing boundaries for commercial use, secondary distribution, and model distillation. Do not assume it is covered under standard open-source licenses.

6. Conclusion: Who Is It For and How to Use It?

Overall, open-source models featuring “ultra-large MoE + sparse attention + tiered thinking budgets” are suitable for the following scenarios: long-horizon code Agents requiring private deployment, internal knowledge retrieval and document pipelines, and engineering teams with strict data residency requirements. Their advantages lie in balancing capacity with long context, alongside tunable inference costs. Their challenges involve deployment thresholds (memory and expert parallelism), quantization precision validation, and trajectory management on the Agent framework side.

A recommended “three-step” deployment approach: First, run through a single-step tool call chain using the low-tier thinking budget to verify interface compatibility and stability. Second, perform single-file code repairs and document Q&A at the mid-tier to establish a quality baseline. Finally, introduce the high-tier and long-horizon loops, paired with trajectory compression strategies, to gradually expand task scope. Throughout this process, use proprietary evaluation sets as the ultimate benchmark, treating public benchmark scores merely as a reference for the upper limit of capabilities.