Architecture and Training Analysis of a 2.8T Open-Source Sparse MoE Model for Long-Horizon Agents

1. Background: New Requirements for Long-Horizon Agent Foundation Models

Over the past two years, the competitive focus of large language models has gradually shifted from “single-turn QA quality” to “multi-step task completion.” A Long-Horizon Agent refers to a model’s ability to maintain goal alignment across dozens or even hundreds of steps—repeatedly calling external tools, reading intermediate results, revising its own plans, and ultimately delivering a verifiable output. Typical scenarios include:

• Repeatedly executing commands, reading errors, and modifying scripts in a terminal until the task is complete;
• Cross-file bug fixing and regression testing in large code repositories;
• Multi-hop information retrieval and research report generation using search tools;
• Structured processing, data verification, and batch generation of office documents.

The demands these tasks place on foundation models differ significantly from traditional conversational models:

1. Context Length and Attention Efficiency: Trajectories for long-horizon tasks easily reach hundreds of thousands of tokens. If attention computation grows quadratically with length, both inference cost and latency become unacceptable.
2. Adjustable Inference Budgets: Simple steps do not require deep reasoning, while critical decision points demand repeated deliberation. Models should support allocating compute based on task complexity.
3. Stability of Tool Calling: Multi-turn tool calls must retain the complete history of reasoning and tool return messages; otherwise, the model will lose its state.
4. Controllable Deployment Costs: Sparse activation and low-bit quantization are prerequisites for making ultra-large parameter models practically servable.

The open-source model discussed in this article represents a systematic engineering attempt to address this background: a 2.8T total parameter, 104B activated parameter MoE architecture, supporting million-token contexts and native image-text inputs, optimized for long-horizon agent scenarios through sparse attention modifications and joint reinforcement learning post-training.

2. Architecture Analysis: From Dense to Gated Sparse Attention

2.1 Fundamental MoE Trade-offs

This model employs a Mixture-of-Experts (MoE) architecture: 2.8T total parameters, but only about 104B activated parameters per token. The core motivation of this design is to decouple model capacity from per-token inference computation. Parameter count determines knowledge capacity and expressive power, while activation volume dictates the floating-point operation cost per token. For a general-purpose agent foundation model that needs to cover hundreds of downstream domains, large capacity plus sparse activation is currently the most cost-effective route.

However, MoE comes with trade-offs: expert routing introduces additional communication and scheduling overhead, and load balancing during training requires specialized handling. Therefore, what truly determines whether an MoE model can be deployed in production is often not the number of experts itself, but the engineering details of the attention and routing layers.

2.2 Gated Sparse Attention Modifications

The most critical architectural change in this version compared to its predecessor is the replacement of the attention layers. It replaces Gated Multi-head Latent Attention (Gated MLA) in certain layers with Gated Sparse Attention (Gated DSA), combined with other efficient attention mechanisms to form a hierarchical hybrid attention structure.

The sparse attention mechanism can be summarized as follows: when computing attention for a given token, it does not review the entire history. Instead, it dynamically selects a fixed-size subset of historical tokens to participate in the computation. In this implementation, the selection size is 2048 historical tokens.

The engineering significance of this design includes:

• Computational complexity transitions from “growing with length” to “approximately constant”: Whether the context is 100,000 or 1 million tokens, the attention computation per token is capped within a fixed budget.
• Information filtering is handled by a learned selection mechanism: The model learns to judge which historical tokens are important during training, rather than relying on manually set sliding windows or fixed spans.
• Backward compatibility with previous architectures: Only a subset of layers is replaced rather than a complete overhaul, facilitating the reuse of existing training and inference infrastructure.

It must be emphasized that sparse attention is not “lossless.” It fundamentally trades selection precision for computational feasibility. Therefore, in actual evaluations, performance on long-context retrieval tasks (such as web search and multi-hop QA) often better reflects the success of this modification than short-text tasks.

2.3 Million-Token Context and Native Image-Text Input

Million-token context is fundamental infrastructure for long-horizon agents. It means the model can accommodate within a single session:

• Complete multi-turn tool call trajectories;
• Key files from medium-sized code repositories;
• Full text of long documents or research reports;
• Mixed image-text UI screenshots and design drafts.

Native image-text input allows the model to directly understand visual information such as UI screenshots, charts, and CAD sketches, reducing the intermediate steps and information loss of “OCR first, understand later.” This is particularly critical for terminal operations and 3D design tasks.

3. Training Methodology: Joint Reinforcement Learning and Low-Bit Quantization

3.1 Joint RL Across Hundreds of Downstream Domains

The model’s post-training does not optimize for a single task. Instead, it employs joint reinforcement learning covering hundreds of downstream domains. The underlying logic is that agent capabilities are highly transferable: terminal operations, code repairs, retrieval verification, and document processing all share the same foundational “observe-plan-act-verify” loop. Rather than training a separate expert model for each domain, multi-domain mixed reward signals are used to teach the model general task progression strategies.

The engineering challenges of joint RL primarily lie in three areas:

1. Reward Design: Success criteria vary drastically across domains. Code tasks can use test pass rates, retrieval tasks can use answer correctness, while office document tasks often require a combination of rule validation and human-defined standards.
2. Sample Ratio: The difficulty and sample volume across domains are highly imbalanced, requiring dynamic adjustment to prevent one domain from dominating the gradient.
3. Training Stability: Rewards in long-horizon tasks are sparse and highly delayed, making credit assignment difficult. This typically requires combining process rewards or intermediate signals.

3.2 MXFP4 Weight Quantization

The released weights are in MXFP4 format. MXFP4 is a Microscaling low-bit floating-point format. Its approach groups weights into small blocks where each block shares a single scaling factor, and elements within the block are represented using 4 bits. Compared to traditional integer quantization, microscaling formats generally preserve numerical distribution characteristics better at extremely low bit widths.

The direct benefits for deployment teams include:

• Significantly reduced memory footprint and weight bandwidth, making single-machine deployment of ultra-large parameter models more realistic;
• Compatibility with quantization operators in mainstream inference frameworks, facilitating integration into existing serving pipelines.

It is worth noting that inference accuracy with quantized weights may differ from previous high-precision weights. It is recommended to perform a regression validation on your own evaluation datasets in real-world business scenarios.

4. Inference Control: Three-Tier Thinking Budgets

The model provides three thinking budgets at the inference stage: low, high, and max, allowing callers to allocate compute based on task complexity. This is a design highly tailored for agent scenarios:

• low: Suitable for high-certainty steps like format conversion, field extraction, and simple tool calls;
• high: Suitable for code modifications and information summarization requiring moderate reasoning;
• max: Suitable for critical decision points like complex bug localization, multi-hop retrieval planning, and CAD parameter derivation.

In an agent pipeline, a sensible approach is to dynamically select the tier per step rather than uniformly applying the highest tier across the entire trajectory. For example:

def pick_budget(step_type: str) -> str:
    if step_type in ("parse_output", "format_json", "simple_lookup"):
        return "low"
    if step_type in ("code_patch", "summarize_docs"):
        return "high"
    return "max"  # Complex planning, root cause analysis, multi-hop retrieval

This hierarchical control significantly reduces the average inference cost of long-horizon tasks without sacrificing quality at critical nodes. Officially published benchmark scores default to the max tier, which indicates that actual business costs and latency will be noticeably lower than benchmark configurations.

5. Capability Coverage and Benchmark Performance

5.1 Code Development and 3D Creation

This model is explicitly designed for long-horizon code agents, software engineering, and 3D design tasks. Official evaluations cover terminal operations, code repairs, and CAD generation, supporting multi-step task completion combined with tools.

Representative data points from public benchmarks include:

• Terminal-Bench 4.0: 65.7
• SWE Marathon v1.1: 51.7
• DeepSWE v1.1: 75.5
• BenchCAD (with tools): 92.7

These numbers convey more insight than a single aggregate score: terminal operations and long-horizon software engineering tasks are significantly harder than single-point code generation. This is because such tasks test not only coding ability but also error recovery, state tracking, and tool usage standards. BenchCAD achieving a high level with tools indicates the model’s viability in the combined pipeline of “visual understanding + parametric generation + tool validation.”

5.2 Agents and Knowledge Work

In the direction of agents and knowledge work, the model supports web search, financial research, office document processing, and tool calling. Relevant scores include:

• BrowseComp: 92.1
• OfficeQA Pro v2: 64.4

BrowseComp leans towards multi-hop web retrieval and information integration; the high score indicates strong capabilities in the “search-filter-cross-validate” pipeline. OfficeQA Pro v2 involves office document understanding and QA; the score of 64.4 suggests there is still room for improvement—complex document structures, mixed tables and text, and cross-page references remain persistent challenges.

It should be noted that some evaluations utilized agentic frameworks (like AI coding tools) as the execution shell. This means the scores reflect not only the model itself but also the framework’s contributions to tool orchestration and context management. Readers should account for this variable when making horizontal comparisons.

6. Deployment and Integration Essentials

6.1 Inference Framework and API Compatibility

The model releases MXFP4 weights, supports mainstream high-performance inference frameworks, and simultaneously provides both OpenAI and Anthropic API-compatible styles. The practical implications for engineering teams are:

• Existing agent code written against these API specifications can be migrated at a low cost;
• The coexistence of both styles makes it easy to reuse agent components from different sources within the same system.

6.2 State Management for Multi-Turn Tool Calling

The official guidance explicitly states: multi-turn tool calls must retain the complete history of reasoning and tool call messages. This is the most easily overlooked aspect of deploying long-horizon agents, and the most prone to causing performance collapse.

Common anti-patterns include:

• Retaining only tool return results while discarding the model’s reasoning process;
• Truncating history in a way that breaks the pairing between tool calls and their returns;
• Rebuilding sessions between turns, causing the model to lose task state.

The correct approach is to maintain the integrity of the message sequence. Only when the context approaches its limit should a strategy of retaining the head and tail + structured summarization be applied, ensuring that tool call IDs and their return results always appear in pairs:

[
  {"role": "user", "content": "..."},
  {"role": "assistant", "content": "...", "tool_calls": [{"id": "c1", "name": "search", "arguments": "{...}"}]},
  {"role": "tool", "tool_call_id": "c1", "content": "..."},
  {"role": "assistant", "content": "Next step plan based on the above results..."}
]

6.3 Licensing Considerations

The model’s weights are released under a proprietary license, rather than a permissive open-source license. Before integrating it into production systems, enterprises should have their legal and compliance teams confirm the scope of use, distribution conditions, and commercial restrictions to avoid risks arising from misunderstandings of the license terms.

7. Practical Recommendations and Selection Criteria

Based on the analysis above, here are several recommendations for engineering teams:

1. Segment scenarios before deciding on thinking tiers. Do not uniformly apply the max tier across the entire chain. Allocating compute based on step type can significantly reduce costs.
2. Treat context management as a first-class citizen. Most long-horizon agent failures occur not because the model lacks capability, but because of state loss or misaligned tool messages.
3. Validate quantized weights on your own evaluation sets. Public benchmarks differ from real-world business distributions. The precision of MXFP4 weights needs to be regressed using actual tasks.
4. Distinguish model capabilities from framework capabilities. Benchmark scores include the contributions of the agent framework. Performance may change if the framework is swapped; conducting ablation studies is recommended.
5. Pay attention to licensing and compliance. A proprietary license does not mean it cannot be used, but the boundaries must be clearly defined.

8. Conclusion

The technical narrative of this 2.8T total parameter, 104B activated parameter open-source sparse MoE model is not simply about “larger parameters,” but rather a comprehensive set of配套 designs built around long-horizon agents: utilizing Gated Sparse Attention to compress the computational cost of million-token contexts to acceptable levels, employing joint RL across hundreds of domains to shape general task progression capabilities, offering three-tier thinking budgets to hand compute allocation control to the caller, and using MXFP4 quantization to lower deployment barriers.

For developers, what is truly worth noting is not a single benchmark score, but the direction revealed by this architecture and training methodology: the competition in long-horizon agents is shifting from “can the model answer correctly” to “can the system run stably to completion.” The latter is a joint result determined by architecture, training, inference control, and engineering integration. A gap in any of these areas will be amplified in long-horizon tasks. Understanding this is far more valuable than memorizing any single benchmark number.