Technical Analysis and Engineering Practice of the Open-Source MoE Long-Horizon Agent Model

1. Model Positioning and Key Capabilities

A recently released open-weight model, with a total parameter count of 2.8T and 104B activated parameters, adopts a Mixture-of-Experts (MoE) architecture. It is explicitly designed for long-horizon agentic tasks, and its core capabilities can be summarized in three areas:

1. Long Context Handling: Supports a 1-million-token context window, making it suitable for scenarios involving long documents, large codebases, and multi-step task trajectories.
2. Native Multimodal Input: Accepts mixed text and image inputs directly, without relying on an external vision encoder as a bridge.
3. Tool Calling and Agentic Capabilities: Supports multi-round tool invocation, environment interaction, and controllable thinking budgets, adapting to complex agentic workflows.

Positioning-wise, the model is not aimed solely at improving single-turn question-answering quality. Its primary target is long-horizon tasks that require sustained reasoning, repeated tool usage, and continuous state tracking—common in software engineering, code development, 3D design, financial research, and office document processing.

2. Architecture Design: Attention Evolution from MLA to DSA

The most notable architectural change is the replacement of Gated MLA (Multi-head Latent Attention) with Gated DSA (Dense-Sparse Attention) in a subset of layers, which then works together with another group of attention layers to form a hybrid attention structure. Understanding this change requires a closer look at how the two attention mechanisms differ.

2.1 Differences Between MLA and DSA

MLA reduces the memory footprint of key-value (KV) caching by applying low-rank compression. The core idea is to approximate the original key-value matrices with low-rank projections, learning a compressed mapping during training and storing only the low-dimensional representations during inference. This significantly lowers inference costs in long-context scenarios.

DSA takes a different approach: instead of having the current token interact with the full historical token set, it selectively attends to a fixed number of historical tokens. In the case of this released model, DSA selects 2048 historical tokens per step. This sparsification strategy reduces attention complexity from linear growth to an approximately constant level, substantially cutting the computational overhead for long sequences.

2.2 Engineering Significance of Hybrid Attention

The hybrid attention mechanism is not a simple stacking of layers. Different layers carry different functional emphasis: some are responsible for capturing fine-grained dependencies, while others focus on long-range information routing. Combining DSA with Delta Attention represents a pragmatic trade-off between memory efficiency and representational completeness.

From an engineering perspective, this design keeps inference costs manageable even with a 1M-token context. For long-horizon agent tasks that must retain extensive historical trajectories—including reasoning traces, tool outputs, and contextual states—this is a critical advantage.

2.3 Parameter Scale and Activation Efficiency

With 2.8T total parameters and 104B activated parameters, the MoE architecture activates only about 3.7% of the model’s parameters per token. This sparse activation design allows the model to retain a large parameter footprint while keeping inference latency and compute costs within acceptable bounds. For enterprise users planning to deploy a long-context model on limited hardware, this feature lowers the infrastructure barrier.

3. Training Methodology: Joint Reinforcement Learning and Domain Coverage

3.1 Value of Joint Reinforcement Learning

The model undergoes large-scale joint reinforcement learning (RL) post-training on top of its pretraining phase, covering hundreds of downstream domains. Traditional supervised fine-tuning is good at imitating existing behaviors, but for agentic tasks requiring multi-step decisions, labeled supervision often cannot cover all possible interaction paths. RL allows the model to learn from delayed feedback in the environment, thereby acquiring better decision-making policies.

Joint training, in particular, enables the sharing of signals across domains. For example, the strategy of “locate the problem first, then fix it” in code repair shares logical similarities with “retrieve information first, then synthesize” in financial research. Joint training helps the model learn such transferable decision patterns rather than optimizing each domain in isolation.

3.2 Three-Level Thinking Budget Control

The model provides three thinking budget levels—low, high, and max—allowing users to adjust reasoning intensity based on task complexity. The design rationale is straightforward: not every task requires maximum reasoning effort.

• Low: Suitable for simple Q&A, format conversion, and information extraction. Lower latency and reduced resource consumption.
• High: Designed for moderately complex analysis and reasoning tasks, balancing quality and speed.
• Max: Reserved for multi-step complex tasks such as repository-level refactoring, complex research work, and multi-constraint 3D design. Longer inference time but higher output quality.

This tiered mechanism is essentially a compute-resource scheduling strategy. In batch processing environments, requests can be routed based on complexity, ensuring that limited compute is prioritized for high-value tasks.

3.3 Weight Quantization and Training Stability

The model uses MXFP4 weight quantization during training. The primary benefit of FP4 is a significant reduction in memory usage, enabling larger batch sizes and longer context windows. Compared to INT4, FP4 offers better dynamic range, which improves numerical stability for reinforcement learning signals that often exhibit long-tail distributions.

That said, quantization-aware training places higher demands on the training framework and hardware. Handling gradient instability under low-precision conditions requires mature engineering capability—a key reason why such models remain relatively rare.

4. Capability Analysis for Agentic Tasks

4.1 Code Development and Software Engineering

In the context of long-horizon code agents, the model must understand codebase structure, localize issues, plan modifications, execute changes, and verify results. This is far more complex than single-turn code generation. Key required capabilities include:

• Multi-step Task Planning: Decomposing complex requirements into executable subtask sequences.
• Tool Orchestration: Working with terminals, code search, and test execution tools to form a closed feedback loop.
• Long-context Tracking: Maintaining accurate state information across multi-turn interactions.

The official evaluation suite covers terminal operations, code repair, and CAD generation. For code repair, the model must reproduce the issue, identify the faulty code region, propose a fix, and verify that no new issues are introduced—a process that demands both deep reasoning and reliable tool invocation.

4.2 Agentic and Knowledge Work

Knowledge work scenarios include web search, financial research, and office document handling. A common trait across these tasks is that information is scattered across multiple sources, requiring the model to autonomously plan retrieval strategies, filter relevant content, and synthesize conclusions.

A typical financial research task may involve the following steps:

1. Parse the user query to determine required data dimensions.
2. Invoke retrieval tools to fetch relevant reports or datasets.
3. Read and extract key information.
4. Cross-validate data consistency across sources.
5. Produce a structured research summary.

Throughout this process, the model repeatedly invokes tools, judges whether the gathered information is sufficient, and decides whether additional retrieval is needed. These abilities cannot be fully acquired from static datasets; they depend on reinforcement learning training.

4.3 3D Design and CAD Generation

3D design tasks are unique in that outputs must satisfy not only functional requirements but also spatial constraints and geometric validity. The model must iteratively verify the validity of generated results through tool calls and adjust its output accordingly. The official BenchCAD evaluation with tool usage reaches 92.7, indicating that a closed verification loop between tool invocation and output validation yields significant gains in this domain.

5. Deployment and Integration Practice

5.1 Inference Framework Support

The model is released in MXFP4 format, with support already available in major inference engines. For real-world deployment, the following points require attention:

• Memory Planning: Even with quantization, the 2.8T total parameter size demands substantial GPU memory. Evaluate single-GPU or multi-GPU setups based on concurrency requirements.
• Context Management: Under long-context scenarios, KV cache is the dominant memory consumer. Set appropriate maximum context lengths and batching strategies.
• Service Wrapping: The model is compatible with OpenAI-style and Anthropic-style APIs, enabling straightforward integration into existing agent frameworks with minimal migration effort.

5.2 Message Management for Multi-Turn Tool Calls

A critical engineering detail: multi-turn tool calls must preserve the full history of reasoning traces and tool messages. The model’s subsequent reasoning depends on the contextual state established in earlier steps. Truncating history and losing the reasoning trajectory will visibly degrade output quality in later turns.

Below is an illustrative structure for a multi-turn message sequence:

{
  "messages": [
    {"role": "user", "content": "Analyze the performance bottleneck in the codebase."},
    {"role": "assistant", "content": "First, I need to identify the critical code path...", "thinking": "..."},
    {"role": "tool", "tool_call_id": "call_1", "content": "{...profile_result...}"},
    {"role": "assistant", "content": "The profiler result shows the hotspot is in...", "thinking": "..."},
    {"role": "tool", "tool_call_id": "call_2", "content": "{...code_snippet...}"},
    {"role": "assistant", "content": "Bottleneck confirmed. The recommended fix is..."}
  ]
}

When building an agent framework, do not discard intermediate reasoning and tool responses in favor of keeping only the final answer. The complete message trajectory is the foundation for the model’s decision-making.

5.3 Thinking Budget and Task Routing

In production, routes can be assigned based on task type. For example, route simple document parsing tasks to the low budget tier and complex code analysis tasks to the max tier. This routing strategy helps optimize overall resource utilization while maintaining service quality.

6. Licensing and Compliance Considerations

The model weights are released under a custom license rather than permissive open-source licenses such as Apache 2.0 or MIT. Before any commercial use, thorough review of the license terms is required, with particular attention to:

• Whether commercial use is permitted.
• Any restrictions related to user scale or usage scenarios.
• Obligations regarding redistribution of modified versions.
• Requirements to retain model identifiers or attribution notices.

For enterprise users, integration should proceed only after the legal team has completed a license assessment. This is especially important if the model is intended to be embedded in commercial products or services.

7. Summary and Selection Recommendations

This model’s long-context capability, tool-calling mechanism, and thought-budget design make it genuinely practical for long-horizon agentic scenarios in software development, knowledge work, and 3D design. The hybrid attention and MoE architecture strike a practical balance between performance and efficiency, while joint reinforcement learning provides the foundation for multi-step decision-making.

When evaluating this model, consider the following dimensions:

• Task Characteristics: Does the workload involve multi-step, long-context, tool-dependent agentic tasks?
• Resource Constraints: Is sufficient GPU memory and compute available to run a 2.8T-parameter model efficiently?
• License Risk: Does the custom license align with your intended business use?
• Engineering Capability: Does your team have experience with multi-turn tool-call message management and model service integration?

For enterprises or development teams with long-horizon agent requirements, this model offers a viable technical path. In real-world applications, designing appropriate tool interfaces and message management strategies for the specific business context often matters more than the raw capability of the model itself.