Engineering Long-Context MoE Models and Long-Horizon AI Agents

Recently, the open-source community welcomed an ultra-large-scale sparse Mixture-of-Experts (MoE) model with a total parameter count of 2.8T and activated parameters of 104B. This model not only supports a 1 million token context window and native image-text inputs but also focuses its technical core on building “long-horizon Agent” capabilities. By introducing a hybrid sparse attention mechanism and joint reinforcement learning (RL) post-training covering hundreds of domains, the model demonstrates outstanding engineering potential in scenarios such as terminal operations, code repair, and complex knowledge work. This article will deeply analyze the technical core of such ultra-large models from four dimensions: architecture analysis, capability evolution, application practices, and deployment guidelines.

1. Architectural Reshaping: Ultra-Long Context and Sparse Attention

When processing ultra-long contexts at the 1 million token level, traditional standard Multi-Head Attention (MHA) or conventional Multi-Head Latent Attention (MLA) faces memory explosions and exponentially growing computational complexity. To break through this bottleneck, the new generation model has undergone deep sparse reconstruction at the architectural level.

1.1 Hybrid Formation of Gated DSA and Delta Attention

Unlike early versions that fully adopted Gated MLA across all layers, the new model replaces 24 of its layers with Gated Dynamic Sparse Attention (Gated DSA) and combines them with Delta Attention in the remaining 69 layers. The design philosophy of this hybrid architecture is “allocating computing resources on demand”:

• Dynamic Sparse Selection: When processing long texts, Gated DSA does not compute global attention between every token and all historical tokens. Instead, it uses a gating network to dynamically filter the most relevant 2,048 historical tokens for computation. This local focus mechanism significantly reduces computational complexity from O(N²) while preserving the ability to capture long-range dependencies.
• Hierarchical Feature Extraction: The 69 layers of Delta Attention handle the rapid flow of basic features and local context, while the 24 layers of Gated DSA perform precise global information retrieval at key depth levels, balancing computational efficiency and the model’s expressive power.

1.2 Integration of Native Multimodality and Long-Range Memory

Beyond text input, the model natively supports image inputs. In Agent scenarios, this means the model can directly process UI screenshots, CAD drafts, or financial research reports containing charts. The 1 million token context window combined with sparse attention allows the Agent to retain a complete “history of thoughts and tool call messages” during prolonged multi-step tasks, avoiding task disorientation caused by context truncation.

2. Joint RL Post-Training for Long-Horizon Tasks

The base model’s capability is only the foundation; what truly makes the model stand out in Agent tasks is large-scale joint reinforcement learning post-training.

2.1 Full-Link Joint Domain Training

The post-training phase covers hundreds of downstream domains, including but not limited to software engineering, financial research, office document processing, and 3D design. This “joint” rather than “sequential” training strategy effectively mitigates catastrophic forgetting in multi-task learning. While learning to call terminal tools to execute code repairs, the model still maintains high-level web retrieval and information synthesis capabilities.

2.2 Dynamic Allocation of Reasoning Thinking Budgets

Complex Agent tasks have inconsistent requirements for reasoning depth. The model innovatively introduces a three-tier thinking budget mechanism: low, high, and max:

• Low tier: Suitable for simple tool calls or single-turn Q&A, enabling fast responses and reducing inference costs.
• High tier: Suitable for routine code repairs or multi-step retrieval tasks, balancing speed and accuracy.
• Max tier: Designed for marathon-style software engineering debugging or highly complex financial modeling, performing deep logical deduction.
  This design grants developers the flexibility to dynamically adjust computing power consumption based on specific business SLAs.

3. Core Application Scenarios and Engineering Practices

Relying on ultra-long context and powerful tool-calling capabilities, the model shows high practical value in multiple long-horizon Agent scenarios.

3.1 Code Development and 3D Creation

For software engineering and 3D design tasks, the model can not only generate code but also act as a “long-horizon code Agent” to intervene in the entire development process.

• Terminal Operations and Code Repair: The model can take over terminal tools, autonomously execute test commands, read error logs, locate bugs, and propose repair patches. In relevant industry benchmarks (e.g., scoring 65.7 on Terminal-Bench 4.0 and 51.7 on SWE Marathon v1.1), the model demonstrated autonomous repair capabilities approaching those of mid-level developers.
• CAD Generation and 3D Design: Combined with the toolchain, the model can understand 3D design intent and generate or modify CAD models through multi-step interactions, achieving a high score of 92.7 on the tool-enabled BenchCAD evaluation.

3.2 Complex Knowledge Work and Automated Workflows

In financial research and office automation, the model can chain web retrieval, document parsing, and internal tool calls. For example, given the instruction “analyze a company’s financial reports over the past three years and generate a summary,” the model can retrieve the latest data via web search, call an office document processing API to parse the PDFs, and finally output a structured report. Scoring 92.1 on the BrowseComp evaluation proves its strong information navigation and synthesis capabilities.

4. Deployment Guidelines and API Integration

For enterprise-level deployment, the ease of model deployment and API compatibility are crucial.

4.1 Quantization Deployment and Inference Framework Support

To alleviate the memory pressure brought by 2.8T parameters, the model offers an MXFP4 weight quantization version. As an advanced low-bit floating-point quantization format, MXFP4 significantly compresses the model size while relatively preserving the precision of expert weights in the MoE architecture. Currently, the model natively supports mainstream high-performance inference frameworks (such as vLLM, SGLang, etc.), allowing developers to utilize PagedAttention or continuous batching technologies from these frameworks to improve service throughput.

4.2 API Compatibility and Message Retention Guidelines

The model provides a standard API compatible with mainstream LLM providers (e.g., OpenAI/Anthropic formats), allowing existing business code to switch with almost zero cost. However, when integrating multi-turn tool calls, special attention must be paid to the message retention guidelines:
Unlike standard conversational APIs, the reasoning process of a long-horizon Agent relies on every preceding thought and tool return result. Developers must fully retain the historical assistant thinking messages, as well as the tool_calls and tool response messages, in every request.

Below is a simplified example of a multi-turn tool call message structure:

{
  "messages": [
    {"role": "user", "content": "Check today's weather and generate outfit suggestions"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\":\"Beijing\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"temp\": 15, \"condition\": \"sunny\"}"},
    {"role": "assistant", "content": "Based on the current 15-degree sunny weather in Beijing, it is recommended to wear a long-sleeved shirt with a light jacket..."}
  ],
  "thinking_budget": "max"
}

4.3 License Compliance Notes

It is highly important for developers to note that although the model’s weights are open, it uses a proprietary license rather than the familiar Apache 2.0 or MIT licenses. Before integrating it into commercial products or redistributing it, corporate legal and engineering teams must carefully review the commercial use restrictions and derivative open-source requirements in the license to avoid compliance risks.

Conclusion

From simple text generation to long-horizon Agents capable of deep reasoning and tool calling, ultra-large MoE models are reshaping enterprise automated workflows. By breaking through long-context computational bottlenecks with hybrid sparse attention, aligning with hundreds of engineering domains via joint RL, and balancing computing costs with flexible thinking budgets, these models offer new solutions for complex software engineering and knowledge work. In actual deployment, developers must focus not only on benchmark scores but also on engineering details such as memory quantization deployment, multi-turn message state management, and license compliance to truly unlock the Agent potential of large models.