A 2.8T Open-Source MoE LLM for Long-Horizon Agent Tasks: Architecture, Training, and Deployment

1. Background: New Architectural Requirements for Long-Horizon Agent Tasks

As Agent technology transitions from the lab to production environments, the core challenge for developers is no longer single-turn dialogue quality, but the stability and efficiency of models in long-horizon tasks. Long-horizon tasks typically involve complex workflows requiring dozens or even hundreds of tool calls, spanning multi-turn reasoning and state maintenance—such as terminal operation automation, repository-level code repairs, and multi-step CAD modeling.

These tasks impose three primary constraints on the underlying model:

1. Context Window Pressure: Fully retaining historical reasoning, tool calls, and return values causes the context length to expand rapidly.
2. Inference Cost Control: Not all steps require deep reasoning; an adjustable “thinking budget” is necessary.
3. Sparse Computation Efficiency: If the attention mechanism maintains dense computation, the VRAM requirements and latency under long contexts become unacceptable.

Recently, an open-source community released a new version of a large model based on the Mixture-of-Experts (MoE) architecture, boasting a total parameter count of 2.8T with 104B activated parameters. It supports a 1 million token context and native image-text input. This version systematically addresses the aforementioned constraints at the architectural level. This article analyzes the model across four dimensions: technical principles, training strategies, evaluation performance, and deployment practices.

2. Architecture Analysis: From Dense Attention to Sparse Selection

2.1 MoE Backbone and Activated Parameters

The model utilizes an MoE architecture with 2.8T total parameters, but only activates approximately 104B parameters during each forward pass. This design maintains a massive capacity for knowledge storage while keeping single-pass computation within acceptable limits. For developers, this means the model can harbor a broader range of domain knowledge while keeping inference costs closer to those of a medium-sized dense model.

2.2 Sparse Attention Mechanism: Combining DSA and Delta Attention

The core bottleneck of long contexts lies in the quadratic complexity of attention computation. Compared to its predecessor, this version introduces two key replacements:

• Replaced 24 layers of Gated MLA with Gated DSA (Dynamic Sparse Attention).
• Combined with 69 layers of Kimi Delta Attention to form a hybrid attention stack.

The core mechanism of DSA is that each attention calculation only selects 2,048 historical tokens to participate, rather than performing dense attention over all historical tokens. This dynamic selection strategy requires the model to learn “which historical information is worth attending to,” typically implemented via learnable routing or scoring mechanisms.

From an engineering perspective, this design yields two direct benefits:

• Controllable VRAM Usage: The KV Cache no longer grows linearly with the context to unbearable levels.
• Predictable Latency: Even when the context reaches the million-token level, the computational cost of a single attention step remains bounded to a near-constant scale.

Delta Attention is responsible for capturing local and recent dependencies. Together, they form a complementary structure of “global sparse + local dense.”

2.3 Native Image-Text Input

The model supports native image-text input, meaning visual and text encoders are aligned during the pre-training phase rather than stitched together later. For Agent tasks requiring the processing of UI screenshots, CAD drawings, or scanned documents, this capability is a necessary foundational support.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Joint RL Covering Hundreds of Downstream Domains

This version underwent large-scale joint reinforcement learning (RL) post-training based on the predecessor model. “Joint” means the training signals are not derived from a single task or a single reward model, but cover hundreds of downstream domains. This strategy aims to optimize the model simultaneously across diverse task distributions, avoiding overfitting to one category of task at the expense of general versatility.

For Agent scenarios, the value of RL is particularly pronounced. The correctness of tool calls, the coherence of multi-step reasoning, and the decision of “when to stop thinking” are difficult to acquire through pure supervised learning, making them better suited for guidance via reward signals.

3.2 MXFP4 Weight Quantization

MXFP4 weight quantization was employed during the training phase. MXFP4 is a 4-bit floating-point microscaling format, a common low-precision quantization scheme in the industry. Its significance lies in:

• Reduced VRAM Bandwidth Requirements: Weights are stored in 4-bit and de-quantized during inference.
• Maintained Numerical Stability: The microscaling mechanism sustains acceptable error rates even at extremely low precision.

It is worth emphasizing that quantization occurred during the training phase rather than as a post-training compression step. This approach typically yields better precision retention than post-training quantization.

4. Capability Dimensions: Three Core Task Scenarios

4.1 Code Development and 3D Creation

The model is tailored for long-horizon code Agents, software engineering, and 3D design tasks. Official evaluations cover:

• Terminal Operations: Executing multi-step tasks in a command-line environment.
• Code Repair: Locating and fixing repository-level defects.
• CAD Generation: Completing parametric modeling using tools.

A common characteristic of these tasks is the need to integrate external tools, maintain cross-step state, and roll back and retry upon failure. The model’s three-tier thinking budget (low / high / max) is particularly useful here—quickly passing simple steps using the low tier, and switching to the max tier for deep reasoning during complex decisions.

4.2 Agent and Knowledge Work

The model supports web retrieval, financial research, office document processing, and tool calling. For enterprise users, this means they can build:

• Retrieval-Augmented Research Assistants: Automatically searching, reading, and summarizing multi-source information.
• Document Processing Pipelines: Parsing tables, extracting fields, and generating reports.
• Tool Orchestration Agents: Calling internal APIs to complete approvals, queries, and notifications.

4.3 Reasoning Budget: low / high / max

The essence of the three-tier thinking budget is the adjustability of Test-Time Compute. Developers can dynamically set it based on task complexity:

# Pseudo-code: Selecting thinking budget based on task type
def select_budget(task_type):
    if task_type == "simple_lookup":
        return "low"      # Quick response, minimal reasoning
    elif task_type == "code_repair":
        return "high"     # Medium reasoning depth
    elif task_type == "cad_generation":
        return "max"      # Maximum reasoning budget, multi-step planning

This design allows the same model to serve both latency-sensitive interactive scenarios and quality-prioritized batch processing scenarios.

5. Official Evaluation Data Interpretation

The following are the evaluation scores published for this version (defaulting to the max thinking tier, with some evaluations using an Agent framework):

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

These numbers should be viewed objectively:

• Terminal-Bench measures multi-step operation capabilities in a terminal environment. A score of 65.7 indicates the model has a functional foundation as a command-line Agent.
• SWE Marathon focuses on long-horizon software engineering tasks. A score of 51.7 shows room for improvement in repository-level repairs, but it still surpasses most peer open-source models.
• The high scores on BrowseComp and BenchCAD (92.1 / 92.7) demonstrate outstanding performance in retrieval and tool-augmented CAD tasks.
• The 64.4 score on OfficeQA Pro reflects that office document understanding and Q&A still present challenges.

Developers should treat these evaluations as a reference baseline rather than an absolute guarantee—actual performance heavily depends on prompt engineering, tool encapsulation, and task decomposition strategies.

6. Deployment and Engineering Practices

6.1 Weight Format and Inference Engines

The model releases MXFP4 weights and supports mainstream inference engines (such as vLLM, SGLang, and TokenSpeed). It provides OpenAI/Anthropic compatible APIs, meaning existing applications based on these interface specifications can migrate at a relatively low cost.

6.2 Key Considerations for Multi-Turn Tool Calling

The official documentation specifically emphasizes: Multi-turn tool calling requires retaining the complete history of reasoning and tool call messages. This is a prerequisite for the correct operation of long-horizon Agents. If history is truncated or discarded between turns, the model will lose state, leading to repeated calls or logical breaks.

For engineering implementation, it is recommended to:

{
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "Reasoning process...", "tool_calls": [...]},
    {"role": "tool", "content": "Tool return result..."},
    {"role": "assistant", "content": "Next step based on the result..."}
  ]
}

The reasoning content and tool_calls within the assistant message of each turn must be fully retained, rather than keeping only the final text.

6.3 License Information

The weights are released under a specific model license, rather than Apache 2.0 or MIT. Enterprise users must carefully review the licensing terms before commercial use to confirm the scope of use, distribution conditions, and compliance requirements. This is particularly crucial for teams planning to integrate the model into commercial products.

7. Conclusion and Selection Recommendations

The core value of this open-source MoE model version lies in: using sparse attention and adjustable reasoning budgets to push long-horizon Agent tasks from “theoretically feasible” to “engineering practical”. The combination of 2.8T total parameters, 104B activated parameters, a 1 million token context, and native image-text input forms a complete technical foundation tailored for Agent scenarios.

For developers and enterprise users, evaluation is recommended across the following dimensions:

1. Task Match: If core scenarios involve terminal operations, code repair, retrieval augmentation, or CAD generation, this model is worth prioritizing for testing.
2. Inference Cost: MoE activated parameters and sparse attention collectively control the single-step cost, but stress testing is still required based on actual QPS and latency requirements.
3. Compliance and Licensing: The specialized license mandates a legal review before commercial deployment.
4. Engineering Integration: Compatible APIs lower migration costs, but the history management for multi-turn tool calls needs to be implemented independently and tested rigorously.

The era of long-horizon Agents is arriving, and the architectural evolution of underlying models—from dense to sparse, and from fixed reasoning to adjustable budgets—is the critical infrastructure supporting this trend.