Large Sparse MoE Model Open-Sourced: A Technical Breakdown of Long-Horizon Agent Capabilities

1. Background: Why Long-Horizon Agents Are the Key Battleground for LLMs

Over the past year, competition among large models has shifted from “single-turn answer quality” to “multi-step task completion.” The term long-horizon agent refers to a model’s ability to maintain goal consistency across dozens or even hundreds of steps—autonomously invoking tools, retrieving information, recovering from errors, and ultimately delivering verifiable results. These tasks pose a threefold challenge:

• Context pressure: Long tasks mean massive message histories, tool outputs, and intermediate reasoning must be retained continuously.
• Reasoning budget control: Different tasks demand vastly different thinking depth; a simple lookup and a complex code fix should not consume equal compute.
• Training objective alignment: Single-turn supervised fine-tuning struggles to cover credit assignment across multi-step decisions, requiring joint reinforcement learning to optimize entire trajectories.

A sparse mixture-of-experts (MoE) model with 2.8T total parameters and 104B activated parameters has now been released with open weights. It natively supports 1M-token context and image-text input, and publishes full evaluation data across terminal operation, code repair, CAD generation, and web retrieval tasks. This article breaks down the technical essentials of such models across three layers: architecture, training, and deployment.

2. Architecture: Co-Design of Sparse Attention and Mixture-of-Experts

2.1 MoE: Trading Activated Parameters for Inference Efficiency

The model uses a 2.8T total-parameter, 104B activated-parameter MoE architecture. This means only about 3.7% of parameters are activated in the forward computation of each token. The core value of MoE is maintaining extremely large model capacity while keeping per-inference compute costs at a level acceptable for dense models.

A typical MoE layer consists of multiple expert feed-forward networks and a gating network. The simplified logic is as follows:

# Illustrative logic for MoE forward computation
def moe_forward(x, experts, gate, top_k=2):
    # gate outputs a weight for each expert
    scores = gate(x)                      # [num_tokens, num_experts]
    weights, indices = topk(scores, top_k) # select top-k experts
    weights = softmax(weights)
    output = 0
    for i in range(top_k):
        expert_out = experts[indices[:, i]](x)
        output += weights[:, i:i+1] * expert_out
    return output

The key engineering difficulty is load balancing: if the gating network persistently favors a few experts, training becomes unstable and inference develops hotspots. Practical training therefore usually introduces an auxiliary load-balancing loss and caps expert capacity.

2.2 Sparse Attention: Making Million-Token Context Affordable

The biggest obstacle to million-level context is the quadratic complexity of attention. The model makes a clear architectural substitution: the original 24 layers of gated multi-head latent attention (Gated MLA) are replaced with gated dynamic sparse attention (Gated DSA), combined with 69 layers of incremental attention. DSA selects only 2,048 historical tokens per pass to participate in attention computation.

The engineering implications of this design are:

• Selective recall: Instead of dense attention over the full history, the model routes to the historical segments most relevant to the current token.
• Layered combination: Sparse attention handles long-range retrieval, while incremental attention handles short-range coherence; the two are complementary.
• Controllable memory: The KV cache growth curve shifts from linear expansion to approximately controllable, making million-token context feasible in engineering terms.

Note that sparse attention is not “lossless compression.” It is highly sensitive to retrieval routing quality, so the routing network typically must be jointly optimized with the main model during training rather than inserted as an external add-on.

3. Training Path: Joint Reinforcement Learning and Quantization

3.1 From Single-Turn to Multi-Turn: The Necessity of Joint Reinforcement Learning

The model was obtained through large-scale joint reinforcement learning post-training on top of its predecessor. “Joint” here means the training objective covers hundreds of downstream domains rather than independent fine-tuning per task. The motivations are:

1. Credit assignment: In long-horizon tasks, the success or failure of the final result must be attributed to a decision made dozens of steps earlier; single-step rewards cannot provide effective gradients.
2. Tool-use consistency: The format, parameters, and exception handling of multi-turn tool calls require unified constraints; otherwise the model tends to “format drift” over long trajectories.
3. Negative transfer across domains: Training domains independently tends to cause mutual interference; joint training with a domain sampling strategy can instead improve generalization.

A typical multi-turn tool-calling trajectory must fully preserve thinking messages and tool-call messages in context:

[
  {"role": "user", "content": "Analyze this financial report and generate a chart"},
  {"role": "assistant", "content": "Need to read the file first", "tool_calls": [...]},
  {"role": "tool", "content": "<file parsing result>"},
  {"role": "assistant", "content": "Proceed to call the charting tool", "tool_calls": [...]},
  {"role": "tool", "content": "<chart generation result>"}
]

Key practical point: During multi-turn tool calling, the full history of thinking and tool-call messages must be preserved. Any truncation may cause the model to lose task state, resulting in repeated calls or goal drift.

3.2 Quantization: Trade-offs of MXFP4 Weights

The model releases MXFP4 quantized weights. The core idea of the MX (Microscaling) format is to divide tensors into small blocks, each sharing a scaling factor, with elements within a block represented at low bit width. Compared with traditional integer quantization, MXFP4 is more robust to outliers at extremely low bit widths.

Practical implications for developers:

• Significantly lower memory footprint: Weight size is roughly one quarter of FP16, making large-parameter models deployable on limited hardware.
• Accuracy loss must be evaluated: Quantization typically affects long-horizon reasoning more than single-turn Q&A; regression testing on target tasks is recommended.
• Inference framework support: Mainstream high-performance inference frameworks have progressively added support for this format; verify version compatibility during deployment.

4. Capability Evaluation: Interpreting Performance on Long-Horizon Tasks

The official evaluation covers multiple long-horizon task dimensions, using the highest thinking tier by default, with some evaluations combined with agentic coding tool frameworks:

Evaluation Dimension	Score	Description
Terminal operation	65.7	Multi-step task execution in command-line environments
Long-horizon software engineering	51.7	Marathon-style tasks spanning files and multiple repair rounds
Deep software engineering	75.5	Complex code understanding and modification
Web retrieval	92.1	Multi-hop information retrieval and integration
CAD generation (with tools)	92.7	Completing 3D design tasks with tool support
Office document QA	64.4	Structured document understanding and extraction

Several points deserve attention when interpreting these numbers:

• Long-horizon scores are generally lower than short-horizon scores: This is an industry-wide phenomenon, indicating that long-horizon consistency remains a bottleneck.
• Tool-assisted evaluations score notably higher: The model’s capability ceiling depends heavily on the maturity of the toolchain.
• Thinking tier matters significantly: The default highest tier means scores were achieved with ample reasoning budget; actual deployment must weigh latency against cost.

5. Deployment and Integration Essentials

5.1 Choosing an Inference Engine

MXFP4 weights are supported by multiple mainstream inference engines, with both OpenAI-compatible and Anthropic-compatible API forms. This means existing agent frameworks can integrate at relatively low cost. During deployment, consider:

• Whether efficient implementations of sparse attention are supported;
• Memory and throughput behavior under long context;
• KV cache reuse strategies across multi-turn sessions.

5.2 Dynamic Scheduling of Thinking Budget

The model offers three thinking budget tiers: low, high, and max. In engineering practice, using the highest tier globally is not recommended; route by task type instead:

# Illustrative thinking-budget routing
def route_budget(task_type):
    if task_type in ("simple_qa", "format_convert"):
        return "low"
    elif task_type in ("code_review", "doc_analysis"):
        return "high"
    else:  # long-horizon agent, complex repair
        return "max"

5.3 License Considerations

The model uses a dedicated license rather than a permissive one such as Apache 2.0 or MIT. Enterprises must review the license terms clause by clause before commercial use, particularly provisions on distribution, derivative models, and commercial use, to avoid compliance risk.

6. Summary and Outlook

The open-sourcing of this model sends several clear signals:

1. Ultra-large MoE + million-token context is moving from papers to deployable engineering.
2. Sparse attention has become the mainstream approach to long context, but routing quality determines the ceiling.
3. Joint reinforcement learning is a key training paradigm for long-horizon agent capabilities.
4. Open quantized weights lower the deployment barrier but bring dual considerations of accuracy and compliance.

For developers, the real value lies not in parameter scale itself but in whether long-horizon agent capabilities can be decomposed into engineering modules that are testable, routable, and rollback-capable. A recommended starting point is a single long-horizon task: first validate the stability of multi-turn tool calling, then gradually expand to complex workflows.