Open-Weight Ultra-Large MoE Model: Long-Horizon Agent Capabilities and Sparse Attention in Practice

Background: What Long-Horizon Agents Demand from Models

Agent-based applications differ fundamentally from single-turn Q&A. A complete agent task often involves dozens or even hundreds of steps: reading environment states, planning sub-goals, calling tools, observing results, revising plans, and calling tools again. This chain places three hard requirements on the underlying model:

1. Context length: Multi-turn tool calls continuously accumulate history, including the model’s own reasoning, tool call parameters, and tool outputs. An insufficient context window directly causes task interruption or early information loss.
2. Long-horizon consistency: The model must maintain goal stability over extended periods, remembering previously established constraints and completed sub-tasks.
3. Controllable inference budget: Simple tasks need low-latency responses, while complex tasks require deeper reasoning. A fixed inference cost cannot satisfy both.

A recently released class of ultra-large MoE models with open weights is designed around these three points. This article analyzes the architecture, training, evaluation, and deployment, and highlights engineering pitfalls to avoid.

Model Architecture: Separating Total and Active Parameters

The model employs an MoE architecture with total parameters around 2.8T and approximately 104B active parameters per forward pass. This “large total, small active” combination is mainstream for ultra-large models: total parameters determine capacity ceiling and knowledge density, while active parameters determine per-inference compute cost. Separating the two allows far greater effective capacity than a dense model within limited memory and compute budgets.

Supporting capabilities include:

• Million-token context: Provides room for multi-turn tool-call histories in long-horizon agents.
• Native image-text input: Visual information enters the context directly without an external vision encoder pipeline.
• Weights released in MXFP4 quantized format: Lowers download and deployment memory barriers, enabling deployment on common accelerators.

It is important to note that MoE memory footprint does not equal active parameters. All expert weights must remain resident or be swappable during deployment. Memory planning should be based on total parameters and quantization bit-width, not active parameters.

Sparse Attention Rework: From Global to On-Demand Selection

Long context introduces the problem of attention computation scaling quadratically with sequence length. The model applies two key modifications at the attention level:

• Replaces some gated multi-head latent attention layers with a gated sparse attention variant.
• Combines these with linear attention layers, forming a hybrid structure of “sparse selection + linear compression.”

The sparse attention mechanism works as follows: each query position selects a fixed number (2048 in this model) of historical tokens for attention computation, rather than attending to the full sequence densely. This design has both benefits and costs:

Benefits: Attention computation is decoupled from total context length, significantly reducing per-step cost in long contexts. The selection mechanism focuses the model on the most relevant historical segments for the current query.

Costs: Selection itself introduces extra indexing and scoring overhead. If selection accuracy is insufficient, critical historical information may be missed. This is why training must jointly optimize the selection module.

From an engineering perspective, this sparse attention places high demands on inference framework operator support. Before deployment, verify that the inference backend implements the corresponding sparse attention kernel; otherwise, it may degrade to dense computation, losing the architectural advantage.

Training Path: Joint Reinforcement Learning with Quantization Awareness

The model’s post-training phase uses joint reinforcement learning covering hundreds of downstream domains. “Joint” has two meanings here:

1. Task jointness: Terminal operation, code repair, retrieval Q&A, document processing, and CAD generation are trained together to avoid overfitting to a single task and losing generality.
2. Reward jointness: Different tasks use their own verification signals (e.g., test case pass rate, answer match, tool call success rate) unified into a single RL objective.

The direct result is more stable performance on multi-step, tool-assisted tasks, rather than simply scoring high on single-turn benchmarks.

For quantization, weights use MXFP4 format. MXFP4 is block-level floating-point quantization that preserves dynamic range better than integer quantization, which is friendlier to MoE expert weights with widely varying numerical distributions. However, quantization inevitably introduces precision loss. Official benchmark scores are typically measured on quantized weights, so real business performance should be revalidated on your own data.

Inference Budget Tiers: Low / High / Max

The model offers three thinking budget tiers to trade off latency and effectiveness:

Tier	Use Case	Engineering Advice
low	Classification, extraction, simple rewriting	Default tier to control first-token latency
high	Multi-step reasoning, moderate tool calls	Suitable for most agent sub-tasks
max	Complex code repair, long-horizon planning, benchmark comparison	Latency and token consumption rise significantly; rate-limit required

Official benchmark scores default to the max tier. This is critical when interpreting results: benchmark scores represent the capability ceiling, not the default deployment behavior. Running everything at max in production will noticeably increase cost and response time. A sensible approach is to route tasks by tier, for example dynamically specifying the tier in the agent orchestration layer based on sub-task complexity.

Interpreting Official Benchmarks

The published results cover terminal operation, software engineering, retrieval, and office Q&A. Code and retrieval metrics are relatively strong, and CAD tasks perform well when tools are provided. Three points are important for interpretation:

1. Evaluation framework differences: Some evaluations rely on external agent frameworks. Scores include the framework’s planning and tool orchestration ability, not just the raw model.
2. Tool dependence: Scores marked “with tools” are not directly comparable to those without tools.
3. Tier differences: As noted, scores use the highest thinking tier and should be discounted for real deployment.

A more useful practice for developers: select one or two benchmarks closest to your business, construct a small evaluation set from your own data, reproduce results at the same tier and tool configuration, and form a comparable baseline.

Deployment and Tool-Call Best Practices

The weights are released in quantized format, with support in mainstream inference backends and API-compatible interfaces for existing agent frameworks. Several details require attention:

1. Preserve complete multi-turn tool-call history

This is the most common pitfall in long-horizon agents. During tool-call turns, the model’s reasoning content and tool-call messages are part of the protocol. If trimmed when assembling the next request, the model loses decision basis, manifesting as repeated calls, parameter errors, or plan collapse. Example pseudo-code:

# Wrong approach: keep only the tool return value
messages.append({"role": "tool", "content": tool_result})
# Correct approach: fully preserve the assistant's reasoning and tool calls
messages.append({
    "role": "assistant",
    "content": assistant_text,          # reasoning content
    "tool_calls": tool_calls            # tool call parameters
})
messages.append({"role": "tool", "content": tool_result})

2. Context budget management

A million-token context does not mean you should fill it indefinitely. Longer histories increase pressure on attention selection, and cost rises linearly with token count. Implement a tiered strategy at the orchestration layer: keep recent messages in full, compress mid-range messages with summaries, and retain only structured conclusions for distant messages (e.g., completed sub-tasks, key constraints). When compressing, preserve the causal chain of tool calls to avoid losing call parameters in summaries.

3. License compliance

The model weights use a dedicated license, not a permissive open-source license. Before commercial use, carefully review each clause, including scope of use, redistribution conditions, and attribution requirements. Do not assume Apache 2.0 or MIT terms apply.

4. Tier and concurrency control

At the max tier, token consumption per request can be several times higher than at low. Set different concurrency limits and timeout thresholds at the gateway layer per tier. Also introduce step limits and failure rollback mechanisms for long-horizon tasks to prevent infinite loops consuming quotas.

Recommended Use Cases and Selection Guidance

Based on architecture and benchmark results, this model class suits the following scenarios:

• Long-horizon code agents: Cross-file fixes, terminal operation, multi-step builds and verification.
• Deep retrieval and research workflows: Multi-round retrieval, cross-validation, and information synthesis.
• Office document processing: Long-document understanding, structured extraction, multi-tool coordination.
• Tool-assisted 3D/CAD generation tasks: Parametric modeling dependent on tool chains.

Less suitable scenarios include: ultra-low-latency online interaction (use a smaller model or low tier), numerical tasks highly sensitive to quantization precision (must verify MXFP4 impact independently), and memory-constrained edge deployment (total parameter size sets the floor).

Summary

The combination of ultra-large MoE with sparse attention provides a viable path for long-horizon agents: total parameters for capacity, active parameters and sparse selection to control per-step cost, tiered thinking budgets to balance latency and effectiveness, and joint reinforcement learning to improve multi-step stability. What truly determines deployment success is often not benchmark scores but engineering details: context management, complete message history, tier routing, and license compliance. Before integration, construct a small evaluation set from your own data, establish a reproducible baseline, and then gradually expand to production traffic.