Long-Horizon AI Agents with a 2.8T-Parameter Sparse MoE Model: Architecture, Dynamic Sparse Attention, and Open-Weight Deployment

As agent workloads evolve from simple Q&A toward long-horizon planning, tool calling, and complex code engineering, models need more than larger knowledge capacity—they also need deeper reasoning, stronger context retention, and stable multi-step execution. A recently published open-weight release, built on a sparse mixture-of-experts (MoE) architecture with 2.8T total parameters and roughly 104B activated parameters per forward pass, has been systematically optimized for long-horizon coding agents, software engineering, 3D design, and knowledge work. This article analyzes the model across architecture, attention mechanisms, training strategy, quantization, deployment, benchmark results, and engineering practice, helping developers understand both the design rationale and how to work with such a model.

1. Model Scale and Positioning

The model adopts a sparse MoE architecture with 2.8T total parameters while activating only about 104B per forward pass. This expands model capacity while keeping inference cost under control, making long-context tasks viable at higher throughput.

It supports a 1M-token context window and natively accepts image-text input. For long-running agent tasks, this means the model can maintain continuous reasoning across extended conversation histories, codebase context, documentation, and tool outputs without frequent truncation or compression.

Compared with the previous generation, the key changes lie in the attention layers and training strategy. The original multi-layer gated MLA structure has been replaced with a gated DSA structure, used in combination with another type of Delta attention layer. DSA’s core mechanism dynamically selects a fixed number of tokens from the historical tokens at each step to participate in attention computation, reducing attention overhead on long sequences.

2. Capability Design for Long-Horizon Agents

Long-horizon agents differ substantially from single-turn Q&A. A single-turn task usually only requires understanding a question and generating an answer, whereas a long-horizon agent must maintain goals across multiple execution cycles, remember completed steps, read tool returns, fix errors, and keep making progress.

2.1 Code Development and 3D Creation

In code development scenarios, the model needs to understand repository structure, locate relevant files, analyze dependencies, generate patches, and handle failing tests. Official evaluations cover terminal operation, code defect repair, and 3D design generation. Taking terminal operation as an example, the model must not only understand user instructions but also select appropriate commands based on the current environment state, process command output, and troubleshoot errors when necessary.

3D design tasks further test spatial understanding and geometric expression. The model must generate usable design results from natural language descriptions or engineering constraints, and coordinate with external tools through multi-step workflows.

2.2 Agents and Knowledge Work

Knowledge work scenarios include web retrieval, financial research, office document processing, and tool calling. These tasks are characterized by scattered information, multi-round retrieval and synthesis, and strict requirements on output format and reliability.

The model provides three thinking budget tiers: low, high, and max. Developers can pick a tier based on task complexity to balance response speed against reasoning depth. For example, simple document formatting can run on a lower budget, while financial research that requires cross-validation across multiple sources suits the max budget.

3. Sparse Attention and Long-Context Efficiency

One of the main challenges for long-context models is that attention computation grows significantly with sequence length. Traditional dense attention creates enormous compute pressure at the million-token scale.

3.1 Dynamic Sparse Attention

The dynamic sparse attention (DSA) mechanism introduced in this version dynamically selects a fixed number of tokens from history at each layer—for example, 2048 tokens—to participate in attention. Unlike static sparsity patterns, dynamic selection can retrieve relevant information from farther-back positions in the history according to the current token’s semantic needs.

This can be understood as an “on-demand retrieval” attention strategy: instead of computing full attention over all historical tokens, each layer filters out the globally most relevant subset for the current computation. For long-horizon agent tasks, this mechanism helps reduce compute overhead while preserving information integrity.

3.2 Hybrid Attention Layers

The model contains both multiple DSA layers and multiple Delta attention layers. Different layers may serve different functional roles: some handle local semantic modeling while others capture long-range dependencies. By combining the two attention structures, the model maintains reasoning coherence across long documents, long conversations, and long tool histories.

On the engineering side, this hybrid structure raises the bar for inference frameworks. The inference engine must be aware of the attention type at each layer and schedule compute resources accordingly. For developers, choosing an inference framework that supports such sparse structures can significantly improve long-context inference efficiency.

4. Joint Reinforcement Learning and Quantization

4.1 Joint Reinforcement Learning

Training uses a joint reinforcement learning strategy spanning hundreds of downstream domains. Unlike single-task RL, joint RL mixes code, math, logic, tool calling, and knowledge retrieval task types during training, with the goal of improving generalization to unseen tasks.

In long-horizon agent tasks, reward signals are often sparse: the model only receives positive feedback after completing the entire task or reaching an intermediate checkpoint. Joint RL can mitigate the training difficulty caused by sparse rewards through intermediate process supervision and subtask rewards.

4.2 Weight Quantization

The open weights use the MXFP4 quantization format. Compared with common integer quantization, floating-point quantization retains more numerical detail across a larger dynamic range while keeping the storage footprint small. For MoE models, where different experts may have different weight distribution characteristics, floating-point quantization helps reduce precision loss while compressing model size.

5. Benchmarks and Performance

Evaluation is essential for understanding a model’s capability boundaries. The model has published systematic results across multiple tasks, using the max thinking tier by default, with some evaluations run on top of agent frameworks.

• Terminal operation: 65.7, indicating basic multi-step operation capability in a real command-line environment.
• Software engineering marathon: 51.7, suggesting room for improvement on long-cycle code repair tasks.
• Deep software engineering: 75.5, showing relative stability on structured engineering problems.
• Knowledge retrieval: 92.1, reflecting strength in complex retrieval and information synthesis.
• 3D design with tools: 92.7, indicating the model can leverage external tools well for geometry generation tasks.
• Office document QA: 64.4, showing that complexly formatted, information-dense documents still need further optimization.

Overall, the model performs strongly on tool-assisted and retrieval tasks, while performance on software engineering tasks that require longer cycles and many intermediate steps still has room to grow. This matches the broader trend in long-horizon agent research: single-step capability is strong, but multi-step consistency and error recovery remain the bottlenecks.

6. Inference Deployment and Multi-Turn Tool Calling

6.1 Inference Frameworks and Compatible APIs

The open weights support several mainstream inference frameworks and provide API interfaces compatible with OpenAI and Anthropic. When selecting a framework, developers should confirm whether it supports dynamic sparse attention and correctly handles the hybrid attention layer structure.

During deployment, pay close attention to GPU memory footprint, batching capacity, and long-context scheduling strategies. Because MoE models exhibit dynamic activation paths during inference, properly setting batch size and cache policies helps strike a basic balance between throughput and latency.

6.2 State Management for Multi-Turn Tool Calls

A key point in practice is that multi-turn tool calls require preserving the complete history of thinking and tool-call messages. The model relies on historical context to maintain goal state during long-horizon tasks; truncating or trimming history can interrupt tasks or cause abnormal behavior.

A recommended implementation pattern:

# Example: preserving full history across multi-turn tool calls
conversation_history = []
def run_agent_step(user_input, tool_executor):
    # Append the user input to the history
    conversation_history.append({"role": "user", "content": user_input})
    # Call the model with the full history preserved
    model_output = model.chat(
        messages=conversation_history,
        thinking_budget="max"
    )
    # Append the model output to the history
    conversation_history.append({"role": "assistant", "content": model_output})
    # If the model requests tool calls, execute them and continue the loop
    if model_output.tool_calls:
        for tool_call in model_output.tool_calls:
            tool_result = tool_executor.execute(tool_call)
            conversation_history.append(
                {"role": "tool", "content": tool_result}
            )
    return model_output

In real deployments, also account for the storage and transmission cost of history messages. When the context grows to hundreds of thousands of tokens, the input payload of a single request can increase substantially. You can use the context caching capabilities provided by inference frameworks to reduce repeated prefix computation, but avoid cache policies that inadvertently drop historical information.

7. License and Usage Boundaries

Note that the open weights are released under a custom license, not permissive licenses such as Apache 2.0 or MIT. Before using the model in commercial products or public-facing services, developers should read the license terms carefully to confirm the scope of use, redistribution restrictions, and any data compliance requirements.

For enterprise users, it is advisable to assess the license’s fit with their business scenarios with support from legal or compliance teams. The availability of open weights does not equate to unrestricted commercial authorization.

8. Summary and Engineering Recommendations

This 2.8T-parameter sparse MoE model offers a fairly complete technical package covering long context, tool calling, dynamic sparse attention, and quantized deployment. For teams planning deep work on long-horizon agents, the following recommendations are worth considering:

1. Choose thinking budgets by task. There is no need to use the max budget for simple tasks; enable high reasoning tiers only for complex multi-step tasks to keep latency and cost under control.
2. Preserve the full tool-call history. Do not truncate history to save transmission overhead—broken state harms long-horizon tasks far more than the added cost.
3. Verify sparse attention support in your inference framework. Before deployment, confirm compatibility with DSA and the hybrid attention layers to avoid performance anomalies caused by missing features.
4. Mind the license constraints. Open weights do not imply full commercial freedom; plan your rollout path with compliance in mind.

As long-horizon agents see deeper adoption in code engineering, knowledge retrieval, and 3D design, model architectures, training strategies, and deployment approaches will continue to evolve. Understanding the current technical characteristics and engineering boundaries is the prerequisite for putting such models to practical, effective use.