Ultra-Large Sparse Models for Long-Horizon AI Agents: Architecture and Engineering Practices

1. Background: Why Long-Horizon Agents Need New Model Architectures

As AI agents transition from simple Q&A to multi-step task execution, the challenges shift from “answering a single question” to “maintaining goal consistency, tool-calling accuracy, and reasoning stability across contexts spanning tens or hundreds of thousands of tokens.”

In traditional dense models, significantly increasing context length causes attention computation and memory usage to surge, while the utilization efficiency of early information declines. To address long-horizon agent tasks, the industry is exploring two key improvements:

1. Sparse Attention: Dynamically selects historical information most relevant to the current token, avoiding full attention computation across all historical tokens in long sequences.
2. Mixture of Experts (MoE): Expands total parameter capacity while maintaining a low number of activated parameters, controlling single-inference costs while boosting model capacity.

The model discussed in this article adopts this technical route: it has a total parameter count of 2.8T but activates only about 104B parameters per inference. It supports a 1 million token context window and native image-text inputs. This design aims to provide a stable foundation for multi-step agent tasks such as long-form code development, software engineering, 3D design, financial research, and office document processing.

2. Model Architecture: MoE and Joint Attention Mechanisms

2.1 Engineering Significance of MoE

The model utilizes a Mixture of Experts architecture with 2.8T total parameters and 104B activated parameters. This design yields direct benefits:

• Increased Capacity: The model can internalize more knowledge and skills without linearly increasing computation overhead per inference.
• Controlled Inference Cost: The computation per token is dictated by the activated parameters, not the total parameter count.
• Flexible Task Routing: Different expert modules can develop domain-specific representation capabilities during joint training.

From a deployment perspective, MoE models typically require more complex parallelism strategies. For example, expert layers often use expert parallelism, distributing different experts across computing devices, while non-expert layers continue to use tensor or pipeline parallelism. Teams aiming for private deployment must plan communication bandwidth and memory topology in advance; otherwise, routing and aggregation can become performance bottlenecks.

2.2 Attention Evolution: From Gated MLA to Gated DSA

A notable architectural change from previous generations is the replacement of 24 layers of Gated Multi-head Latent Attention (MLA) with Gated Dynamic Sparse Attention (DSA), working in conjunction with incremental attention mechanisms in the remaining 69 layers.

The core idea of Dynamic Sparse Attention is that for each query position, the model does not attend to all historical tokens. Instead, it uses a learnable scoring mechanism to select a fixed number of historical tokens for attention computation. In this model, that number is 2048 historical tokens.

This mechanism introduces several changes:

• Improved Long-Sequence Processing Efficiency: When the context reaches hundreds of thousands or a million tokens, full attention becomes computationally unfeasible. Fixed-window or fixed-count sparse attention keeps computational complexity within acceptable limits.
• Enhanced Information Selection: Unlike hard truncations like sliding windows, the dynamic selection mechanism allows the model to look back across distances based on semantic relevance—such as tracing a function definition hundreds of lines back in a code repair task.

It is worth noting that sparse attention comes with trade-offs. The training process requires the model to learn “what to attend to,” necessitating large-scale long-context training data. If long-range dependency samples are insufficient in the training data, sparse attention may degrade into “only attending to local context,” impairing performance on long-horizon tasks.

2.3 Context Length and Image-Text Input Capabilities

The 1 million token context window provides a larger operational space for long-horizon agents. In software engineering tasks, development repositories may contain numerous source files, dependency definitions, and test scripts; the model needs to establish correlations across multiple files. In financial research, long contexts can accommodate complete historical reports, announcements, and market data sequences.

Native image-text input means the model receives image information alongside text sequences. For tasks like 3D design, UI understanding, and chart analysis, this capability reduces the information loss and latency associated with chaining external vision models. In engineering, image-text input is typically achieved through a unified tokenization scheme: images are encoded into feature sequences and concatenated with text tokens, removing the need for the model to internally differentiate between source types.

3. Training Strategies: Joint Reinforcement Learning and Quantization

3.1 Joint RL Across Hundreds of Downstream Domains

Instead of limiting reinforcement learning (RL) to a single vertical domain, this model is jointly trained across hundreds of downstream scenarios. The advantages of this approach include:

• Cross-Task Generalization: The “locate problem, formulate hypothesis, verify hypothesis” pattern learned in code repair can partially transfer to financial document analysis or office workflow processing.
• Reduced Overfitting: Single-domain reward signals can bias model policies toward specific solutions. Multi-domain joint training encourages the model to learn more general tool-calling and reasoning strategies.
• Tool Usage Stability: In multi-step tasks, the model must repeatedly switch between reasoning, calling external tools, observing results, and resuming reasoning. Joint training provides richer interaction samples, improving policy stability.

For external observers, this training approach manifests as usable performance across various benchmarks—terminal operations, code repair, CAD, web retrieval, and office document Q&A—rather than excelling in only one category.

3.2 Weight Quantization and Deployment Friendliness

The model weights are released using the MXFP4 quantization format. Compared to traditional 16-bit or 8-bit precision, 4-bit quantization significantly reduces the resources required for model storage and loading. For a trillion-parameter scale model, 4-bit quantized weights can typically be compressed to the hundreds of GBs level, making single-machine or multi-machine memory loading feasible.

However, quantization introduces a trade-off between precision and inference quality. Engineering teams typically need to evaluate:

• Whether quantization errors accumulate during long-horizon tasks;
• Whether sparse attention scoring is sensitive to quantization;
• Whether tool-calling output formats remain stable at lower precisions.

Therefore, deployment teams are advised to conduct small-scale validations in critical business scenarios, comparing differences in task completion rates and output format validity between quantized and unquantized versions.

4. Reasoning Tiers: Allocating Thinking Budgets for Different Tasks

A noteworthy design feature is the provision of three reasoning tiers—low, high, and max—corresponding to different reasoning depths and computational overheads. This mechanism acknowledges that different tasks require varying depths of reasoning.

• Low Thinking Tier: Suited for structurally clear, low-step tasks like simple information extraction, format conversion, or short-flow Q&A. The model outputs faster, reducing latency.
• High Thinking Tier: Suited for tasks requiring multiple tool calls or comparing multiple candidate solutions, such as code bug localization, document review, or moderately complex data analysis.
• Max Thinking Tier: Used for the most complex agent tasks and typically the default in official benchmarks. The model expands longer reasoning chains and exercises more caution in deciding when to call tools and how to interpret returned results.

From a practical standpoint, the thinking budget should not simply be “the higher, the better.” Higher tiers bring additional output tokens and latency, yielding no significant benefit for simple tasks. Therefore, in production systems, it is recommended to establish a tiered routing strategy based on task type, acceptable user wait time, and cost budgets.

5. Tool Calling and Long-Horizon Agent Practices

5.1 Context Management for Multi-Turn Tool Calls

The core characteristic of a long-horizon agent is the “tool call -> observe result -> continue reasoning” loop. In long contexts, each tool call adds new message blocks, including the model’s generated thoughts, tool names, call parameters, and returned results.

An easily overlooked issue is whether the returned results of early tool calls remain useful in subsequent reasoning. If the model needs to trace back to the initial error message or file directory structure after several rounds, whether sparse attention can successfully relocate key content in these historical messages directly impacts the task success rate.

Therefore, in engineering practice, multi-turn tool calling must retain the complete history of thoughts and tool call messages. This is not only necessary state information for model reasoning but also the foundation for subsequent debugging, auditing, and regression analysis. Truncating intermediate messages to save context may cause the model to make incorrect judgments without the full historical chain.

5.2 General Deployment and Invocation Methods

With the model weights available, common industry inference frameworks can serve as deployment foundations. Whether accessed via an OpenAI-compatible interface or other general agent interfaces, the integration elements typically include:

Model service endpoint
Model name
Context window size
Enable tool calling
Maximum output tokens
Thinking budget tier

Within an agent framework, the model is typically configured as the main controller, deciding whether to continue generating text or initiate a tool call. For agents that need to execute code, operate terminals, or access external data sources, the server must additionally provide sandbox environments and permission controls to prevent unexpected model behavior.

5.3 Minimal Invocation Example

Below is a conceptual pseudo-code snippet illustrating message organization during multi-turn tool calling, independent of any specific vendor API:

messages = [
    {"role": "system", "content": "You are a long-horizon code agent."},
    {"role": "user", "content": "Locate the root cause of the test failure in the repository."},
    {"role": "assistant", "content": "I will first list the key directory structure."},
    {"role": "tool_call", "name": "list_files", "arguments": {"path": "."}},
    {"role": "tool_result", "content": "src/ tests/ pyproject.toml ..."},
    {"role": "assistant", "content": "There are multiple failed cases in the test directory. Next, I will check the corresponding source files."}
]

In this message structure, all historical messages are retained in the context. Sparse attention dynamically selects the historical fragments to attend to based on the current query, rather than simply deleting or compressing early messages.

6. Performance and Interpretation

The official release includes results across multiple tasks. The table below summarizes them for comparative analysis:

Evaluation Task	Score
Terminal Operation Benchmark	65.7
Software Engineering Multi-Turn Tasks	51.7
Deep Software Engineering Tasks	75.5
Complex Browsing and Retrieval	92.1
Tool-Assisted CAD	92.7
Office Document Q&A	64.4

These scores were achieved using the max thinking tier, with some tasks utilizing external agent frameworks as execution environments. Key takeaways from the distribution:

• The model performs strongly in browsing/retrieval and tool-assisted design tasks, indicating a mature tool-calling and multi-step information gathering pipeline.
• The relatively lower score in software engineering multi-turn tasks reflects the inherent difficulty of long-range dependencies, complex state management, and multi-file collaboration in real code repositories—consistent with broader industry trends.
• Office document Q&A sits at a mid-level, likely related to challenges involving unstructured layouts, table understanding, and cross-document verification.

It must be emphasized that benchmark scores only reflect performance on specific datasets and differ from real production environments. For enterprise users, a more reliable approach is to build a small-scale regression set based on internal business data to continuously track model stability across different versions and deployment conditions.

7. Licensing and Usage Boundaries

The model weights are released under a proprietary license rather than a permissive open-source agreement. Users must carefully read and comply with the license terms, especially concerning commercial distribution, derivative models, and service provisioning.

For enterprise users, it is recommended that legal and compliance teams evaluate license restrictions before integrating the model into internal development workflows or external services. Proprietary licenses may impose constraints regarding:

• Whether the model weights can be redistributed;
• Whether developing commercial products or services based on the model is permitted;
• Whether specific disclaimers must be retained or usage restrictions observed;
• Whether there are additional requirements regarding user volume, service forms, or application domains.

Compared to permissive agreements, proprietary licenses protect the developer’s rights while introducing compliance costs for users. Teams should incorporate licensing risks into the technical evaluation phase rather than addressing them only before delivery.

8. Implementation Best Practices

8.1 Design Thinking Tier Routing Based on Task Complexity

Do not use the max thinking tier for all requests. Route based on task characteristics:

• Simple extraction, summarization, format conversion: Low tier;
• Scenarios requiring tool calls but limited steps: High tier;
• Multi-file code repair, cross-document analysis, complex terminal operations: Max tier.

Initial routing rules can be established based on domain tags or user descriptions, and dynamically adjusted using historical completion rates.

8.2 Retain the Complete Tool Calling Chain

In long-horizon agents, tool call messages should not be easily discarded. Even if context length reduction is necessary, prioritizing summary compression or local sliding windows over simple truncation is advised. The complete historical chain is critical for model backtracking, system auditing, and error localization.

8.3 Evaluate the Impact of Quantization on Business Metrics Early

Although quantization significantly lowers deployment resource requirements, task sensitivity to quantization varies. Conduct comparative experiments on critical business scenarios, focusing on:

• Whether tool-calling parameter formats remain valid;
• Whether sparse attention selection exhibits significant drift;
• Whether detail recall in long-document Q&A degrades.

8.4 Establish Localized Regression Evaluation

Official benchmarks are important references but cannot replace evaluations within your specific business context. Extract real tasks to build a regression set covering typical success paths and failure modes, and run it automatically after every model or deployment parameter change. This effectively identifies performance regressions and policy drift.

9. Conclusion

Ultra-large sparse models offer a viable path for long-horizon agents, balancing model capacity and inference costs. By controlling activated parameters via MoE, achieving selective attention in long contexts via Dynamic Sparse Attention, and leveraging multi-domain joint RL and tiered thinking budgets, these models demonstrate usable capabilities in code development, knowledge work, and 3D creation scenarios.

For developers and enterprise users, understanding the architectural principles and engineering constraints yields more long-term value than simply focusing on leaderboard scores. Every环节—from thinking tier routing and tool calling chain management to quantization validation and license evaluation—affects the final implementation outcome. Only by closing the loop between technical capabilities and engineering practices can long-horizon agents be transformed into stable, maintainable, and scalable productivity systems.