Unlocking 2.8T Sparse LLMs for Long-Horizon AI Agents: Architecture, Tool Calling, and Deployment

1. Background: Required Capabilities for Long-Horizon Agents

Long-horizon agent tasks differ fundamentally from single-turn question answering. While single-turn QA typically requires a relatively stable answer within a limited context, long-horizon agents involve multi-step reasoning, environmental feedback, tool calling, state maintenance, and long-term context tracking. Typical scenarios include:

• Code Agents: Locating issues based on repository structure, modifying multiple files, running tests, and continuing fixes based on test results.
• Terminal Operation Agents: Executing multi-step command-line operations based on natural language instructions, observing outputs, and dynamically adjusting strategies.
• Knowledge Work Agents: Retrieving materials, reading long documents, extracting information, and generating structured deliverables.
• 3D Design and CAD Generation: Understanding geometric constraints and progressively generating and validating design results using tools.

These tasks impose several common requirements on the model. First, it must have a sufficiently long context window to accommodate the historical messages generated by multi-turn tool calls. Second, it must possess stable tool-calling capabilities to maintain intent consistency across multi-step workflows. Third, it must offer controllable reasoning depth, as different tasks have varying tolerances for computational overhead and response latency. Fourth, a balance must be struck between model scale and inference cost.

Recently, large-scale models utilizing sparse Mixture-of-Experts (MoE) architectures have begun opening their weights to developers. These models have a large total parameter count but only activate a subset of parameters during each inference step, striking a compromise between capacity and efficiency. This article analyzes a typical open-weight practice across four dimensions: architecture, training, inference deployment, and evaluation.

2. Model Scale and Core Architecture

The newly open-weighted model utilizes a Mixture-of-Experts architecture with approximately 2.8T total parameters and 104B activated parameters. This means the model does not use all parameters in a single forward pass; instead, it dynamically selects a subset of experts to participate in the computation based on the input. For developers, the direct benefit of this design is that single-inference computational costs remain controllable while maintaining a large model capacity.

Regarding the context window, the model supports up to 1 million tokens. This length is particularly critical for long-horizon agents. In multi-turn tool-calling scenarios, each round typically includes:

• User instructions
• Model’s thinking process
• Tool call requests initiated by the model
• Tool return results

If a task requires twenty or more tool calls, historical messages expand rapidly. A context window that is too short forces developers to discard early information mid-task, breaking task coherence. A 1-million-token window means most single long-horizon tasks can be completed without truncating history.

The model also supports native image and text input. For tasks like document processing, UI understanding, and CAD drawing recognition, visual information is often as important as text. Native multimodal input avoids the additional adaptation costs between external visual encoders and the language model, reducing information loss during cross-module transmission.

3. Sparse Attention and Inference Efficiency Optimization

A core challenge for long-context models is the computational complexity of attention mechanisms. The computation required by traditional full attention mechanisms grows quadratically with sequence length. When the context reaches hundreds of thousands of tokens, both inference latency and memory usage increase significantly.

The model employs a combined sparse attention approach:

1. Gated Sparse Attention: Certain layers use a gated mechanism to select approximately 2,048 tokens from the history to participate in the attention computation.
2. Incremental Attention: The remaining layers continue to use an incremental attention structure, responsible for transmitting and integrating global information.

The underlying premise is that not all historical tokens are equally important for generating the current token. By dynamically filtering the most relevant historical segments through a gating mechanism, computational load is reduced without significantly degrading semantic quality. For long-horizon agent tasks, this selection mechanism offers a potential advantage: the model can focus more intently on tool outputs and environmental feedback directly relevant to the current step, rather than being distracted by a large volume of stale messages.

It is worth noting that sparse attention does come with trade-offs. The gating selection itself requires extra computation and parameters, and selection bias may cause information omission in certain long-tail tasks. Therefore, the model retains incremental attention layers to compensate for the potential lack of global information that pure sparse attention might cause.

4. Joint Reinforcement Learning Post-Training

Building upon the base model, a large-scale joint reinforcement learning (RL) post-training phase was conducted, covering hundreds of downstream domains. Unlike single-task fine-tuning, the goal of joint RL is to improve the model’s capabilities across diverse task distributions simultaneously while avoiding overfitting to any single domain.

From a training methodology perspective, this type of post-training typically involves:

• Constructing datasets covering multiple domains such as code, math, tool calling, document understanding, and visual reasoning.
• Defining automatically evaluable reward signals, such as test pass rates, terminal output match rates, and tool call format correctness.
• Utilizing RL algorithms to optimize model behavior within the policy space, ensuring the model outputs not just “plausible-sounding” answers, but “verifiably correct” results.

For developers, the practical changes brought by joint RL are primarily reflected in the model’s behavior: it is more inclined to verify intermediate results in multi-step tasks rather than generating the final answer in one go; it is more likely to adjust its strategy after a tool call failure rather than simply retrying; and it demonstrates better strategy transfer capabilities across different domains.

5. Inference Budget Tiers

The model provides three configurable thinking budget tiers: Low, Medium, and High. This design addresses the varying demands for “reasoning depth vs. response speed” in different application scenarios.

Low Thinking Tier

Suitable for latency-sensitive scenarios, such as simple information extraction in interactive conversations, format conversion, and short text generation. In this tier, the model reduces internal reasoning steps to deliver a conclusion quickly. For tasks with clear boundaries and relatively deterministic answers, this setting significantly reduces time-to-first-token without noticeably compromising performance.

Medium Thinking Tier

Suitable for most daily development and knowledge work scenarios. The model performs necessary step-by-step reasoning under a medium budget but does not indefinitely expand its chain of thought. For moderately complex tasks like code modifications, multi-document summarization, tabular analysis, and structured output, the medium tier is usually a cost-effective choice.

High Thinking Tier

Suitable for complex long-horizon tasks, such as multi-file code refactoring, multi-turn terminal debugging, and complex CAD generation. In the highest tier, the model conducts deeper exploration and verification, potentially generating longer internal reasoning chains. Official evaluations default to this tier to achieve the highest task completion rate. Developers should decide whether to enable the maximum thinking budget based on actual task requirements.

6. Best Practices for Multi-Turn Tool Calling

A core capability of long-horizon agents is multi-turn tool calling. The model needs to:

• Parse user intent and determine whether a tool call is necessary.
• Generate a correctly formatted tool call request.
• Receive tool return results and evaluate if they meet expectations.
• Decide whether to continue calling, modify parameters, or terminate the task based on the results.

In engineering practice, there are several key considerations for multi-turn tool calls.

Preserve Complete Historical Messages

The multi-turn tool call message chain contains both the model’s thinking content and tool call records. In subsequent rounds, these historical messages must be passed back to the model in their entirety. Truncating history due to context length constraints may cause the model to lose critical environmental state, leading to incorrect decisions. Developers should implement robust message management at the infrastructure level, prioritizing historical integrity.

Strictly Validate Tool Call Formats

Different tool calling protocols have varying message format requirements. Developers should perform format validation before sending requests to avoid call failures caused by missing fields or type errors. When a tool call fails, the model typically consumes additional inference resources to correct it, simultaneously increasing latency and token consumption.

Design Reasonable Termination Conditions

A long-horizon agent requires a clear termination condition. Common strategies include setting a maximum number of rounds, defining task completion flags, or enforcing termination after consecutive failures reach a threshold. An agent without termination conditions might repeatedly call the same tool in a loop, wasting resources and failing to deliver results.

7. Evaluation Performance and Methodology

A series of official benchmark results have been released, covering terminal operations, software engineering, deep software engineering, browsing and retrieval, CAD generation, and office document Q&A. The following figures are derived from official disclosures, utilizing the highest thinking tier by default, with some evaluations employing code agent frameworks.

• Terminal Operation Benchmark: 65.7
• Software Engineering Marathon: 51.7
• Deep Software Engineering Benchmark: 75.5
• Browsing and Retrieval Benchmark: 92.1
• CAD Generation Benchmark (with tools): 92.7
• Office Document Q&A Benchmark: 64.4

It is important to note that the task definitions, scoring methods, and difficulty distributions across different benchmarks vary significantly, making direct horizontal comparisons inappropriate. For example, software engineering benchmarks typically involve fixing real bugs in large code repositories, which is significantly harder than single-function generation tasks. Conversely, browsing and retrieval tasks focus more on information finding and synthesis capabilities. Developers should use official evaluation results as a reference for model selection rather than the sole decision-making criteria.

8. Deployment and Inference Framework Adaptation

The open weights utilize a floating-point compression format specifically designed for sparse MoE models, reducing memory usage while maintaining model accuracy. Regarding inference frameworks, multiple community service engines currently offer compatible support, allowing developers to choose based on their infrastructure.

The model provides two compatible interfaces: one compatible with the OpenAI style and the other with the Anthropic style. This design reduces migration costs, enabling developers to integrate the model without extensively modifying existing agent code.

At the deployment level, a typical minimum viable architecture includes:

• Model Service Layer: Responsible for loading weights, handling inference requests, and managing the KV cache.
• Agent Orchestration Layer: Responsible for parsing user requests, maintaining session state, and executing the tool calling loop.
• Tool Execution Layer: Responsible for executing external operations such as code, terminal commands, and retrieval requests.

For large-scale production environments, engineering considerations like load balancing, request queuing, auto-scaling, and monitoring alerts must also be addressed. The expert routing characteristics of sparse MoE models result in an inference load distribution that differs from traditional dense models. Deployment should include targeted tuning for expert parallelism and resource allocation configurations.

9. Licensing Information

The open weights are released under a proprietary license, rather than a commonly used permissive open-source protocol. This means developers must carefully review the licensing terms before use to confirm whether commercial use is allowed, whether redistribution is permitted, and if there are additional restrictions on downstream applications. Different licenses impose varying compliance requirements on enterprises, making a legal review advisable before introducing the model into a production environment.

Looking at industry trends, an increasing number of large-scale models are opting to open their weights while retaining relatively strict licenses to balance ecosystem building with commercial protection. This represents a significant practical constraint for developers: while the model’s capabilities are accessible, the boundaries of usage must be clarified in advance.

10. Practical Recommendations and Conclusion

Synthesizing the above analysis, teams looking to utilize such open-weight models for long-horizon agent scenarios should consider the following recommendations:

First, clearly define the task’s actual requirements for context length, reasoning depth, and tool calling frequency. Not all scenarios require a 1-million-token context; an excessively long context may introduce unnecessary latency and costs.

Second, configure inference tiers rationally. Use the low tier for latency-sensitive services and the high tier for complex long-horizon tasks. If necessary, design a strategy for automatic tier switching.

Third, ensure robust engineering management of the tool calling pipeline, including guaranteeing historical message integrity, format validation, and designing retry and termination conditions.

Finally, pay close attention to model licensing compliance and deployment resource evaluation. Although large-scale sparse models offer better inference efficiency than dense models of the same parameter count, their absolute resource consumption remains substantial. Memory, bandwidth, and service capacity should be planned in advance.

Overall, ultra-large sparse models combined with long context windows and RL post-training are pushing long-horizon agents toward more complex, real-world task scenarios. For developers and enterprise users, the real challenge is not merely “acquiring the model weights,” but building a stable, efficient, and maintainable agent system around the model. The model is the foundation, but engineering capability is the decisive factor in determining final delivery quality.