Technical Deep Dive into Open-Source Long-Horizon Agent Models: Sparse Attention and Reinforcement Learning Practices

1. Background: Foundational Model Requirements for Long-Horizon Agents

AI agents are evolving from single-turn Q&A toward multi-step task execution. Whether addressing repository-level bug fixes, cross-document information synthesis, or constrained CAD modeling, models must maintain goal consistency, tool-calling stability, and reasoning traceability over extended time horizons. This imposes four core capability requirements on foundation models:

1. Information Retention in Long Contexts: Tasks may span hundreds of thousands of tokens, encompassing historical tool outputs, intermediate reasoning steps, and partial execution results. The model must effectively distinguish noise from critical states.
2. Controllable Reasoning Budgets: Simple tasks should not consume excessive computation, while complex tasks require sufficient depth. Configurable reasoning tiers are therefore necessary.
3. Native Multimodal Input: Visual information such as engineering drawings, screenshots, and document layouts frequently intertwine with text.
4. Stable Tool Calling and Message Protocols: Multi-turn tool invocation requires preserving the complete thought history and message structure to prevent context fragmentation and redundant calls.

A recently open-sourced long-horizon agent model provides a comprehensive design reference in these areas. The model features a total parameter count of 2.8T with 104B activated parameters. It utilizes a Mixture of Experts (MoE) architecture, supports a 1-million-token context window, and natively accepts image-text inputs. Rather than focusing on a specific product, this article deconstructs the architectural choices, training methodologies, and engineering deployment experiences behind it from a general technical perspective.

2. Model Architecture: Attention Evolution from MLA to DSA

2.1 The Foundation of the MoE Architecture

The core idea of the MoE architecture is to partition the feed-forward network into multiple experts, activating only a subset of experts per token based on a routing strategy. With 2.8T total parameters and only 104B activated parameters, this model mobilizes approximately 3.7% of its parameters during each forward pass, significantly reducing floating-point operations during inference. For long-horizon agents, this sparse activation design controls the computational overhead of long-sequence reasoning without sacrificing model capacity.

2.2 Replacing Dense Attention with Sparse Attention

A key update in this version is the replacement of 24 layers of Gated Multi-head Latent Attention (Gated MLA) with Gated Dynamic Sparse Attention (Gated DSA), forming a hybrid structure with Delta attention mechanisms in the remaining layers. Delta attention reduces redundant computation by maintaining state increments across layers, while DSA layers dynamically select 2,048 historical tokens to participate in the attention calculation during each inference step.

The value of this design lies in the realization that long context does not necessitate traversing the entire history at every layer. For a coding agent, the context of the current editing position is typically only relevant to a few specific definitions, call sites, and recent modification records. Through a gating mechanism, DSA learns “which historical tokens are worth revisiting,” compressing the computational complexity of attention from linear growth with sequence length to a fixed-window level. Notably, DSA selects “historical tokens,” whereas the current token itself usually does not participate in this selection. This marks a significant departure from traditional sliding window attention or global sparse attention.

2.3 Engineering Implications of the Context Window

A 1-million-token context window means the model can accommodate the entire content of a medium-sized code repository or hundreds of pages of technical documentation. However, in actual agent execution, the context is jointly occupied by tool call results, intermediate reasoning traces, and system prompts. Therefore, the context window size is merely the foundation; what matters more is the model’s “effective attention radius” and “state compression capability” within long sequences. This is precisely the core problem addressed by the combination of DSA and Delta attention.

3. Training Strategy: Joint Reinforcement Learning and Quantization Deployment Preparation

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

Rather than applying reinforcement learning (RL) for a single task, the model employs a joint training framework covering hundreds of downstream domains, including code, mathematics, tool calling, document processing, financial analysis, and CAD design. The primary challenge of joint training is the vast difference in reward signals across domains: code repair and terminal operations require self-feedback based on execution results, document QA relies on accuracy and citation completeness, and 3D design requires structured evaluations like geometric constraint satisfaction.

The advantage of joint RL is enhancing the model’s robustness against “task switching.” In practical use, agents often need to complete multiple subtasks within a single session, such as “read error logs — locate code position — modify — run tests — write modification notes.” Single-domain training can easily lead to capability degradation during subtask transitions. Multi-domain mixed sampling and dynamic task weighting mitigate this issue to a certain extent.

3.2 MXFP4 Quantization and Inference Efficiency

The model weights are released in MXFP4 format. MXFP4 is an accelerator-friendly 4-bit floating-point quantization scheme. Unlike traditional INT4 or standard FP4, it expands the dynamic range by sharing exponent bits. For MoE models, the transmission bandwidth of quantized parameters becomes one of the bottlenecks for inference throughput. While maintaining model output quality, MXFP4 significantly reduces memory footprint and weight loading overhead.

It is important to note that quantization is not lossless. For inference chains requiring extremely high precision, it is recommended to retain higher-precision KV caches and activations on the server side, applying quantization only to static weights. Furthermore, the KV cache expands as the context grows during long-horizon agent inference. During deployment, the overall memory consumption of both the KV cache and the weights must be evaluated to avoid optimizing only the weights while neglecting cache growth.

4. Inference Configuration: Tiered Reasoning Budgets and Performance

4.1 Design Logic of Reasoning Budgets

The model offers low, medium, and high reasoning budget tiers. The low tier is suited for simple information extraction and short-answer generation; the medium tier fits tool calls requiring a certain reasoning chain; the high tier is designed for the planning and verification of complex, multi-step tasks. The essence of the reasoning budget is limiting the number of internal reasoning tokens the model generates before producing the final answer. This mechanism is directly tied to the reasoning length constraints used during the RL training phase.

For developers, a sound budget allocation strategy is to first use a low budget for task classification and simple tool calls. When the system detects low confidence or a tool call failure, it can progressively escalate to the medium or high tier. This dynamic escalation path significantly reduces average inference costs while ensuring task completion rates.

4.2 Capability Profile from Official Benchmarks

According to public benchmark data, the model scores 65.7 on terminal operation-related benchmarks, 51.7 on software engineering marathon benchmarks, 75.5 on deep software engineering benchmarks, 92.1 on web browsing reasoning benchmarks, 92.7 on tool-augmented CAD benchmarks, and 64.4 on office document QA benchmarks. These metrics were measured under the high reasoning tier, with some evaluations utilizing external agent frameworks as task executors.

Notably, the significant score gap between terminal operations and the software engineering marathon benchmarks indicates that the model still faces challenges in “open-ended repository-level modification” scenarios. Reasons for this variance may include the accumulation of planning errors in long-horizon tasks, context drift caused by overly long tool call sequences, and consistency maintenance across multi-file modifications. Developers should treat these benchmarks as upper-bound capabilities rather than a stable level achievable across all tasks.

4.3 Message Integrity in Multi-Turn Tool Calls

The model requires preserving the complete thought history and tool call messages during multi-turn tool invocations. A common engineering pitfall exists here: to save context space, developers truncate early intermediate reasoning content, retaining only tool calls and results. This practice can cause the model to “lose its reasoning thread,” leading to inconsistent decisions in subsequent steps.

A more robust approach is to apply tiered compression to historical messages rather than outright deletion. For completed and verified subtasks, their reasoning processes can be summarized and retained; for the currently active reasoning chain, the messages should remain intact. Additionally, the system prompt should explicitly define the format and citation method for tool return results to reduce the model’s need to repeatedly read historical information.

5. Deployment Architecture and Engineering Practices

5.1 Inference Engine Compatibility Design

The model is compatible with multiple mainstream inference engines and provides compatibility interfaces for two common API styles. For teams requiring rapid integration, this means existing client code and tool-calling encapsulation layers can be reused without redesigning message formats. However, in actual deployment, different inference engines vary in their support for long contexts and sparse attention quantization kernels. A full performance regression test is recommended when switching engines.

5.2 Key Configuration Items for Inference Services

When deploying large-scale MoE models like this, attention should be paid to the following configurations:

# Illustrative inference configuration items (not directly runnable code)
config = {
    "max_context_len": 1_000_000,
    "reasoning_budget": "medium",        # low / medium / high
    "attention_backend": "sparse",       # Enable DSA sparse attention kernel
    "kv_cache_dtype": "fp16",
    "weight_quant": "mxfp4",
    "expert_parallelism": 4,             # Expert parallelism degree
    "tensor_parallelism": 8,             # Tensor parallelism degree
    "enable_prefix_cache": True,         # Prefix caching to reduce repeated prompt computation
}

Among these, prefix caching is particularly critical for agent scenarios. Each conversation turn for an agent carries the same system prompt and historical message prefix. Enabling prefix caching avoids redundant attention computation for unchanging prefixes, significantly improving first-token latency.

5.3 Context Management Strategies

For a 1-million-token context, server-side memory management and scheduling strategies directly impact system stability. A “reserved window” mechanism is recommended: divide the context into a fixed system area, a rolling history area, and a dynamic working area. The rolling history area employs a hybrid strategy of content summarization and retaining the complete last K turns; the dynamic working area is used for current task tool calls and intermediate results. This partitioning prevents a single task from indefinitely occupying the context while ensuring the integrity of the current reasoning chain.

5.4 Private Deployment and Licensing Considerations

The model weights are released under a proprietary license rather than a traditional permissive open-source agreement. This means commercial deployment, model modification, and redistribution may be subject to additional constraints. Enterprise users should carefully review the usage boundaries and limitations in the licensing terms before integrating it into internal infrastructure or commercial services. For scenarios with strict compliance requirements, it is advisable to complete model evaluation and performance testing in an isolated environment before deciding to introduce it into the production pipeline.

6. Practical Recommendations for Developers

Synthesizing the above analysis, model selection and engineering implementation for long-horizon agents should adhere to the following principles:

Assess task complexity before selecting a reasoning tier: Do not default to the highest reasoning tier. Establish a complexity grading standard for tasks—for instance, automatically selecting the budget based on metrics like the number of tool calls, code files, or document pages—to reduce overall costs without lowering the success rate.

Treat tool design as the primary system element: Model capability is only one part of agent performance. The stability of tool interfaces, the structural degree of return results, and the interpretability of error messages directly impact task success rates. It is recommended to provide a clear input-output JSON Schema for each tool and include status codes and error classifications in the return results.

Establish long-context monitoring and alerting: Track metrics such as context fill rate, KV cache growth curves, and the distribution of historical tokens selected by the attention layer during actual execution. When a certain type of task frequently triggers context truncation or reasoning chain rebuilding, promptly adjust the partitioning strategy or streamline the tool return content.

Ensure regional deployment and data compliance assessments: For long-horizon agents handling sensitive business data, localized inference is a more robust choice. Evaluate the inference engine’s compatibility with quantized weights and sparse attention operators, and conduct long-sequence stress tests on the target hardware to avoid inference failures or sudden performance drops caused by hardware characteristic differences.

The technological evolution of long-horizon agents is shifting from “larger parameters” to “smarter sparse computation” and “more controllable reasoning budgets.” For developers, understanding the trade-off logic behind the architecture holds more long-term value than merely chasing benchmark scores. When the model’s capability boundaries are clear, the deployment pipeline is controllable, and tool protocols are stable, long-horizon agents can truly transition from demos to production.