Open-Source Long-Horizon Agent Models Explained: Sparse Attention, Joint Reinforcement Learning, and Tool Calling in Practice

1. Background: Why Long-Horizon Agents Need New Model Capabilities

Agent tasks differ fundamentally from single-turn question answering. An agent typically has to maintain goal consistency over an extended time span, continuously observe environment feedback, select tools, revise its plan, and complete the task after many steps of execution. Typical scenarios include:

• Code repair and software engineering tasks
• Terminal operation and command execution
• Web retrieval and research report generation
• Office document processing and information extraction
• 3D design and CAD generation
• Financial data research and multi-source comparison

These tasks place several key demands on the model:

1. Long-context understanding: the model must retain long tool histories, intermediate reasoning, error messages, and repair processes.
2. Stable reasoning and planning: in multi-step tasks, the model needs to keep its reasoning consistent and avoid forgetting or misusing earlier context.
3. Tool calling: the model must not only generate natural language but also emit executable tool call parameters.
4. Controllable inference cost: task complexity varies widely, so the thinking budget needs to be configurable.
5. Deployment and engineering readiness: models at this scale require quantization, sparse computation, and an efficient inference stack.

In recent years, Mixture-of-Experts (MoE) architectures and sparse attention mechanisms have become important approaches for balancing model capability against inference efficiency. The model discussed in this article follows this technical route: it is post-trained with joint reinforcement learning for long-horizon agent scenarios, and its weights are released openly so developers can deploy it themselves.

2. Model Architecture: Mixture-of-Experts and Sparse Attention

The model uses a Mixture-of-Experts architecture with roughly 2.8T total parameters and about 104B activated per forward pass. Unlike a dense model, an MoE model activates only a subset of its expert networks on each forward computation, controlling compute cost while preserving a large parameter capacity.

2.1 The Basic Advantages of MoE

The core idea of MoE is to split the feed-forward network into multiple expert modules, with a routing mechanism selecting a subset of experts for each input. The advantages include:

• Increasing model capacity without linearly increasing inference cost
• Allowing experts to specialize implicitly, covering a broader range of downstream tasks
• Suitability for large-scale parallel training and inference

MoE also introduces challenges, such as expert load balancing, routing stability, and expert utilization efficiency during fine-tuning. Through joint reinforcement learning post-training, this model attempts to optimize expert utilization and task performance across multiple downstream domains.

2.2 Sparse Attention Replacement

In the original version, some layers used gated multi-head latent attention. The new version replaces 24 layers of gated multi-head latent attention with gated dynamic sparse attention, combined with the incremental attention mechanism used by the remaining layers.

Dynamic sparse attention works as follows: at each attention computation step, a fixed number of tokens are selected from the historical tokens to participate in the attention computation. This model selects roughly 2,048 historical tokens per step.

Compared with full attention, this approach significantly reduces computational complexity for long sequences. At the 1-million-token context level, full attention carries high compute and memory costs, whereas sparse attention restricts attention computation to the key historical segments.

Key design considerations include:

• The selection strategy must preserve the historical information most relevant to current reasoning
• Both local context and long-range dependencies need to be covered
• In long-horizon tasks, the model must not lose early goal definitions, constraints, or key error information
• The dynamic selection mechanism should be coordinated with the inference budget and task type

From an engineering perspective, sparse attention requires a balance between accuracy and efficiency. If the selection strategy is too aggressive, important historical information may be lost; if the selection scope is too large, the performance gains are insufficient. This is why the joint reinforcement learning stage also needs to optimize the sparse attention selection behavior.

3. Training Method: Joint Reinforcement Learning Across Multiple Domains

The model does not rely solely on pretraining or supervised fine-tuning. On top of the base model, it undergoes large-scale joint reinforcement learning post-training.

3.1 Goals of Joint Reinforcement Learning

Traditional fine-tuning usually targets a single task or a small set of tasks. The difficulty with long-horizon agent models is that task types are diverse, reward signals are sparse, and tool call results often arrive as delayed feedback. Joint reinforcement learning attempts to optimize the policy across hundreds of downstream domains simultaneously, enabling the model to generalize to unseen task combinations.

During training, the model must learn:

• When to call a tool and when to answer from internal knowledge alone
• How to handle error messages returned by tools
• How to stay consistent with the goal across multi-step execution
• How to allocate reasoning budget according to task difficulty
• How to perform effective information retrieval and compression over long contexts

3.2 Weight Compression and Quantization

The model weights are quantized in MXFP4 format. Compared with traditional 8-bit or 4-bit integer quantization, MXFP4 is more flexible in its floating-point representation and can mitigate, to some extent, the precision loss caused by low-bit quantization.

In real deployments, weight quantization reduces memory footprint and transfer costs, but it also requires inference engines that support the format. For large-scale MoE models, the quantization strategy usually needs to be combined with sparse activation to achieve better overall gains.

4. Reasoning Budget: Three Thinking-Effort Tiers

The model offers three reasoning budget tiers: low, high, and max. Each tier corresponds to a different number of internal reasoning steps or amount of compute consumed.

4.1 Why Tiers Matter

Not every task needs max-tier reasoning. Simple information extraction, format conversion, and short-text generation achieve good results with low-tier reasoning, while complex code generation, long-horizon planning, and multi-tool coordination may require high-tier reasoning to guarantee stability and correctness.

From a cost engineering perspective, the reasoning budget directly affects response time and compute spend. Choosing the right tier achieves a better balance between quality and cost.

4.2 Common Tier Selection Strategies

• For low-complexity tasks, prefer the low tier to reduce latency
• For tasks requiring careful verification, such as code repair or mathematical derivation, use the high tier
• For agent tasks involving multi-turn tool calls and long contexts, default to the max tier
• Implement timeout and fallback policies at the application layer, automatically escalating when low-tier results are unstable

Different applications have different latency sensitivities. Online interactive scenarios may prioritize first-token latency, while offline batch jobs can tolerate longer total runtimes.

5. Typical Application Scenarios

5.1 Code Development and Software Engineering

In code development scenarios, the model must understand the repository structure, locate the files causing problems, generate fix patches, and, when necessary, run tests or static checks.

A long-horizon coding agent typically follows this loop:

1. Read the task description and repository information
2. Retrieve relevant code snippets
3. Analyze the root cause of the error
4. Generate a modification plan
5. Call tools to run tests
6. Continue fixing based on test results
7. Output the final patch

This loop requires the model to retain its full reasoning trace and tool call history across many turns of interaction. If the context is truncated or the attention selection is poor, the model may miss earlier error information, leading to repeated attempts or incorrect fixes.

5.2 Terminal Operation and Automation

Terminal operation tasks require the model to understand the command-line environment and decide the next action based on command output. The recurring characteristics of such tasks are:

• Highly structured feedback but highly varied error messages
• Single-step actions that affect subsequent state
• The need to handle permissions, paths, and dependencies
• The risk of irreversible operations

The model therefore needs strong instruction understanding and risk awareness. The reasoning budget should be set so the model performs thorough verification before each execution, avoiding mistakes.

5.3 Web Retrieval and Research

In web retrieval scenarios, the model must decompose a complex question into multiple queries, judge the relevance and credibility of search results, and synthesize a conclusion.

This scenario places particularly high demands on long context. Retrieved results may come from many sources with uneven quality. The model needs to retain key clues across a long context and avoid being distracted by noise. In such tasks, dynamic sparse attention should prioritize preserving the question background, constraints, and confirmed key facts.

5.4 Office Document Processing

Office document processing includes table question answering, long-document summarization, contract review, and meeting-notes organization. Inputs may contain native image-and-text content, requiring cross-modal understanding.

In these tasks, the model must accurately locate key paragraphs, tables, and figures in the document. The reasoning tier can be adjusted according to document length and question complexity.

5.5 3D Design and CAD Generation

3D design tasks require the model to understand spatial structure, geometric constraints, and parametric modeling logic. The model typically interacts with CAD tools or scripting environments, generates executable modeling code, and iterates based on rendering or validation results.

These tasks place high demands on multi-step tool calling and spatial reasoning. After generating modeling code, the model must verify geometric validity through tools—for example, detecting topology errors, dimension conflicts, or out-of-range parameters. Long-context capability is used to record the design change history and constraints.

6. Evaluation and Benchmarks

The model has published evaluation results on a number of tasks. Evaluations default to the max reasoning tier, and some tasks are executed within agent frameworks.

6.1 Metric Definitions

Evaluation metrics vary considerably across task types:

• Terminal operation tasks: task completion rate
• Software engineering tasks: patch pass rate or issue resolution rate
• Web retrieval tasks: answer accuracy
• CAD generation tasks: geometric usability or parameter match rate
• Office document tasks: structured question answering scores

Note that agent task scores are usually affected jointly by the reasoning tier, tool configuration, execution environment, and evaluation scripts. Horizontal comparisons should therefore be based on consistent settings.

6.2 Interpreting the Results

According to the published data, the model shows strong overall capability across terminal operation, software engineering, web retrieval, and CAD generation tasks. It is particularly outstanding in tool-assisted CAD generation, indicating a degree of spatial understanding and tool coordination ability.

Scores on office document tasks are slightly lower than on other tasks, which may be related to the strict structured-output requirements of this category. In real applications, further validation should be carried out against your own business data.

7. Deployment and Inference Serving

The model ships open MXFP4 weights and supports common high-performance inference frameworks. Keep the following points in mind when deploying.

7.1 Handling Multi-Turn Tool Call Messages

Multi-turn tool calling requires preserving the complete history of reasoning and tool call messages. Some inference frameworks truncate or compress historical messages under their default configuration, causing the model to lose key context.

Recommended practices:

• Maintain a complete message list for each request
• Append tool call results as separate messages
• Do not manually delete reasoning content in intermediate layers
• Set the correct role markers as required by the model

An illustrative message structure:

[
  {"role": "system", "content": "You are a long-horizon task agent..."},
  {"role": "user", "content": "Please fix the failing tests in the current project"},
  {"role": "assistant", "content": "First, check the test output..."},
  {"role": "tool", "name": "run_test", "content": "3 passed, 1 failed..."},
  {"role": "assistant", "content": "The failing case is located in..."},
  {"role": "tool", "name": "read_file", "content": "..."},
  {"role": "assistant", "content": "The final patch is as follows..."}
]

7.2 Reasoning Tier Configuration

The reasoning tier is usually set through sampling or API parameters. Different tiers may affect the number of reasoning tokens the model generates, the number of internal thinking rounds, or the search depth. When deploying, expose the tier as a configurable option to the application layer.

Example configuration approach:

reasoning_effort: high  # low | high | max
max_tokens: 4096
context_window: 1000000

7.3 Context Management and Caching

For a 1-million-token context, the inference service should enable prefix caching and incremental decoding. In multi-turn conversations, if the historical prefix is unchanged, existing caches can be reused to reduce repeated computation.

Dynamic sparse attention raises the bar for cache management. Tokens added in each turn may change which historical tokens are selected, so the caching strategy must match the attention selection logic.

7.4 API Compatibility

The model provides access methods compatible with mainstream chat interfaces. In practice, a unified gateway can wrap different inference backends behind a consistent API, simplifying migration and canary releases for business systems.

8. Engineering Practice Recommendations

8.1 Task Tiering and Routing

Do not send every request directly to a long-horizon agent model. First classify tasks with rules or a lightweight classifier:

• Simple Q&A: use a lightweight model or low-tier reasoning
• Single-step tool calls: use medium-tier reasoning
• Multi-step agent tasks: use max-tier reasoning with full context

This keeps overall costs down while maintaining quality.

8.2 Structuring Tool Return Information

Tool return content should be structured as much as possible, avoiding long, meaningless logs. Summarize results and classify errors on the tool side to reduce irrelevant information occupying the context, while retaining the details the model needs for judgment.

8.3 Failure Recovery and Resumable Execution

Long-horizon tasks may be interrupted by network issues, permission problems, or abnormal model output. Record checkpoints at the application layer—completed steps, tool call history, and intermediate artifacts—so that recovery does not require restarting from scratch.

8.4 Safety Boundaries

Terminal operation, file modification, and similar tasks carry real risk. Set up command whitelists, path restrictions, and permission isolation. For irreversible operations, require the model to request explicit confirmation before invoking the tool.

9. Conclusion

Long-horizon agent models are evolving from pure dialogue generation toward tool calling, long-context reasoning, sparse attention, and multi-domain joint optimization. The model discussed here attempts to balance capability and efficiency through a Mixture-of-Experts architecture, dynamic sparse attention, joint reinforcement learning post-training, and multi-tier reasoning budgets.

In real applications, developers need to care about more than benchmark numbers: inference serving architecture, multi-turn message management, context caching, tool feedback quality, and safety policies all matter. Only when model capability and engineering implementation work well together can long-horizon agent models deliver their full value on complex tasks.