Open-Source Long-Horizon Agent Models: MoE Architecture, Sparse Attention, and Reinforcement Learning in Practice

1. Background: Why Long-Horizon Agents Need New Model Structures

Agent tasks are fundamentally different from single-turn question answering. A typical long-horizon agent workflow may include goal decomposition, tool selection, environment observation, failure retry, context correction, and final result validation. The model must not only generate the next action but also maintain stable memory of historical states, tool returns, error information, and intermediate reasoning chains throughout the entire task cycle.

In this setting, traditional dense models tend to encounter three types of bottlenecks:

1. Insufficient context length: They cannot retain multiple rounds of tool calls and historical reasoning content.
2. Excessive inference cost: Full attention over all historical tokens in long contexts causes compute overhead to rise rapidly with the number of steps.
3. Insufficient task generalization: Fine-tuning on a single domain is hard-pressed to cover highly divergent downstream tasks such as code repair, terminal execution, web retrieval, and document processing.

As a result, a new generation of long-horizon agent models typically adopts sparse mixture-of-experts architectures, hierarchical sparse attention, and joint reinforcement learning across multiple domains to balance capability, cost, and context length. This article takes one class of open-source long-horizon agent models as its subject and analyzes its architecture design, training strategy, inference configuration, and deployment considerations.

2. Overall Architecture: MoE and Activated Parameter Design

The model uses a MoE architecture with 2.8T total parameters and 104B activated parameters. Two concepts need to be clarified here:

• Total parameters: The sum of all expert modules, attention layers, embedding layers, and other parameters in the model.
• Activated parameters: The number of parameters that actually participate in computation during the forward pass for a single token.

The key advantage of a MoE model is that it can increase total capacity by adding more experts while keeping per-token inference cost relatively controllable. For example, out of 2.8T total parameters, only 104B are activated, meaning each token computation uses only about 3.7% of the parameters, striking a compromise between model capacity and computational efficiency.

This structure suits multi-task agent scenarios. Different downstream tasks can be handled by different expert combinations: code tasks may activate experts biased toward structured reasoning, document tasks may activate experts biased toward semantic extraction, and multi-round tool calls require expert paths involved in state tracking. The expert routing mechanism gradually differentiates during training without manually specifying task boundaries.

From an engineering perspective, the MoE architecture also imposes requirements on deployment. When the full 2.8T parameters cannot fit into a single GPU’s memory, expert parallelism, multi-GPU sharding, or weight quantization must be used. The model chooses MXFP4 weight quantization, a low-bit floating-point quantization method that reduces memory usage during inference while preserving the numerical distribution of expert weights as much as possible. In actual deployment, attention should still be paid to the impact of quantization error on long-horizon task stability, especially for numerically sensitive computational tasks.

3. Sparse Attention: Evolution from MLA to DSA

In long-context inference, attention computation is usually the main performance bottleneck. Assuming a sequence length of 1 million tokens, the computational complexity of traditional dense attention is O(n²), making the per-step cost extremely high, and a large amount of historical token content is irrelevant to the current decision.

The model makes hierarchical attention modifications at the structural level. In the original version, 24 layers of Gated MLA were replaced with Gated DSA, while the remaining 69 layers continue to use Kimi Delta Attention. The core mechanism of DSA is that at each step, 2048 tokens are selected from the historical sequence to participate in attention computation rather than performing full computation.

This design can be understood as a form of structured sparse attention:

Input sequence length: N
Traditional attention computation: O(N²)
DSA computation: each layer dynamically selects K=2048 relevant tokens from N tokens
Actual complexity drops substantially while retaining key historical information

The specific strategy for selecting 2048 historical tokens is critical. Common implementations include:

1. Similarity retrieval: Compute relevance scores between query vectors and historical key vectors, then take the top-K.
2. Local window retention: Always retain the local context near the current token to ensure grammatical coherence.
3. Stride sampling: Sample earlier history at equal intervals by stride to ensure temporal coverage.
4. Special token prioritization: Special segments such as tool calls, error messages, and user instructions are prioritized into the candidate set.

A reasonable sparse attention strategy allows the model to maintain access to key information at long distances within a 1-million-token context while avoiding the compute waste of full attention. However, sparse attention is not always better for every scenario. For tasks that require fine-grained comparison across multiple paragraphs of a long document, 2048 historical tokens may be insufficient; in such cases, a higher thinking tier should be set or more key segments should be retained.

Combined with the model’s 69 layers of Delta Attention, the model adopts different attention strategies at shallow and deep layers. Shallow layers can focus more on local modeling, while deep layers capture long-distance dependencies through sparse attention, forming hierarchical long-context processing capability.

4. Joint Reinforcement Learning: Long-Horizon Task Training Across Multiple Domains

The training focus of this model is not accuracy improvement on a single task but coverage of hundreds of downstream domains through large-scale joint reinforcement learning. Common reinforcement learning objectives include:

• Code repair and generation
• Terminal command execution
• Multi-step web retrieval
• Financial data analysis
• Office document processing
• 3D CAD design

The core challenge of joint training is that reward signals differ greatly across tasks. Code tasks can provide objective feedback through unit tests, document processing tasks may rely on rules or human annotation, and terminal operations require the environment to provide a success state. Simply merging data during training can easily bias the model toward domains with the densest reward signals and fastest feedback.

Common engineering approaches include:

1. Task-grouped training: Train in groups by domain first, then perform cross-domain joint fine-tuning.
2. Reward normalization: Normalize reward scales across different tasks to prevent one task type from dominating gradients.
3. Curriculum learning: Gradually transition from short-horizon single-step tasks to long-horizon multi-step tasks.
4. Trajectory filtering: Retain only successful trajectories or trajectories with valid intermediate steps to reduce noise.

The main benefit of joint reinforcement learning is task generalization. The model is no longer just a “code model” or a “document model” but a foundation model that can complete tool calls and task closure within various agent frameworks. This training approach also requires sufficient context capacity at inference time, because long-horizon tasks typically generate a large number of intermediate messages.

5. Inference Configuration: Thinking Budget and Tool Calls

At inference time, the model offers three thinking budget tiers: low, high, and max. The thinking budget controls the number of reasoning tokens generated in a single-step decision and the internal search depth.

Thinking Tier	Applicable Scenarios	Inference Cost
low	Simple queries, short-document QA, command completion	Low
high	Moderately complex code repair, multi-step retrieval	Medium
max	Long-horizon agents, complex CAD, multi-round terminal operations	High

Tier selection directly affects task success rate and inference cost. For agent workflows that require continuous tool additions, the max tier typically delivers a more stable task completion rate but increases per-step latency. Developers can perform tiered routing based on task budget: use the low tier for simple steps first, then escalate the tier for retry when anomalies or low confidence occur.

Long-horizon tool calls also require attention to message structure. During multi-round tool calls, historical thinking content and tool return messages should be fully retained to avoid the model losing state due to message truncation. A common practice is:

Fields to save:
- role: user/assistant/tool
- content: text content
- tool_calls: tool call parameters
- tool_result: tool return result
- thinking summary: optional, used to compress historical reasoning content

It is not recommended to retain only the final answer while discarding intermediate tool messages, because subsequent decisions may depend on earlier environment return values or failure records.

6. Typical Task Capabilities and Evaluation

The model is evaluated across multiple task categories, covering terminal operations, software engineering, deep code repair, complex web retrieval, CAD design, and office document QA. Its public evaluation methodology uses the maximum thinking tier, with some tasks combining a general agent framework for multi-round tool execution.

6.1 Code and Terminal Tasks

Code agent tasks typically require the model to complete the full workflow from problem understanding and file localization to patch generation and test validation within a given environment. For example, a terminal benchmark task may require the model to perform file operations, run commands, read output, and correct errors.

In such tasks, the model needs not only code comprehension ability but also the ability to parse terminal return information and decide the next step. Evaluation scores are only a reference; actual projects must also consider engineering factors such as sandbox environment stability, command permission control, and timeout settings.

6.2 Knowledge Work and Retrieval

Knowledge work tasks include web retrieval, financial research, and office document processing. The difficulties in these tasks include:

• Retrieval results may contain noise or contradictory information
• Document structures are complex and require cross-paragraph extraction and comparison
• Numerical reasoning in financial scenarios must strictly rely on source data

The model needs to combine retrieval tools for multiple queries and cross-check results. Long-context capability is critical in this scenario because multi-round retrieval results often need to be stored in context simultaneously.

6.3 3D Design and CAD

CAD tasks require the model to generate structured data that can be parsed by 3D design tools and to generate or modify geometric objects through tools. This process is typically closed-loop: the model generates parameters or scripts, the tool returns results or errors, and the model corrects based on feedback.

Such tasks place high demands on spatial reasoning and understanding of geometric constraints, and evaluation results are also easily affected by tool versions, geometry engine precision, and parameterization methods.

7. Deployment and Engineering Considerations

7.1 Weight Format and Inference Engine Compatibility

The model releases MXFP4 quantized weights and supports multiple common inference engines. During deployment, check:

• Whether the inference engine supports the MXFP4 format
• Whether it supports MoE expert sharding and dynamic loading
• Whether it provides an interface layer compatible with common message protocols

In actual deployment, the inference service is typically exposed as a chat-like interface, such as a /chat/completions-style request-response pattern. The model can also be encapsulated as an internal agent inference service for task scheduling systems to call.

7.2 Memory and Compute Planning

Due to the large total parameter scale, a single GPU cannot hold the full weights. Recommended planning approaches include:

• Use multi-GPU expert parallelism to shard different expert modules across devices
• Use weight quantization to reduce overall memory usage
• Enable sparse attention for ultra-long sequences to reduce attention computation peaks
• Estimate per-token cost and throughput based on concurrent request volume

7.3 Message Strategy and Context Management

For long-horizon tasks, context can expand rapidly. The following strategies can help maintain controllable costs:

1. Summarize and compress completed steps, retaining only necessary return values.
2. Set length limits on tool results and truncate redundant parts of overly long logs.
3. Set a task-level maximum step count, triggering human review or a degradation strategy when exceeded.
4. Store structured tool returns separately, managing them independently from natural language messages.

7.4 License and Compliance

The model weights use a dedicated license rather than a common permissive open-source license. Enterprises should carefully review when adopting:

• Whether commercial use is permitted
• Whether model outputs are restricted from being used to train other models
• Whether there are user scale or industry restrictions
• Whether open-source obligations must be fulfilled

Although such model weights are downloadable, they cannot simply be treated as completely free or usable without restriction. Legal evaluation should be completed before launch.

8. Engineering Example: Message Organization for Multi-Round Tool Calls

The following is a general message organization approach for multi-round tool calls, demonstrating the model’s interaction flow in a terminal operation task:

{
  "messages": [
    {
      "role": "user",
      "content": "Find all files in the current directory whose names contain test and whose modification time is within the last 3 days, and output the file sizes."
    },
    {
      "role": "assistant",
      "content": "I will first list the directory contents and filter file information.",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "terminal",
          "input": {
            "action": "run_command",
            "command": "find . -name '*test*' -mtime -3 -type f -ls"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "162542   4 -rw-r--r--   1 user  user  1024 Nov 3 10:22 ./test_card.jsonl\n162543   8 -rwxr-xr-x   1 user  user  4096 Nov 2 15:11 ./test_runner.sh"
    },
    {
      "role": "assistant",
      "content": "The terminal returned two files; output suggestions for subsequent processing for each.",
      "tool_calls": [
        {
          "id": "call_2",
          "type": "terminal",
          "input": {
            "action": "run_command",
            "command": "du -h ./test_card.jsonl ./test_runner.sh"
          }
        }
      ]
    }
  ]
}

In the above structure, tool call parameters, tool return content, and natural language messages together form the context. The model needs to read earlier commands and return results during multi-round interaction, so implementations should not discard tool_calls and tool messages. For ultra-long tasks, historical natural language content can be compressed while retaining key returns.

9. Applicable Scenarios and Boundaries

Such models are better suited to scenarios requiring long-cycle reasoning, multi-step tool calls, and cross-task generalization, such as:

• Code rewriting, repository-level refactoring, automated repair pipelines
• Initial online fault localization, log analysis, assisted execution of operations commands
• Financial data retrieval, research material organization, automated office document QA
• CAD script generation, geometric parameter adjustment, design iteration assistance

At the same time, note that for scenarios with strong real-time requirements, strict determinism, or high compliance requirements, model outputs should be treated as auxiliary suggestions rather than final results, with human confirmation checkpoints in place.

10. Summary

Open-source long-horizon agent models are evolving toward larger total parameters, fewer activated computations, longer contexts, and joint training across more downstream tasks. The combination of MoE, sparse attention, and joint reinforcement learning discussed in this article demonstrates a feasible technical path: use expert capacity to improve task coverage, use sparse attention to reduce long-context costs, use multi-domain reinforcement learning to enhance tool-call stability, and use thinking tiers and message management to keep the model controllable in complex environments.

For developers, understanding these mechanisms not only helps with model deployment but also guides the design of upper-layer agent frameworks. Reasonable context strategies, task decomposition methods, thinking budget settings, and tool return management often improve actual task success rates more than simply pursuing larger models.