Open-Source MoE Agent Model for Long-Horizon Tasks: Sparse Attention, Reinforcement Learning, and Engineering Practice

1. Background: Why Long-Horizon Agents Need New Model Capabilities

Agent tasks differ significantly from single-turn Q&A. A complete agent workflow typically involves goal understanding, plan decomposition, tool invocation, environment observation, error correction, and multi-step execution. The model must maintain state consistency over long contexts, locate key information among large volumes of historical messages, and allocate reasoning resources dynamically based on task difficulty. As context length scales to the million-token level, the compute and memory overhead of conventional dense attention grows sharply, and modeling long-range dependencies becomes far more demanding.

A recently open-sourced MoE model offers a new reference implementation in these areas. The release adopts a Mixture-of-Experts architecture with 2.8T total parameters and 104B active parameters, and supports a 1M-token context window with native image-text input. Its core changes center on sparse attention, joint reinforcement learning post-training, thinking budget control, and agent-oriented engineering deployment.

2. Architecture: From Dense Attention to Sparse Attention

2.1 The MoE Architecture

Mixture-of-Experts models split feed-forward computation across multiple expert networks and activate only a subset of experts in each inference pass, reducing compute. This release has a large total parameter count but relatively few active parameters, so it retains model capacity while keeping per-token inference cost under control. The design suits multi-task, multi-tool, long-context agent scenarios: the model can switch between domain experts without activating all parameters for every token.

2.2 Sparse Attention Rework

The previous K3 release used Gated MLA in some layers. The new version replaces 24 Gated MLA layers with Gated DSA and combines them with Delta Attention in the remaining layers. The core idea of sparse attention is that the model does not need to compute attention over the entire history at every generation step; instead, it selects a subset of key tokens from the historical sequence to participate in attention.

Specifically, DSA selects 2,048 historical tokens for each attention computation. The benefits include:

• Lower computational complexity for long-context inference;
• Reduced memory footprint in long-sequence scenarios;
• A selection mechanism that preserves the history most relevant to the current step;
• Combined with Delta Attention, a balance between long-range state maintenance and local information use.

Note that sparse attention “selection” is not a simple sliding window. It typically relies on a learnable scoring mechanism or policy to retrieve relevant segments from the token history. In practice, making the selection process efficient, differentiable, or stable at inference time is the main challenge in productionizing sparse attention.

2.3 Native Text and Image Input

Native image-text input means the model can process text and images within the same sequence. For tasks such as terminal operation, CAD generation, document understanding, and web page analysis, visual signals are often essential inputs. The model encodes images into tokens that are mixed with text tokens, maintaining cross-modal alignment during attention computation. This capability also allows agents to take task context from screenshots, design sketches, and scanned documents.

3. Training: Joint Reinforcement Learning Across Domains

3.1 Why Joint Reinforcement Learning

Conventional supervised fine-tuning teaches a model to imitate a given behavior, but in multi-step agent tasks, imitation alone struggles with error recovery, exploration, and long-horizon return optimization. Reinforcement learning lets the model continuously adjust its policy through reward signals while interacting with an environment, which better fits tool calling, code repair, terminal operation, and other scenarios that require multi-round trial and error.

This release applies joint reinforcement learning covering hundreds of downstream domains. “Joint” means the model is not trained with a dedicated policy for a single task; instead, it is optimized uniformly across data and reward signals from multiple domains. The goal is generalization: stable performance across code, knowledge work, 3D design, web retrieval, and other scenarios.

3.2 Thinking Budget Control

The model provides three thinking budget levels: low, high, and max. For simple tasks or latency-sensitive scenarios, a low budget reduces reasoning steps and compute time; for complex tasks, a high budget allows more thorough planning, reflection, and verification.

This design corresponds to a variable-compute strategy during inference. The model can perform multi-step internal reasoning before producing the final answer, or output a structured thinking process. For agent tasks, the thinking budget also affects planning depth before tool calls, retry strategy after errors, and the quality of final answer synthesis.

3.3 Weight Quantization and Training Efficiency

Large-scale models face storage and deployment pressure after training. This release uses MXFP4 weight quantization, a low-bit-width floating-point format. Compared with FP16 or BF16, weight quantization significantly reduces model file size and load-time memory requirements while preserving numerical accuracy as much as possible. For models with hundreds of billions of parameters, quantized weights are considerably easier to distribute and deploy.

4. Typical Application Scenarios

4.1 Coding and Long-Horizon Engineering Tasks

In long-horizon coding agent scenarios, the model usually needs to understand repository structure, locate relevant code, edit multiple files, run tests, and fix issues based on the results. Unlike single-shot code completion, such tasks may span dozens or even hundreds of interaction rounds.

Model evaluation covers terminal operation, code repair, and software engineering tasks. According to officially published figures, the model scores 65.7 on the terminal operation benchmark, 51.7 on long-horizon software engineering tasks, and 75.5 on deep software engineering tasks. Note that these figures come from official evaluations; real-world results should be assessed against your own workloads.

A typical coding agent flow looks like this:

User request: fix an exception in a module and add tests
Step 1: Understand the request and retrieve the repository file structure
Step 2: Locate the relevant files and functions
Step 3: Read existing tests and logs
Step 4: Propose a modification plan
Step 5: Apply code edits
Step 6: Run test commands
Step 7: Continue fixing based on failure output
Step 8: Produce a final change summary

Throughout this process, the model must maintain task state, remember what has already been modified, read tool outputs, and avoid forgetting earlier information in the long context. Sparse attention can play a key role here: at each inference step, it recalls only the historical tokens relevant to the current step instead of uniformly processing the entire context.

4.2 3D Design and CAD Generation

CAD generation differs from ordinary text generation in that its output is usually structured geometry data or executable scripts. The agent must build the model step by step under design constraints, possibly invoking tools for geometric computation, parameter adjustment, and validation.

In this scenario, visual input and text instructions may appear together. For example, a user provides a sketch and asks for a 3D model, or the model needs to modify parameters based on an existing design drawing. The tool-augmented benchmark score of 92.7 indicates that when the model can call external tools, its performance on structured generation tasks improves markedly. This also confirms a broader trend: for domain-specific agents, the tool ecosystem matters as much as model capability.

4.3 Knowledge Work and Tool Calling

Knowledge work typically includes web retrieval, financial research, office document processing, and tool calling. The model must identify the question type, plan the retrieval, read search results, extract key information, and reason and compare where necessary.

Official evaluations show 92.1 on web retrieval and reasoning tasks and 64.4 on office Q&A. Office document processing often involves complex tables, unstructured text, and cross-page information integration, which is challenging. Financial research requires the model to stay consistent across documents, data tables, and reasoning chains.

5. Engineering Deployment and API Compatibility

5.1 Inference Framework Support

The open weights run on mainstream inference frameworks, including vLLM and SGLang, and support OpenAI- or Anthropic-compatible APIs. This means existing LLM applications can adopt the new model by changing configuration rather than substantially rewriting business code.

A generic request example:

{
  "model": "agent-model",
  "messages": [
    {"role": "user", "content": "Analyze the current project structure and fix a build error"}
  ],
  "max_tokens": 8192,
  "temperature": 0.2,
  "reasoning_budget": "high"
}

Whether specific parameters are supported depends on the inference framework and deployment configuration. reasoning_budget is used here only to illustrate the thinking budget concept and is not a fixed field in any particular framework. Always defer to the actual API definition of your inference service.

5.2 Multi-Turn Tool Calls and History Retention

In agent scenarios, multi-turn tool calling generates a large number of messages: user messages, model replies, tool call requests, tool results, continued model reasoning, and subsequent tool calls. To keep the model’s understanding of task state complete, the full thinking and tool-call history should be retained.

If historical messages are truncated or summarized during deployment, key intermediate states may be lost, causing the model to repeat steps it has already completed or forget earlier conclusions. It is advisable to keep the message sequence structurally intact while relying on sparse attention to reduce long-context inference cost.

5.3 License Considerations

The model weights are released under a custom license, not the usual Apache 2.0 or MIT. Before commercial integration, redistribution, or hosted deployment, review the license terms carefully and confirm they are compatible with your business model. Open weights do not mean unrestricted use; enterprise users in particular should pay attention to redistribution, service, and compliance risks.

6. Interpreting the Benchmark Results

Officially published benchmark scores are listed below. The default setting uses the max thinking level, and some evaluations use agent frameworks:

Benchmark	Score
Terminal-Bench	65.7
SWE Marathon	51.7
DeepSWE	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro	64.4

Scores should be interpreted in the context of each task. Terminal operation and long-horizon software engineering scores remain relatively low, indicating substantial room for improvement in complex, open-ended scenarios that require long-term planning. The high scores on browsing/retrieval and tool-augmented CAD tasks show that when the model can leverage external tools, it performs better in structured environments. The relatively lower office Q&A score may be related to complex document structure, dispersed information, and strict grading criteria.

In addition, different evaluations use different agent frameworks, which introduces variation in orchestration strategies, prompt templates, and tool interfaces. These numbers are therefore better suited for understanding capability trends than as a single basis for precise cross-model comparison.

7. Practical Recommendations

For teams planning to integrate this type of model into long-horizon agent applications, the following points are worth considering:

1. Assess task length and context requirements. If your tasks are mostly short Q&A, sparse attention offers limited benefit; if they involve multi-turn tool calling and long-document processing, this architecture is worth evaluating.

2. Configure thinking budgets by task difficulty. Setting differentiated budgets for requests of different priority and complexity balances response speed and output quality. High budgets suit code repair and research analysis; low budgets suit simple information extraction and conversation.

3. Retain full message history. Do not casually compress or delete historical tool returns in multi-turn tool calling. Even with long-context support, keep the message structure consistent to avoid losing key information.

4. Check the model license. Before commercial use, confirm what deployment, distribution, and service modes the license permits to avoid compliance risk.

5. Evaluate on your own data. Official benchmarks provide an initial reference, but real business environments often involve private data, custom tools, and domain-specific language. Compare the base model and fine-tuning strategies on an internal evaluation set.

6. Plan deployment resources. Although MoE and weight quantization reduce inference cost, 2.8T total parameters still impose significant memory requirements. When using multi-GPU deployment, tensor parallelism, or quantized inference, verify that throughput and latency meet your business targets.

8. Conclusion

This open-source release represents a comprehensive evolution of long-horizon agent models across architecture, training, and deployment: MoE to control compute scale, sparse attention to handle million-token contexts, joint reinforcement learning to cover multi-domain tasks, thinking budgets to balance quality and latency, and compatible APIs plus quantized weights to lower the barrier to adoption.

From an engineering perspective, the difficulty of long-horizon agents lies not only in the model itself but also in environment interaction, tool calling, state management, and evaluation systems. Gains in model capability must be combined with solid agent frameworks and deployment practices to deliver real business value.