Evaluating Long-Horizon Task Capabilities in an Open-Source MoE Model: Architecture and Engineering Practices

Introduction: The Growing Demands of Long-Horizon Tasks

As use cases in AI agents, automated programming, document processing, and 3D design continue to expand, the industry is placing higher demands on language models in terms of context length, tool-calling reliability, and multi-step reasoning. Traditional models perform reasonably well on short-text Q&A, but when tasks involve hundreds of thousands of tokens of historical information, multiple tool callbacks, and cross-domain reasoning, they often suffer from information loss, instruction drift, or unstable outputs.

In this context, sparse expert models have emerged as an important technical approach for long-horizon tasks. By scaling total parameter count while reducing the number of activated parameters per inference, these models can improve expressive capacity and knowledge coverage without significantly increasing computational cost. At the same time, the introduction of joint reinforcement learning, sparse attention mechanisms, and low-bit quantization has further enhanced the models’ usability in real-world production environments.

This article analyzes a recently released open-weight sparse expert model, drawing on its public evaluation results and engineering practices, to examine the key capabilities and deployment pathways of models designed for long-horizon tasks.

Model Architecture: Sparse Experts and Long-Context Support

The model employs a sparse expert architecture with approximately 2.8T total parameters and around 104B activated parameters per inference. The total parameter count determines the model’s knowledge capacity and multi-domain coverage, while the activated parameter count directly influences computational overhead during inference. Through the sparse activation mechanism, only a subset of experts is invoked in each forward pass, striking a balance between performance and cost.

On context length, the model supports an input window of approximately 1 million tokens, allowing it to accommodate very large codebases, multi-turn tool invocation logs, extensive document collections, or complex engineering logs. Long-context capability is especially critical for agentic scenarios, where the system must simultaneously retain user instructions, intermediate reasoning steps, tool results, and environmental state. Without a sufficiently large context window, the system would be forced to frequently truncate or compress historical information, leading to loss of key details and ultimately affecting task completion rates.

For input modalities, the model natively supports image-text input. This provides a foundation for tasks such as document comprehension, UI screenshot analysis, and processing of design and CAD drawings, while also enabling agents to receive richer environmental signals beyond plain text.

Attention Mechanism: From Gated MLA to Gated DSA

Compared to the previous generation, this version replaces 24 layers of Gated MLA with Gated DSA, combined with 69 layers of Kimi Delta Attention. DSA selects 2,048 tokens from the historical sequence for each attention computation, rather than performing full attention over all historical tokens.

The primary purpose of this design is to reduce computational complexity in long sequences. For a 1-million-token context, full attention would cause a dramatic increase in both computation and memory usage. Sparse attention significantly reduces the computational load by selecting the most relevant historical tokens, without substantially compromising information. The selection mechanism is typically driven by learnable parameters or similarity scoring, and the model gradually learns during training which historical information is most important for current reasoning.

The “Gated” component in Gated DSA means the attention module incorporates a gating mechanism that dynamically adjusts information flow. This design allows the model to automatically weigh trade-offs between long-range dependencies and local details, making it well-suited for multi-turn tool invocation and long-document reasoning. It is worth noting that upgrading the attention mechanism often requires corresponding adjustments in training procedures and inference framework adaptation; otherwise, the full performance potential may not be realized.

Joint Reinforcement Learning and Quantization-Aware Training

The model incorporates joint reinforcement learning covering hundreds of downstream domains during its training phase. The term “joint reinforcement learning” refers to a training paradigm that goes beyond reward signals from a single task, simultaneously introducing multi-dimensional rewards such as code execution results, tool-call correctness, document Q&A accuracy, and format compliance. This enables the model to learn transferable reasoning and decision-making strategies across tasks.

This training approach is particularly important for long-horizon tasks. Taking code repair as an example, a single generation accuracy metric is insufficient to evaluate model quality; other factors must be considered, such as whether the patch passes tests, whether it introduces new issues, and whether it follows project coding conventions. Through joint reward modeling, the model can progressively learn more robust strategies.

Additionally, the model uses MXFP4 weight quantization during training. MXFP4 is a low-bit floating-point format that maintains numerical range while reducing storage and memory access pressure. Unlike post-training quantization, quantization-aware training allows the model to adapt to low-precision representations during the training phase, thereby minimizing precision loss. For a model with 2.8T total parameters, the memory and bandwidth savings from quantization are substantial and represent a critical component in engineering deployment.

Task Capabilities and Thinking Budget Design

The model is primarily designed for three categories of tasks: code development and 3D creation, agentic and knowledge work, and long-horizon tool invocation.

Code Development and 3D Creation

In code development, the model covers terminal operations, code defect repair, and CAD generation. Terminal operations require the model to understand command-line environments, filesystem states, and error output, while completing complex tasks through multi-turn interactions. Code repair requires the model to identify root causes of defects, generate patches, and validate results. CAD generation involves structured geometric expression and design-intent understanding, where the model must translate natural language descriptions into executable modeling operations.

These tasks share common characteristics: multiple steps, delayed feedback, and the need for tool coordination. Only by maintaining an accurate understanding of the global objective at every step can the model avoid drifting off course midway. Long-context and sparse attention mechanisms play a crucial role here.

Agentic and Knowledge Work

Agentic scenarios include web search, financial research, and office document processing. The model must be capable of invoking external tools such as search APIs, database queries, or spreadsheet processing functions. Unlike pure text generation, agentic tasks require the model to initiate tool calls at appropriate times and adjust subsequent reasoning based on returned results.

The reasoning capability offers three tiers of thinking budgets: low, medium, and high. Each tier corresponds to different reasoning depth and computational cost. Simple tasks can use the low tier to reduce latency, while complex tasks benefit from the high tier to allow more thorough reasoning. This design gives developers flexibility to configure the trade-off between effectiveness and cost, and also facilitates tiered deployment across different business scenarios.

Message Management in Long-Horizon Tool Invocation

In multi-turn tool invocation scenarios, the system must retain complete reasoning history and tool-call messages. This is because the model relies on past state to decide subsequent actions. If intermediate messages are truncated or simplified, the model may lose key data from tool returns or prior reasoning conclusions, affecting its subsequent behavior.

On the engineering side, message sequences typically require structured storage that distinguishes between user messages, model reasoning, tool invocation requests, and tool return results. The inference framework should support dynamic context expansion and employ appropriate compression or summarization strategies for extremely long sequences, rather than simple truncation.

Interpreting Evaluation Results

The official release includes evaluation results across different tasks, using the highest thinking tier by default, with some scenarios using a general agent framework. Key metrics are as follows:

• Terminal task evaluation: approximately 65.7
• Software engineering marathon task: approximately 51.7
• Deep software engineering evaluation: approximately 75.5
• Browse and retrieval evaluation: approximately 92.1
• CAD generation with tools: approximately 92.7
• Office Q&A professional evaluation: approximately 64.4

Several observations emerge from these figures. First, the model performs strongly on structured tasks that require tool integration, such as CAD generation and browse-and-retrieve. Second, scores on software engineering marathon tasks are relatively lower, reflecting the difficulty of real-world long-horizon coding tasks and indicating room for improvement on complex projects. Third, the significant variance across domains suggests the model’s capabilities are not uniform, and developers should conduct supplementary evaluations aligned with their specific business scenarios.

It should be emphasized that evaluation scores are influenced by multiple factors including the evaluation set, prompting strategy, tool framework, and sampling parameters. They should not serve as the sole basis for model selection. A more sound approach is to build internal evaluation sets that closely mimic production use cases and compare against baseline models under identical conditions.

Deployment and Engineering Recommendations

The open weights are provided in MXFP4 format and support multiple mainstream inference frameworks, with APIs compatible with common protocols. This is particularly important for developers who want to integrate the model into existing systems quickly. API compatibility means upper-layer applications can switch model services without significant changes, reducing migration costs.

For deployment, the following areas require careful attention:

Memory and Throughput Planning

A model with 2.8T total parameters requires substantial memory capacity even after quantization. For scenarios where a single GPU cannot accommodate the model, multi-GPU tensor parallelism or pipeline parallelism is typically employed. The choice of inference framework affects memory utilization and throughput, and actual stress testing on target hardware is recommended.

Context Caching Strategy

In long-context scenarios, reusing historical key-value caches can significantly reduce time-to-first-token. For multi-turn dialogue or agentic tasks, maintaining cache continuity is important to avoid recomputing the full prefix on every request.

Thinking Budgets and Task Tiering

In production environments, different thinking tiers can be assigned based on task complexity. For example, simple Q&A uses the low tier, complex code repair uses the high tier, and dynamic routing can select the appropriate tier. This approach maintains user experience while controlling overall computational costs.

Message Logging and State Management

Long-horizon agentic tasks should establish reliable message persistence mechanisms that record inputs, outputs, tool calls, and external returns at every step. This serves both debugging purposes and provides training data for future model improvements.

Licensing and Compliance Considerations

The model weights are released under a dedicated license rather than a permissive open-source license. Users must carefully review the license terms to confirm whether commercial use is permitted, whether any restrictive conditions apply, and whether specific attribution requirements must be followed. Enterprises should have legal and compliance teams assess licensing risks before adoption to avoid future disputes.

For technical teams planning to maintain the model asset long-term, understanding both the weight license and data usage terms is equally important. Some licenses may restrict how model outputs can be used or require preserving notices in derivative models. These details should be clarified before project initiation.

Conclusion

Long context, tool invocation, and multi-step reasoning are becoming core capabilities for next-generation language models. This article has examined an open-source sparse expert model across architecture design, attention mechanisms, joint reinforcement learning, quantization-aware training, and task evaluation. By reducing inference cost through sparse expert structure, supporting ultra-long contexts through sparse attention, and combining multi-tier thinking strategies with general-purpose tool interfaces, the model demonstrates practical value in code development, agentic tasks, and knowledge work.

For developers and enterprise users, model selection and deployment should not rely solely on public benchmark scores. Task-specific characteristics, deployment resources, and licensing compliance must all be factored into the decision. Only with well-designed engineering pipelines, message management, inference scheduling, and cost control can the full value of long-horizon models be realized. As training methods, attention mechanisms, and inference frameworks continue to evolve, long-context models built for agentic scenarios will gradually move toward broader industrial adoption.