Open-Source Massive MoE Model: Deep Dive into Long-Horizon Agents and Sparse Attention

As Large Language Models (LLMs) evolve from single-turn conversations to the automated execution of complex tasks, long-horizon agents capable of handling ultra-long contexts and robust tool calling have become a critical breakthrough in the industry. Recently, an open-source community released the weights of a next-generation LLM. This model adopts a massive Mixture of Experts (MoE) architecture with 2.8T total parameters and 104B activated parameters, supporting a 1 million token context window and native multimodal inputs (text and image). This release provides not only the foundational weights but also introduces an improved sparse attention mechanism at the architectural level, establishing a new infrastructure for long-horizon tasks such as terminal operations, complex software engineering, and 3D design.

1. Massive MoE Architecture and Native Long Context Capabilities

The model is built on an MoE (Mixture of Experts) architecture with 2.8T total parameters and 104B activated parameters. In an MoE architecture, a routing network dynamically assigns each token to the most appropriate expert sub-network for computation. The 2.8T total parameter count equips the model with vast knowledge capacity and representation capabilities, while the 104B activated parameters effectively control the computational complexity during inference, making it feasible to run ultra-large-scale models on limited hardware resources.

Furthermore, the model natively supports a 1 million token context length and image-text inputs. For long-horizon agents, complex tasks often entail extensive interaction histories, massive retrieval results, and multimodal environmental feedback. Native million-level context handling avoids the information truncation issues associated with traditional sliding windows or external vector retrieval, allowing the model to maintain state consistency and logical coherence throughout a complete task lifecycle.

2. Long-Horizon Agent Capabilities and Application Practices for Complex Scenarios

During the post-training phase, the model underwent large-scale joint reinforcement learning covering hundreds of downstream domains, with a focus on enhancing its agent capabilities in software engineering, 3D creation, and knowledge work.

Software Engineering and Code Development

In terminal operation and code repair scenarios, the model must not only understand code logic but also translate it into specific operational steps. Evaluation data shows that it scored 65.7 on terminal operation benchmarks and 51.7 on long-horizon software engineering tests. In practical engineering applications, developers can integrate it into intelligent coding tools. Through multi-turn dialogue, the model can complete a closed-loop process: from requirement analysis and code generation to terminal compilation, error reading, and automated repair.

3D Creation and CAD Generation

3D design and CAD generation are tasks heavily reliant on spatial imagination and parametric modeling. The model supports multi-step tasks in conjunction with professional tools, achieving a score of 92.7 on a CAD generation benchmark with tool calling. This demonstrates its ability to effectively understand geometric constraints and translate them into precise API call instructions for underlying rendering or modeling engines.

Knowledge Work and Tool Calling

For knowledge-intensive tasks such as financial research, web search, and office document processing, the model provides comprehensive tool calling support. During inference, the model offers three levels of thinking budgets: low, high, and max. This design allows developers to dynamically allocate resources based on task latency and depth requirements: using the low budget for rapid responses in simple information extraction, or enabling the max budget for complex research report analysis, giving the model sufficient internal reasoning space to balance computational cost and output quality.

3. Underlying Breakthroughs: Sparse Attention Mechanism and Training Optimization

The traditional pain point of processing a 1 million token context is the quadratic complexity explosion in attention computation. The model’s architecture has undergone deep sparse transformation to balance long context and computational efficiency.

Compared to previous versions, the new architecture replaces 24 layers of Gated MLA (Gated Multi-head Latent Attention) with Gated DSA (Gated Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention. In the DSA mechanism, instead of computing attention for every token against all historical tokens, the system dynamically selects the 2,048 most relevant historical tokens for each computation. This sparse selection mechanism significantly reduces the memory footprint and computational consumption for long sequences, while the gating mechanism ensures the retention of critical information.

At the training and inference deployment level, the model adopts MXFP4 weight quantization technology. MXFP4 is a microscaled floating-point format that represents weights in 4 bits. Without severely compromising model precision, it significantly reduces the memory footprint of model weights. This alleviates the communication bandwidth pressure during multi-card distributed inference of ultra-large-scale MoE models, thereby improving overall throughput.

4. Engineering Deployment and Multi-Turn Tool Calling Practice Guide

For enterprise deployment, the model releases MXFP4 quantized weights and natively supports mainstream inference acceleration frameworks. During deployment, developers should focus on the following engineering details:

Inference Framework Compatibility

The model is compatible with mainstream open-source inference engines and provides standardized OpenAI/Anthropic compatible APIs. This means developers can replace models via standard HTTP interfaces without extensively modifying existing business gateways. For cloud resource selection, enterprises can leverage general-purpose container platforms combined with elastic GPU resource pools, utilizing automated monitoring and alerting systems to dynamically scale inference nodes based on memory usage and concurrent throughput.

State Management for Multi-Turn Tool Calling

The core of a long-horizon agent lies in multi-turn tool calling. During API interactions, developers must retain the complete history of thoughts and tool call messages. If historical tool call records are truncated midway, the model will lose its perception of the environmental state, leading to infinite loops or incorrect output instructions. It is recommended to build a session ID-based message state machine on the client side, or introduce a general message queue to persist the intermediate states of long-horizon tasks.

Below is an example of a multi-turn tool calling structure based on a standard API (pseudocode):

import requests
# Initialize the Agent session
session_messages = [
    {"role": "user", "content": "Please search for the latest industry research reports, extract key financial data, and generate a summary."}
]
def call_model_api(messages, budget="max"):
    # Set the thinking budget and multi-turn tool calling history
    payload = {
        "model": "open-source-moe-2.8t",
        "messages": messages,  # Must include the complete history of thoughts and tool calls
        "thinking_budget": budget,
        "tools": [{"type": "function", "function": {"name": "search_web", "parameters": {}}}]
    }
    # Request the OpenAI-compatible API endpoint
    response = requests.post("http://localhost:8000/v1/chat/completions", json=payload)
    return response.json()
# First round call: The model decides to call a tool
response = call_model_api(session_messages)
assistant_msg = response["choices"][0]["message"]
# Append the model's tool calling decision to the history; this step is mandatory
session_messages.append(assistant_msg)
# The developer locally executes the tool and gets the result
tool_result = execute_search_web(query="2024 industry research reports")
# Append the tool result to the context
session_messages.append({
    "role": "tool",
    "tool_call_id": assistant_msg["tool_calls"][0]["id"],
    "content": str(tool_result)
})
# Second round call: The model generates the final inference based on the tool result
final_response = call_model_api(session_messages, budget="high")
print(final_response["choices"][0]["message"]["content"])

5. Open Source License and Compliance Considerations

It is important to note that this model uses a specific proprietary open-source license, rather than the commonly used Apache 2.0 or MIT licenses. For enterprise users, before integrating the model weights into commercial products or redistributing them, the legal team must strictly review the commercial restriction clauses and derivative open-source requirements in the proprietary license to mitigate potential compliance risks.

In summary, this newly open-sourced massive MoE model, through dynamic sparse attention and large-scale reinforcement learning, pushes the engineering boundaries of LLMs into the era of truly long-horizon autonomous agents. When integrating the model, developers and enterprises should fully utilize its tiered thinking budgets and standardized APIs, while properly managing long-context states and optimizing underlying memory to maximize its potential for executing complex tasks.