Deep Dive into Open-Source Massive MoE Models: Long-Horizon Agent Architecture and Engineering Practices

Recently, an ultra-large-scale Mixture-of-Experts (MoE) model with 2.8T total parameters and 104B activated parameters was open-sourced. This model not only breaks through parameter scale barriers but is also architecturally designed to meet the demands of long-horizon agents and complex software engineering tasks. Stripping away specific product attributes, this article analyzes the design philosophy and application value of such massive models across four dimensions: underlying technical architecture, agent capability building, sparse attention optimization, and engineering deployment practices.

I. MoE Architecture Design and Native Multimodal Capabilities

In the pursuit of capability leaps in large models, relying solely on dense models leads to an exponential rise in compute costs. The MoE architecture decouples parameter scale from computation through a dynamic routing mechanism. The model discussed utilizes a 2.8T total parameter and 104B activated parameter configuration. This means that during a single inference pass, only about 3.7% of the parameters are activated, drastically reducing inference compute overhead while maintaining a high upper limit for the model’s overall knowledge capacity.

Beyond its massive parameter scale, the model natively supports a 1 million token context window and multimodal image-text inputs. Long context capability is the cornerstone of long-horizon agent operations. In scenarios involving multi-turn tool calls, code repository analysis, or lengthy document processing, the model must retain extremely long context memory. Native multimodal support allows the model to directly understand visual information like UI screenshots and CAD designs, eliminating the information loss and system complexity associated with additional external vision encoders.

II. Long-Horizon Agent and Complex Task Orchestration

The core challenges of long-horizon agents lie in multi-step task planning, state maintenance during tool calls, and dynamic adaptation to environmental feedback. Through large-scale joint reinforcement learning (RL) during its post-training phase, this open-source model has developed robust task decomposition and execution capabilities.

1. Code Development and 3D Creation Tasks

In software engineering, an agent must be capable of terminal operations, code reading, issue localization, and bug fixing. This model has demonstrated excellent performance in terminal operation benchmarks, code marathon repair tests, and CAD generation tasks. In practice, developers can use it as the core brain, integrating it with common intelligent programming frameworks to build automated workflows capable of autonomously traversing codebases, executing test cases, and submitting fix patches.

2. Agent and Knowledge Work Generalization

In general knowledge work, the model supports complex scenarios like web search, financial research, and office document processing. To balance inference speed and task depth, the model offers three thinking budget tiers: low, high, and max. Developers can dynamically configure these based on task complexity:

• Low budget: Suited for simple entity extraction or single-document summarization, prioritizing low-latency responses.
• High budget: Ideal for cross-document retrieval and preliminary analysis, balancing speed and depth.
• Max budget: Designed for deep financial research or multi-step code refactoring requiring long-chain reasoning, maximizing the utilization of model compute.

III. Sparse Attention Mechanism and Underlying Compute Optimization

Running inference with a 1 million token context on a 104B activated parameter MoE model would cause traditional global attention mechanisms to hit memory walls and compute explosions. The model introduces targeted innovations at the underlying architecture level, replacing some traditional Gated MLA layers with Gated DSA (Dynamic Sparse Attention) and mixing them with Kimi Delta Attention layers.

1. Gated DSA Dynamic Sparse Attention

The core of the DSA mechanism is “dynamic selection.” When processing the current token, instead of computing its similarity with all historical tokens, the model uses a gating network to dynamically select 2,048 of the most relevant historical tokens for attention calculation. This mechanism significantly reduces the complexity of long-text attention computation from O(N²) while preserving the extraction of critical information.

2. Joint RL and Quantization on the Training Side

On the training side, the model employs a joint RL strategy covering hundreds of downstream domains, transforming it from a mere “language model” into an “action model” capable of tool calling and logical reasoning. For deployment, an MXFP4 weight quantized version is available. By compressing weights to 4-bit floating-point numbers, MXFP4 significantly reduces the memory bandwidth and capacity required for model loading, making it possible to deploy ultra-large MoE models on standard consumer or enterprise compute clusters.

IV. Engineering Deployment and Multi-Turn Tool Calling Standards

For enterprise developers, the engineering deployment friendliness directly determines a model’s application value. It supports mainstream inference engines (such as vLLM, SGLang, and similar acceleration frameworks) and provides API interfaces compatible with major LLM providers, greatly reducing migration costs for business systems.

1. State Management for Multi-Turn Tool Calls

When building agents, multi-turn tool calls are the norm. Developers must note: in every API request, the complete historical thinking process and tool call messages must be retained. In long-horizon tasks, the model relies on previous rounds of thinking chains to assess the current execution state and plan the next action. Truncating historical messages will cause the model to lose context, leading to hallucinations or infinite loops.

Below is a structural code example demonstrating how to retain a complete tool call history:

messages = [
    {"role": "user", "content": "Analyze the financial reports of a cloud provider over the past three years and generate a summary"},
    # Model's first round of thinking and tool call request
    {"role": "assistant", "content": "Thinking: Need to retrieve financial report data.", "tool_calls": [{"id": "call_1", "function": {"name": "finance_search", "arguments": "{\"year\": 2022}"}}]},
    # Tool return result
    {"role": "tool", "tool_call_id": "call_1", "content": "{...financial data...}"},
    # Model's second round of thinking and output (depends on the previous complete history)
    {"role": "assistant", "content": "Based on the above data, we can conclude..."}
]
# Continue appending new calls and results; do not truncate earlier records

2. Enterprise Deployment Architecture Recommendations

For enterprise deployments, it is recommended to use a cloud-native container platform for elastic scaling. Given the high memory consumption of model inference, object storage can be combined to distribute and load model weights, utilizing message queues for peak shaving. Additionally, since tool calling tasks often take a long time, the frontend should adopt asynchronous streaming responses, paired with robust logging services and monitoring/alerting systems. This allows real-time tracking of the agent’s execution chain, enabling timely human intervention if the model falls into an invalid loop.

V. License Compliance and Future Outlook

Developers adopting this open-source model should pay close attention to compliance details. It uses a specific proprietary license rather than the standard Apache 2.0 or MIT protocols. Before commercialization, corporate legal and technical teams must carefully review the licensing terms to clarify commercial restrictions and open-source obligations for derivative works.

From a technological evolution perspective, the open-sourcing of a 2.8T parameter MoE model marks the official entry of LLM applications into the “long-horizon agent” era. Breakthroughs in underlying sparse attention and the introduction of tiered thinking budgets provide the industry with a new paradigm balancing performance and cost. In the future, combined with container platforms and cloud-native toolchains, ultra-large agents will unleash immense potential in high-value scenarios like automated software engineering and deep financial analysis.