Open-Source Practice for a 2.8T MoE Model and Technical Analysis of Long-Horizon Agent Capabilities

1. Introduction: The Technical Context for Open-Sourcing Large-Scale MoE Models

As large language models continue to scale, the tension between training cost and inference efficiency in dense models has become increasingly pronounced. The Mixture of Experts (MoE) architecture, with its sparse activation property, has emerged as a mainstream approach for maintaining very large total parameter counts while keeping per-token inference compute manageable. Recently, an MoE model with a total parameter size of 2.8T and approximately 104B activated parameters was released with open weights, drawing significant attention across the industry. The model demonstrates strong capabilities in long-horizon Agent tasks, multi-step tool calling, code generation, and 3D design workflows. Its architectural innovations, training methodology, and deployment strategy merit close examination.

This article provides a comprehensive technical review of the model’s architecture design, attention mechanism evolution, reinforcement learning post-training strategy, inference and deployment ecosystem, representative application scenarios, and licensing compliance considerations.

2. Model Architecture: Core Design of the 2.8T MoE Model

2.1 Overall Parameter Scale and Sparse Activation

The model adopts an MoE architecture with a total parameter count of 2.8 trillion, while each forward pass activates only about 104 billion parameters. This means the model possesses a massive knowledge capacity, yet the actual per-token computational cost at inference is far lower than a dense model of equivalent total size. The core idea of MoE is to split the feed-forward network layers into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts for each input token. During training, this design introduces communication overhead for expert load balancing, but at inference time, careful deployment strategies can significantly reduce latency.

2.2 Evolution of Gated Linear Attention

At the attention mechanism level, this model introduces an important change from its predecessor: the original 24 layers of Gated MLA (Gated Multi-head Latent Attention) are replaced with Gated DSA (Gated Dynamic Sparse Attention). The key idea of DSA is that within each attention head, only the 2048 most relevant historical tokens are dynamically selected for attention computation, rather than attending to the full context as in conventional full attention. When processing extremely long contexts such as 1 million tokens, this design reduces attention computational complexity from O(n²) to approximately O(n·k), where k=2048 is the number of selected historical tokens, thereby supporting ultra-long sequence processing both theoretically and practically.

At the same time, the model retains 69 layers using Kimi Delta Attention. This hybrid arrangement of DSA plus Delta Attention balances the efficiency gains of sparse selection with the expressive power of full attention in specific layers.

2.3 Million-Token Context and Native Multimodality

The model natively supports a context window of 1 million tokens, which is critical for long-horizon Agent tasks. An Agent needs to maintain a complete chain of thought, multiple tool call results, and intermediate states within a single conversation session. Additionally, the model has native image-text input capability without requiring a separate visual encoder. It can directly process joint image and text inputs, laying the foundation for multimodal tasks such as CAD generation and UI understanding.

3. Training Methodology: Joint Reinforcement Learning and Quantization Strategy

3.1 Joint Reinforcement Learning Across Hundreds of Domains

Unlike traditional supervised fine-tuning (SFT) or single-domain reinforcement learning, this model employs a joint reinforcement learning post-training strategy, with training data covering hundreds of downstream domains. The overall pipeline can be summarized as follows:

1. Base pre-training: Autoregressive pre-training on large-scale general corpora to establish foundational language capabilities.
2. Joint reinforcement learning: Simultaneous RL optimization across multiple task categories, including code repair, terminal operations, financial analysis, and document processing. The reward model incorporates multi-dimensional signals such as task completion, tool-calling correctness, and reasoning step validity.
3. Thinking budget tiers: During training, three thinking budget levels are introduced: low, high, and max. The model learns to allocate reasoning effort dynamically according to task complexity, avoiding excessive computation on simple tasks while ensuring sufficient reasoning depth for complex ones.

The advantage of this joint training approach is that the model does not overfit to any single domain. Instead, it develops cross-domain general reasoning and planning capabilities, which are precisely what long-horizon Agent applications require.

3.2 MXFP4 Weight Quantization

To lower deployment barriers and reduce memory footprint, the model weights are released in MXFP4 (4-bit floating point) format. Compared with FP16 or BF16, 4-bit quantization can reduce memory requirements by roughly 4x, making it feasible to load a 2.8T-parameter model on limited hardware resources. It is important to note that quantization introduces some precision loss. In production deployments, quantization-aware fine-tuning or post-quantization calibration should be considered to control quality degradation.

4. Inference and Deployment Ecosystem

4.1 Flexible Configuration of Thinking Budgets

The model provides three thinking budget tiers (low / high / max), allowing developers to configure inference behavior based on task characteristics:

• Low tier: Suitable for lightweight tasks such as simple Q&A and format conversion, with fast response times.
• High tier: Suitable for moderately complex analysis and code generation.
• Max tier: Suitable for deep reasoning scenarios such as multi-step planning, complex code repair, and long tool-calling chains.

The official benchmark results are reported using the max thinking tier by default, meaning the model’s reasoning potential is fully exercised under high thinking budgets.

4.2 Framework Compatibility

The released MXFP4 weights are compatible with several mainstream inference frameworks, including widely used high-performance inference serving engines and distributed inference frameworks. The release also includes service interfaces compatible with common LLM API protocols, allowing existing applications to migrate quickly.

4.3 Message Management for Multi-Round Tool Calling

In long-horizon Agent scenarios, the model performs multiple rounds of tool calling. Each round involves reasoning output, tool call requests, tool return results, and other messages. A key practice is that the complete history of reasoning chains and tool-call messages must be preserved, rather than truncated or summarized. This ensures that the model can trace back the full context in subsequent steps and make correct decisions based on the global state.

5. Representative Application Scenarios

5.1 Long-Horizon Code Agents and Software Engineering

For tasks involving terminal operations, code repair, and multi-file refactoring, the model exhibits strong Agent capabilities. Taking terminal operations as an example, the model needs to understand command-line output, determine the next action, execute new commands, and dynamically adjust its strategy based on results. This type of task is essentially a multi-step Markov decision process that requires long-term memory and state-tracking capabilities.

In code repair scenarios, the model can not only locate bugs but also generate patches, run tests, and iteratively refine changes based on test feedback, forming a complete repair loop.

5.2 3D Design and CAD Generation

With tool-calling capabilities, the model can drive CAD tools to complete 3D model generation. This requires the model to understand geometric description languages or API call protocols and translate natural language intent into precise parametric design instructions.

5.3 Knowledge Work and Financial Research

In scenarios such as web search augmentation, financial data analysis, and office document processing, the model obtains real-time information through tool calls and then performs comprehensive analysis and report generation. The tiered thinking budget design allows developers to strike a balance between cost and quality.

6. Official Benchmark Results

The following table summarizes the officially published results across various tasks, using the max thinking tier by default:

Benchmark	Score	Task Type
Terminal-Bench 4.0	65.7	Terminal operation Agent
SWE Marathon v1.1	51.7	Software engineering Agent
DeepSWE v1.1	75.5	Code repair Agent
BrowseComp	92.1	Web browsing and retrieval
BenchCAD (with tools)	92.7	CAD generation
OfficeQA Pro v2	64.4	Office document processing

Some evaluations used commonly adopted Agent frameworks to drive the model’s task execution. The scores reflect the model’s overall performance within an Agent-based paradigm.

7. License and Compliance Considerations

It is important to note that the model weights are released under a custom license, not under common open-source licenses such as Apache 2.0 or MIT. Developers should carefully review the license terms before use, paying particular attention to restrictions on commercial use, derivative works, and redistribution to ensure compliance.

8. Summary and Outlook

The open release of a 2.8T-parameter MoE model marks a significant step in moving large-scale sparse models from research into engineering practice. The Gated DSA plus Delta Attention hybrid architecture, joint RL post-training, MXFP4 quantization, and three-tier thinking budget design collectively provide a solid technical foundation for long-horizon Agent applications. For developers, the key considerations lie in understanding the deployment challenges introduced by sparse activation, the trade-off between quantization precision and efficiency, and how to properly design message management and tool-calling pipelines in Agent scenarios. As more large-scale open models emerge, building efficient, reliable, and compliant agent systems will become a central engineering challenge in the next phase.