Ultra-Large MoE Model Goes Open Source: Quantization-Aware Training and Local Deployment in Practice
This article examines the engineering practices behind a 2.8T-parameter open-weight MoE model, covering its hybrid expert architecture, gated sparse attention, MXFP4/MXFP8 quantization-aware training, weight distribution strategy, local deployment hardware requirements, inference framework integration, and benchmark performance in real-world agent scenarios.
Ultra-Large MoE Model Goes Open Source: Quantization-Aware Training and Local Deployment in Practice
In recent years, as large model parameter counts push toward the trillion scale, Mixture-of-Experts (MoE) architectures have become the industry’s mainstream approach to breaking through compute bottlenecks. A leading model team recently fulfilled its open-source commitment by releasing the full weights of its latest-generation MoE model, which has approximately 2.8 trillion total parameters, to the community. The model not only excels in long-context and multimodal processing but also topped the open-weight model leaderboard in real-world agent evaluation benchmarks. This article strips away brand-specific attributes and provides an in-depth analysis of the engineering practices for deploying models at this scale, covering底层 architectural innovations, quantization storage strategies, local deployment thresholds, and agent benchmark evaluations.
1. Underlying Architecture and Sparse Attention Mechanism
In ultra-large language models, controlling the computational overhead of long-context modeling while scaling up parameters is the core challenge in architecture design. This open-source model adopts a 93-layer deep Transformer structure, comprising 69 layers of KDA (Knowledge-intensive Dynamic Routing) and 24 layers of Gated DSA (Gated Sparse Attention).
1. Hybrid Expert Layer Design
The model’s expert layers contain a total of 896 independent expert networks. During each forward pass, the routing mechanism activates only 16 experts per token, supplemented by 2 always-shared experts. This “sparse activation + shared knowledge accumulation” design enables the model to reach 2.78 trillion total parameters while activating only 104B parameters per token, dramatically reducing floating-point operations during inference.
2. Gated Sparse Attention (Gated DSA)
Traditional full attention mechanisms incur quadratic memory growth when processing million-token sequences. In this generational evolution, the model replaces the Gated MLA layers of full attention with Gated DeepSeek Sparse Attention. Through a gating mechanism that dynamically determines which tokens need to participate in global attention computation, it effectively filters noise in long texts, significantly reducing memory and compute overhead for long contexts while preserving long-range dependency modeling capability. The model natively supports 1 million token contexts and multimodal image-text inputs.
2. Quantization-Aware Training and Memory Footprint Optimization
Weight distribution and loading for trillion-parameter models have long been engineering challenges. The model’s mixed-precision strategy for storage and activation is key to enabling deployment on single machines or small clusters.
1. MXFP4 and MXFP8 Mixed Precision
Model weights are stored in MXFP4 (Microscaling FP4) format, while activations use MXFP8. The MX series format groups multiple values into blocks that share a scaling factor, maintaining good dynamic range even at extremely low bit widths. The official team explicitly states that this is not simple post-training quantization but rather quantization-aware training. This means the model already simulated low-precision errors during the training phase, resulting in virtually no accuracy loss at final deployment.
2. Weight Volume and Distribution Strategy
Thanks to MXFP4’s extreme compression, the weight repository for 2.78 trillion parameters occupies approximately 1.56TB at the storage layer. Although downloading and disk I/O for 1.5TB of files poses a challenge for individual developers, this compression ratio has substantially lowered the hardware barrier for local deployment compared to FP16 formats that routinely exceed 5TB. Teams needing to audit, fine-tune, or privately deploy can directly obtain Safetensors-format sharded files through the open-source platform.
3. Local Deployment Hardware Assessment and Inference Frameworks
For trillion-parameter MoE models of this class, mainstream inference engines (such as vLLM and SGLang) have quickly provided adapted deployment recipes.
1. Inference Engine Adaptation and Multi-Turn Tool Calling
In terms of API compatibility, the model is highly compatible with mainstream OpenAI and Anthropic interface specifications, while also offering a high-speed endpoint reportedly achieving 2 to 3 times the throughput of the standard version. In agent applications, the model supports a reasoning_effort parameter (options: low, high, max; default: max), allowing developers to balance reasoning depth and response speed based on the scenario.
For the pain point of multi-turn tool calling, the model requires developers to pass back the previous turn’s reasoning_content and tool_calls together. This design ensures chain-of-thought continuity and prevents context breaks or instruction forgetting after multiple rounds of tool invocations. Below is an example of context assembly for multi-turn calling:
{"messages": [{"role": "user", "content": "Check today's weather and analyze whether it's suitable for outdoor sports"},{"role": "assistant","content": null,"reasoning_content": "I need to fetch weather data first, then judge based on temperature and precipitation probability.","tool_calls": [{"id": "call_01", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}}]},{"role": "tool","tool_call_id": "call_01","content": "{\"temp\": 25, \"rain_prob\": 0.1}"}],"reasoning_effort": "max"}
2. Hardware Threshold Estimation
With 104B activated parameters, combined with KV Cache and MXFP8 activation memory usage, local deployment hardware requirements remain extremely high. Third-party industry estimates indicate that a viable single-machine deployment configuration requires approximately 8 high-end GPUs (such as B300-class), with single-machine memory reaching around 2.3TB, and overall hardware costs between $550,000 and $650,000. This indicates that despite open weights, the primary audience remains enterprise R&D teams with substantial compute resources. For small and medium-sized businesses, directly calling cloud APIs or leveraging high-performance inference instances provided by platforms like Baidu AI Cloud is a more cost-effective option.
4. Performance Benchmarks in Real-World Agent Scenarios
While foundational capability evaluations are important, reliability in complex Agent tasks is the decisive factor for enterprise deployment. Third-party Agent Arena calculates a “net improvement” metric by collecting long-horizon conversation data from real users. This evaluation system not only examines whether the model can complete tasks but also comprehensively reads multi-dimensional signals including whether users explicitly confirm success, user praise and complaints, response improvements after model correction, and whether shell commands actually execute successfully.
In the leaderboard’s evaluation based on 17,500 real conversations, this model (in Max mode) achieved a +5.1% net improvement, ranking first among open models and thirteenth overall. It leads the open-source field across all three categories: Work, Code, and Chat. Particularly in the Bash reliability subcategory, the model achieved a +7.9% positive gain (its predecessor scored only -4.5%).
This leap in data fundamentally reflects a qualitative transformation in the model’s code generation accuracy and tool-calling format adherence. For automated agents that need to execute high-risk system commands, generating correctly executable Bash scripts is the cornerstone of system stability. This improvement benefits from the gated sparse attention architecture’s precise capture of long codebase contexts, as well as quantization-aware training’s complete preservation of complex logical reasoning capabilities.
5. Summary and Engineering Outlook
The open-sourcing of ultra-large MoE models marks the transition of large model technology from mere parameter stacking into deeper waters of architectural innovation, precision optimization, and engineering deployment. From this model’s design, future technical evolution will revolve around the following dimensions:
- Extreme memory compression: QAT combined with MXFP4/8 will become standard for trillion-scale model distribution.
- Low-cost long contexts: Sparse attention will gradually replace full attention layers, making million-token inference costs accessible.
- Agent-native support: Model-level built-in chain-of-thought passthrough and tool-calling context management, reducing application-layer glue code.
For developers and architects, understanding these underlying mechanisms not only enables better use of open-source weights for secondary development but also allows more precise performance and cost evaluation when selecting model inference services from large cloud platforms like Baidu AI Cloud, building highly reliable agent applications.
