Ultra-Large MoE Model Weights Open-Sourced: Architecture Evolution and Local Deployment Guide

Recently, an industry team fulfilled its open-source promise by releasing the complete weights of a Mixture of Experts (MoE) model with approximately 2.8 trillion total parameters. In real-world user session evaluations, the model ranked first among open-weight models and thirteenth on the overall leaderboard. This release provides the open-source community with robust foundational infrastructure, offering development teams a high-quality base for deep auditing, fine-tuning, and private deployment in complex scenarios such as agentic orchestration, ultra-long context processing, and multimodal understanding. This article breaks down the core technical details, analyzing the architecture design, quantization strategies, deployment barriers, and agentic evaluation.

1. Core Architecture: Deep Evolution of Sparsification and Hybrid Attention

The model has a total parameter count of 2.8 trillion, but only activates approximately 104B parameters per token during inference. This typical MoE architecture design aims to balance model capacity with inference compute consumption. According to the model specification card, its architecture has undergone iterative optimizations across multiple dimensions.

1.1 Deep Hybrid Attention Mechanism

The model contains 93 structural layers, notably featuring a mixed configuration in its attention layers: 69 layers of KDA (Kernelized Dense Attention) and 24 layers of Gated DSA (Gated Deep Sparse Attention). Unlike its predecessor, which used a full attention mechanism, this generation introduces a gated sparse attention mechanism. The core purpose of this design is to save compute power during long-context modeling. When processing inputs with millions of tokens, the memory footprint and computational complexity of full attention grow quadratically. By introducing gated sparse attention, the model dynamically determines which tokens require global computation and which undergo local aggregation. This significantly reduces long-context inference latency and memory overhead without sacrificing core contextual understanding capabilities.

1.2 Expert Routing and Multimodal Fusion

For the expert layers, the model is configured with up to 896 expert networks. Each token dynamically selects 16 experts for activation, with an additional 2 shared experts. The inclusion of shared experts ensures the stability of base feature extraction, while the high-frequency routing switching endows the model with an extremely high expressive ceiling. Furthermore, the vision component integrates a lightweight vision encoder with approximately 401M parameters, supporting image-text inputs. This decoupled design of a lightweight vision module and an ultra-large language decoder ensures that cross-modal document processing does not slow down the overall language inference speed.

2. Storage and Quantization Strategy: QAT Optimization for Memory Limits

For a model with a 2.8T parameter scale, the physical storage and VRAM loading of weights represent the first deployment barrier. Upon release, the model’s weight files occupied approximately 1.56TB. With such a massive footprint, using traditional FP16 or BF16 storage would exceed the memory capacity of almost all existing single-server machines.

To address this, the model comprehensively adopts a low-precision quantization strategy: weights are stored in MXFP4 format, and activations use MXFP8 format. It is crucial to emphasize that this is not simple Post-Training Quantization (PTQ), but rather Quantization-Aware Training (QAT).

The core of QAT lies in simulating the precision loss caused by quantization during the training phase. During forward propagation, weights and activations are forcibly truncated to low-precision representation ranges. During backpropagation, a Straight-Through Estimator (STE) is used to pass gradients. This allows the model to actively adapt to the low-precision environment during convergence, thereby maintaining minimal performance degradation even under the extreme compression of MXFP4/MXFP8. For developers, this means local deployment not only significantly reduces VRAM usage but also leverages the underlying compute acceleration instructions of next-generation hardware to improve token generation throughput.

3. Local Deployment Practice: Hardware Evaluation and Framework Adaptation

Although the weights are openly available, successfully running the model in a local environment still presents extreme engineering challenges.

3.1 Hardware Barriers and VRAM Estimation

Currently, open-source inference frameworks in the industry have陆续 introduced deployment solutions for this model. However, hardware configuration remains the core limiting factor. According to estimates from third-party developers, a viable configuration involves a professional server equipped with 8 high-performance AI accelerator cards (approximately 288GB VRAM per card, totaling around 2.3TB VRAM). Based on current hardware market prices, the server hardware cost alone falls in the range of $550,000 to $650,000. This estimate does not include high operational and energy costs. Clearly, local deployment at this scale is more suited for enterprise R&D teams with substantial compute infrastructure.

3.2 Inference Engines and Cloud-Native Deployment

Teams with sufficient compute resources can directly pull the weights and deploy them using mainstream inference acceleration engines (such as high-throughput engines or high-performance concurrency frameworks). Enterprises lacking the conditions to build their own supercomputing clusters can rely on container platforms and GPU cloud server instances from general cloud service providers for elastic deployment. Under a cloud-native architecture, combining object storage for model weight distribution, alongside logging services and monitoring alert systems to track inference latency and VRAM fluctuations in real-time, is the best practice for ensuring the high availability of large model services.

4. API Integration and Agentic Tool Calling Best Practices

For teams that do not require private auditing, an API service compatible with mainstream large model interface standards is available. In addition to the standard interface, an accelerated version is offered, with response speeds reaching 2 to 3 times that of the standard version. In complex agentic scenarios, proper parameter configuration and session management are critical.

4.1 Dynamic Control of Reasoning Depth

The API supports configuring the reasoning_effort parameter, with accepted values low, high, and max (default is max). This design allows developers to dynamically adjust the model’s thinking depth based on task complexity. For simple text summarization, setting it to low can greatly reduce latency. For complex code generation or multi-step logical reasoning, keeping it at max fully utilizes the model’s chain-of-thought capabilities.

4.2 Context Loop for Multi-Turn Tool Calling

When building agents with tool-calling capabilities, the model requires strict state management. After the model initiates a tool call, developers must pass in the tool execution result in the next request. Additionally, the reasoning_content (reasoning process content) and tool_calls (tool call instructions) generated by the model in the previous turn must be passed back exactly as they were.

Below is a simplified example of the multi-turn tool calling context structure:

{
  "messages": [
    {"role": "user", "content": "Query the weather in Beijing today"},
    {
      "role": "assistant", 
      "reasoning_content": "The user needs to obtain real-time weather data for Beijing. I should call the weather query API to get the specific information.",
      "tool_calls": [{"id": "call_001", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}}]
    },
    {
      "role": "tool", 
      "tool_call_id": "call_001",
      "content": "{\"temp\": 22, \"condition\": \"sunny\"}"
    }
  ],
  "reasoning_effort": "max"
}

This mechanism of binding the reasoning process with the session state effectively prevents the model from experiencing “identity amnesia” or “logic breaks” during long multi-turn conversations, which is key to building highly reliable agents.

5. Agentic Evaluation: From Academic Metrics to Engineering Reliability

Regarding model capability evaluation, traditional static benchmarks can no longer accurately reflect a model’s performance in complex business workflows. On the day of its open-source release, the model topped an open-model agentic arena, and the evaluation mechanism behind it is worth noting for the industry.

The arena’s data is entirely sourced from real user Agent sessions, discarding single-dimensional text generation quality scores. Instead, it evaluates the engineering reliability of the model through multi-dimensional “net improvement” signals:

1. Task Success Confirmation: Whether the user explicitly confirms task completion before the session ends.
2. Natural Language Feedback: Parsing user praise and complaint texts during interactions.
3. Error Correction Responsiveness: When a user points out an error and provides correction information, whether the model can correctly understand and adjust subsequent actions.
4. Shell Execution Feedback: In code execution agentic scenarios, whether the underlying Shell commands are actually executed successfully.

Based on statistics from 17,500 real long-horizon sessions, the model achieved a net improvement of +5.1%, specifically reaching +7.9% in the Bash reliability sub-category. This indicates that current model capability competition has shifted from simple “knowledge Q&A accuracy” to “robustness in complex task execution.” For enterprise applications, the ability to maintain stability without crashing during long-horizon tool calls is the core metric determining whether it can truly land in production environments.

Conclusion

The open-sourcing of an ultra-large MoE model with 2.8T total parameters marks the entry of the open-source large model ecosystem into a new phase of “ultra-large parameters + extremely high agentic reliability.” From hybrid gated sparse attention to quantization-aware training, and from high local deployment costs to standardized tool calling context management, every technical detail tests the engineering conversion capabilities of developers. For technical teams, deeply understanding these underlying architectures and interface specifications, combined with their own compute reserves and business scenarios to choose the appropriate model integration and deployment strategy, will be the key to building next-generation core AI applications.