Ultra-Large-Scale MoE Models: Architecture, Quantization-Aware Training, and Deployment Practices

The open-source community recently welcomed an ultra-large-scale mixture-of-experts (MoE) model with 2.8 trillion total parameters (2.8T) and roughly 104 billion (104B) parameters activated per token. The model supports million-token long contexts and image-text multimodal input, and it ranks first among open-weight models on Agent capability leaderboards. Setting aside brand-specific details, this article takes a deep look at the architectural evolution and quantization storage techniques behind this class of model, and offers developers and enterprise teams an end-to-end engineering guide covering everything from hardware selection to API integration.

1. Core Architecture Evolution: Deep Synergy Between Sparsity and Long Context

The development of ultra-large-scale MoE models has shifted from simply stacking parameters toward refined engineering of network layer structures and attention mechanisms. This class of model exhibits several notable architectural trends:

1.1 Hybrid Attention Layer Design

The model has 93 layers in total. Earlier baseline architectures typically built deep networks entirely from full-attention mechanisms (such as multi-head latent attention). However, at million-token context lengths, full-attention layers introduce compute and memory overhead that grows quadratically.

To address this, the new architecture replaces 24 full-attention layers with gated sparse attention, while the remaining 69 layers retain gated latent attention. This hybrid design preserves global feature extraction capability while substantially reducing the computational redundancy of long-context modeling, achieving more efficient context compression and retrieval.

1.2 Expert Routing and Shared Experts

At the expert layers, the model adopts a fine-grained routing strategy. Each layer contains 896 expert networks. For every input token, the router dynamically selects the 16 best-matching experts for activation, and additionally introduces 2 shared experts that process common features across all tokens.

This “sparse routing + shared experts” design brings the model’s total parameter count to 2.78 trillion, while only about 104B parameters are actually activated in a single forward pass. This effectively controls inference latency and improves the model’s generalization across multi-task scenarios.

1.3 Lightweight Multimodal Fusion

To support image-text input, the model integrates a vision encoder with approximately 400 million (401M) parameters. Compared with the massive language model backbone, the vision module is extremely lightweight in design: a projection layer aligns image features into the language model’s feature space, ensuring that multimodal input does not impose a significant performance burden on the core inference pipeline.

2. Quantization-Aware Training (QAT) and Memory Optimization

Deploying a 2.8T-parameter model on real hardware makes memory capacity the biggest obstacle. Instead of the traditional FP16/BF16, the model was released entirely with a low-bit quantization scheme.

2.1 Mixed-Precision Storage with MXFP4 and MXFP8

• Weight storage (MXFP4): Model weights are stored in a 4-bit floating-point format. In repositories on open platforms such as Hugging Face, the model occupies roughly 1.56TB, meaning developers only need about 1.6TB of storage to download and initialize the weights.
• Activation compute (MXFP8): Activations during inference use 8-bit floating point, which effectively alleviates the excessive VRAM consumption of the KV cache during long-context inference.

2.2 The Core Value of Quantization-Aware Training

Conventional post-training quantization often causes severe performance degradation on complex reasoning tasks. The model’s developers explicitly adopted quantization-aware training (QAT): during pretraining and alignment, QAT introduces quantization noise into backpropagation to simulate the truncation errors of low-precision inference, allowing the model’s weights to adapt to low-bit representation during training itself.

This “natively low-bit” strategy ensures that, at MXFP4/MXFP8 precision, the model maintains logical reasoning capability highly consistent with its high-precision counterpart when processing long documents and handling multi-turn Agent tool calls.

3. Local Deployment: Hardware Requirements and Inference Framework Support

For an ultra-large MoE model with 104B activated parameters, local deployment requires rigorous hardware evaluation and framework configuration.

3.1 Hardware Sizing Estimate

Although the weights take up only 1.56TB, actual inference also requires loading the KV cache, activations, and the CUDA context. One feasible third-party estimate of the required configuration is:

• Compute node: 8 high-end AI accelerators (B300-class), each with roughly 288GB to 320GB of memory, for a total of about 2.3TB of device memory per machine.
• Hardware cost: Hardware for servers of this class costs roughly $550,000 to $650,000. This means local deployment remains exclusive to a small number of leading enterprises and research institutions; small and mid-sized teams are better served by accessing the model through cloud APIs.

3.2 Inference Engine Support

Mainstream open-source inference frameworks have already provided support for this architecture. Developers can follow this deployment logic:

• vLLM / SGLang / TokenSpeed: These frameworks natively support MoE PagedAttention and parallel routing.
• Configuration recommendations: Enable tensor parallelism (TP=8), adjust max_num_seqs and gpu_memory_utilization according to available physical memory, and set the quantization parameter to the corresponding low-bit format when loading weights.

The following is an illustrative configuration based on a generic inference framework:

from inference_engine import AsyncLLMEngine, SamplingParams
# Initialize the engine with tensor parallelism and quantization enabled
engine = AsyncLLMEngine(
    model="/path/to/moe_model_weights",
    tensor_parallel_size=8,
    quantization="mxfp4",           # Load weights in MXFP4 format
    trust_remote_code=True,
    max_model_len=1000000,          # Support million-token contexts
    gpu_memory_utilization=0.92
)
# Build a multimodal and long-context prompt
prompt = "Analyze this 500,000-word financial report and summarize the core risks."
sampling_params = SamplingParams(temperature=0.6, max_tokens=4096)
# Generate asynchronously
outputs = await engine.generate(prompt, sampling_params)
print(outputs)

4. Agent Development and API Integration Best Practices

The model tops the open-model leaderboard on Agent Arena, indicating industry-leading multi-turn tool calling and complex planning capabilities. Developers integrating these capabilities should pay attention to the following API conventions.

4.1 Compatibility with Mainstream OpenAI and Anthropic APIs

To reduce migration costs, the API layer is typically compatible with mainstream LLM API standards. Developers can switch over by simply replacing base_url and api_key. The model name can be set to a standard version such as kimi-k3.1-pro, or to a highspeed variant for high-concurrency, low-latency scenarios (roughly 2 to 3 times faster).

4.2 Dynamic Control of Reasoning Depth

Not every step in a complex Agent task requires deep thinking. Developers can dynamically control the model’s reasoning depth through the reasoning_effort parameter.

• Options: low (fast responses), high (deep thinking), and max (extreme logical reasoning, the default).
• Use cases: Use low during intent recognition and switch to max for code generation or mathematical problem solving, striking the best balance between latency and accuracy.

4.3 Maintaining Context Across Multi-Turn Tool Calls

This is where developers most often run into trouble when building Agents. After the model decides to call an external tool (such as a search API or database query) and receives the result, the developer must pass the previous turn’s reasoning_content (reasoning process) and tool_calls (tool call plan) back as history messages when sending the next request.

If reasoning_content is omitted, the model loses the chain-of-thought context from the previous turn, causing subsequent reasoning to drift or fall into loops.

The following is a standard example of a multi-turn tool-call context structure:

{
  "messages": [
    {
      "role": "user",
      "content": "Check today's weather in Beijing for me"
    },
    {
      "role": "assistant",
      "content": "",
      "reasoning_content": "The user wants real-time weather in Beijing. I should call the get_weather tool with the location parameter set to 'Beijing'.",
      "tool_calls": [
        {
          "id": "call_001",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\": \"Beijing\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_001",
      "content": "{\"temp\": 25, \"weather\": \"sunny\"}"
    }
  ],
  "reasoning_effort": "max",
  "model": "moe-pro-highspeed"
}

5. Summary and Outlook

From the sparse architecture design of a 2.8T-parameter model, to the implementation of native quantization-aware training (QAT), to dynamic reasoning depth and comprehensive Agent workflow support, ultra-large-scale MoE models are driving LLM engineering into a new stage of maturity. For enterprise applications, although the hardware threshold for local deployment remains high, standardized cloud API access allows developers to benefit from top-tier model inference at low cost. As inference frameworks continue to deepen their optimization for low-bit compute and long contexts, there is good reason to believe that million-token-context Agent applications will achieve large-scale adoption across industries.