Serving Ultra-Large MoE Models on Domestic AI Accelerators: From Architecture Analysis to Production Deployment

1. Background: Engineering Challenges of a 3-Trillion-Parameter Model

A mixture-of-experts (MoE) model with approximately 3 trillion parameters was recently open-sourced, supporting an ultra-long context window of 1 million tokens. The model has been optimized through large-scale post-training and reinforcement learning, achieving significant improvements in code generation, 3D content creation, financial research, knowledge-based QA, and long-horizon agent tasks. On code and agent benchmarks, its performance has reached or approached that of leading closed-source models — for example, scoring 65.7% on a terminal-operation benchmark and 61.2% on a long-context code benchmark, notably higher than the 55.95% of the previous generation.

Models of this scale place unprecedented demands on inference infrastructure:

• Massive parameter count: Even with FP8/INT8 quantization, storing 3 trillion parameters requires multiple terabytes of memory, necessitating multi-GPU or multi-node distributed deployment.
• Extremely long context: A 1-million-token context means KV cache memory consumption is substantial, placing strict requirements on memory management strategies.
• Sparse activation in MoE: Although each inference step activates only a subset of experts, the All-to-All communication overhead introduced by expert routing cannot be ignored in distributed scenarios.
• Latency-sensitive inference: Agent-type workloads demand both low time-to-first-token and high generation throughput, requiring joint software-hardware optimization.

To address these challenges, domestic AI accelerator vendors are actively pursuing Day-0 adaptation for large models — meaning deployment verification is completed on the same day the model is open-sourced, ensuring developers can run the latest models on domestic hardware immediately.

2. Core Concepts: Inference Characteristics of Ultra-Large MoE Models

2.1 Sparse Activation in MoE Architectures

The core idea behind mixture-of-experts models is replacing the standard FFN layer in a Transformer with multiple expert networks and using a gating router to select a small number of experts for each token. A simplified MoE layer can be illustrated as follows:

# Simplified MoE routing logic
class MoELayer(nn.Module):
    def __init__(self, num_experts, top_k, hidden_dim):
        self.experts = nn.ModuleList([
            FeedForward(hidden_dim) for _ in range(num_experts)
        ])
        self.gate = nn.Linear(hidden_dim, num_experts)
        self.top_k = top_k
    def forward(self, x):
        # Compute routing scores for each token across all experts
        router_logits = self.gate(x)  # [batch*seq, num_experts]
        routing_weights, selected_experts = torch.topk(
            router_logits, self.top_k, dim=-1
        )
        routing_weights = F.softmax(routing_weights, dim=-1)
        # Activate only top_k experts for computation
        output = torch.zeros_like(x)
        for i, expert_idx in enumerate(selected_experts):
            expert_out = self.experts[expert_idx](x[i])
            output[i] = (routing_weights[i].unsqueeze(-1) * expert_out).sum(dim=0)
        return output

This design means only a small fraction of the 3 trillion parameters is activated per inference step — typically ranging from tens of billions to hundreds of billions — keeping computation manageable while preserving model capacity. The engineering difficulty, however, lies in the fact that all expert weights must reside in memory, even if only a few are used in the current step.

2.2 KV Cache Pressure from Long Context

The KV cache challenge posed by a 1-million-token context can be illustrated with a rough estimate. Assume the model has 80 layers, 8 KV heads per layer (using grouped-query attention), a head dimension of 128, and FP8 storage:

KV cache size = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
             = 2 × 80 × 8 × 128 × 1,000,000 × 1 byte
             ≈ 163 GB

A single request can require more than 160 GB of KV cache, placing extreme pressure on memory capacity and bandwidth. Production deployments typically require a combination of PagedAttention, KV cache quantization, and paging strategies.

3. Technical Adaptation Path: Key Work in the Software Stack

3.1 Overall Adaptation Workflow

Day-0 adaptation of a large model to domestic AI accelerators generally follows this technical path:

Model analysis → Operator mapping → Graph optimization → Accuracy alignment → Performance tuning → Stability validation
      ↓               ↓                  ↓                    ↓                    ↓                   ↓
Architecture     Operator          Computation-graph     FP8/INT8         Parallel strategy     Long-duration
analysis         substitution      optimization and      quantization     and communication    stress testing
characteristics  custom operators  fusion/reordering     calibration      optimization         and regression

3.2 Core Work in Inference Path Migration

Adapting an MoE model with long-context support for inference requires the following key work at the software-stack level:

(1) Operator-level adaptation

MoE models involve many non-standard operators, including:

• Top-K selection and scatter/gather operations in expert routing
• Grouped GEMM for batched computation across different experts
• Dynamic shape handling under sparse activation

These operators need efficient implementations in high-performance compute libraries that fully leverage hardware capabilities.

(2) Memory management and optimization

Addressing the memory challenges of a 3-trillion-parameter model requires a multi-layered strategy:

Strategy	Purpose	Typical benefit
Weight quantization (FP8/INT8)	Reduce weight storage	2–4× compression
KV cache quantization	Reduce cache footprint	2× compression
PagedAttention	On-demand memory allocation	Reduces fragmentation
Expert parallelism	Distribute weights across GPUs	Linear scaling
CPU offload	Offload cold experts	Breaks single-GPU limits

(3) Communication optimization

In distributed deployment, MoE All-to-All communication and TP/PP collective operations are performance bottlenecks. The software stack needs to:

• Overlap computation with communication
• Optimize communication algorithms for specific hardware topologies
• Support combinations of parallelism strategies (TP + PP + EP + DP)

# Parallel strategy configuration example
parallel_config = {
    "tensor_parallel": 8,      # Tensor parallelism
    "pipeline_parallel": 4,    # Pipeline parallelism
    "expert_parallel": 8,      # Expert parallelism
    "data_parallel": 2,        # Data parallelism
    # Total GPUs = 8 × 4 × 8 × 2 = 512
}

3.3 Accuracy Alignment and Validation

Accuracy alignment is a critical step in ensuring correct model outputs during large-model adaptation. Common approaches include:

1. Layer-by-layer comparison: Compare intermediate layer outputs on domestic hardware against a reference implementation to locate sources of precision deviation.
2. Quantization calibration: Calibrate errors introduced by FP8/INT8 quantization to keep them within acceptable ranges.
3. End-to-end evaluation: Validate model capabilities on standard benchmarks to ensure no performance degradation after adaptation.

4. Practical Considerations and Best Practices

4.1 Pre-deployment Preparation

• Confirm model format: Check weight format (Safetensors/PyTorch Bin) and quantization scheme (native FP8 or conversion required).
• Estimate memory requirements: Calculate per-GPU memory needs based on the parallelism strategy to ensure hardware meets requirements.
• Prepare calibration data: If quantization is involved, prepare a representative calibration dataset.

4.2 Inference Service Configuration Example

# Inference service configuration example
model:
  name: "large-moe-model"
  max_model_len: 1000000
  dtype: "fp8"
parallel:
  tensor_parallel_size: 8
  expert_parallel_size: 8
  pipeline_parallel_size: 2
memory:
  kv_cache_dtype: "fp8"
  gpu_memory_utilization: 0.92
  enable_paged_attention: true
  max_num_seqs: 64
scheduler:
  max_batch_size: 256
  chunked_prefill: true
  prefill_chunk_size: 8192

4.3 Performance Tuning Directions

1. Batching strategy: Set an appropriate maximum batch size to balance throughput and latency.
2. Chunked prefill: Split long-context prefill into chunks to avoid blocking short requests.
3. Expert caching: Keep frequently activated experts resident in memory to reduce swapping overhead.
4. Communication fusion: Merge multiple small communication operations to reduce synchronization overhead.

4.4 Stability Assurance

• Long-duration testing: Run continuously for 24–72 hours, monitoring for memory leaks and performance drift.
• Failure recovery: Implement request-level retries and failover mechanisms.
• Monitoring and alerting: Set alerts for key metrics such as memory utilization, communication latency, and throughput.

5. Summary and Outlook

The open-sourcing of a 3-trillion-parameter MoE model and its Day-0 adaptation to domestic AI accelerators mark a significant step forward in the large-model support capabilities of domestic AI infrastructure. From a technical perspective, the core value of this adaptation lies in:

1. Validating the maturity of the domestic software stack: The ability to rapidly migrate inference paths for ultra-large models demonstrates that operator coverage, graph optimization, and quantization toolchains have reached engineering-level usability.
2. Accumulating deployment experience for MoE with long context: This provides a reusable methodology for adapting future models with even larger scale and longer context.
3. Strengthening the domestic AI ecosystem: Day-0 adaptation capability is key to attracting developers and a fundamental building block for a self-sustaining AI ecosystem.

For developers evaluating inference infrastructure, the following factors deserve attention:

• Speed and breadth of adaptation to mainstream model architectures
• Memory management capability in long-context scenarios
• Communication efficiency in distributed inference
• Balance between quantization accuracy and performance

As model scale continues to grow and application scenarios become more diverse, joint software-hardware optimization will be the key path to improving large-model inference efficiency. Sustained investment by domestic AI accelerator vendors in this direction will provide a stronger foundation for large-scale production deployment of AI applications.