Ultra-Large MoE Models: Architecture and Cloud Deployment Guide
This article uses a model with roughly 2.8 trillion total parameters and about 104 billion active parameters per inference as an example to explain the architecture of sparse mixture-of-experts models and the main constraints on cloud deployment. It covers weight memory, All-to-All communication, expert load imbalance, multidimensional parallelism, quantization estimates, storage and network considerations, inference engine configuration, and a sample API call for an Agent-style application.
Introduction
Mixture of Experts (MoE) models have recently been released with open weights. This article uses a model with approximately 2.8 trillion total parameters and about 104 billion active parameters per inference as an example. The capability ceiling of such models continues to rise, and deployment difficulty rises with it. This article discusses architecture principles, memory and communication bottlenecks, cloud deployment methods, and Agent application design in sequence. The numbers here are mostly estimates; actual results depend on model version, quantization method, hardware, and network environment.
1. Basic Principles of MoE Architecture
MoE models use sparse activation to break the linear binding between parameter count and compute. In a standard Transformer, every token passes through a feed-forward network (FFN) layer. MoE architecture replaces this layer with multiple parallel “expert” networks. A gating network first computes the match between the current token and each expert, then selects the Top-K experts with the highest scores for computation, while the remaining experts remain idle.
Therefore, total model capacity grows with total parameter count, while compute per forward pass is mainly determined by active parameters. In the example above, only about 104 billion of the 2.8 trillion total parameters participate in a single computation. Compute is close to that of a 100-billion-class dense model, but storage cost remains at the 2.8-trillion-class level.
Regarding capability, some public tests have shown that ultra-large MoE models perform well on complex multi-step tasks. One possible explanation is that more parameters can hold richer knowledge, and knowledge from different domains can be more easily distributed across different experts. However, specific evaluation scores depend on model version, test set, and evaluation conditions. Readers should refer to the model’s official evaluation reports and should not directly apply conclusions from a single leaderboard.
2. Main Bottlenecks in Memory and Communication
1. Memory Occupied by Weights
BF16 is the bfloat16 format, with 16 bits per parameter, or 2 bytes. It is not double-precision floating point (FP64, 8 bytes), and the two cannot be mixed. At 2.8 trillion parameters and 2 bytes per parameter, weights alone require about 5.6 TB of memory. A single 8-GPU server with 80 GB per GPU totals 640 GB, which is far from enough to hold the full weights. Therefore, deployment must rely on a cross-node distributed cluster.
5.6 TB is only the lower bound. KV cache, activations, framework overhead, and communication buffers all consume additional memory, so actual demand will be higher.
2. Cross-Node Communication
MoE inference includes All-to-All communication. The hidden state of a token must be sent to the GPU where its routed expert resides. When experts are distributed across different nodes, this data must traverse the inter-node network. Under expert parallelism (EP), communication can become a major source of latency, but the specific share depends on network topology, bandwidth, batch size, and load distribution. When bandwidth is insufficient or the topology is poorly designed, communication time may exceed compute time.
3. Expert Load Imbalance
Different experts have different probabilities of being selected. Experts that handle general syntax or high-frequency knowledge may become “hot experts,” and their GPUs remain saturated for long periods while other GPUs are relatively idle, reducing overall throughput. Common mitigations include auxiliary load-balancing loss during training, replica scaling for hot experts during inference, and request scheduling optimization.
3. Cloud Deployment and Optimization
1. Multidimensional Parallelism Combinations
Large-scale deployment usually combines three parallelism strategies:
- Expert parallelism (EP): distribute different experts across different GPUs.
- Tensor parallelism (TP): split matrix computation within a single expert, usually confined to a single node to use high-speed interconnect.
- Pipeline parallelism (PP): divide model layers across different nodes, suitable for cross-node scaling.
The combination must be adjusted according to network conditions. When intra-node bandwidth is high, TP can be increased. When inter-node bandwidth is limited, cross-node communication should be reduced.
2. Quantization Estimates
Quantization can reduce weight storage. For 2.8 trillion parameters, the theoretical memory occupied by weights alone is as follows:
| Precision | Bytes per Parameter | Theoretical Weight Occupancy |
|---|---|---|
| BF16 | 2 | About 5.6 TB |
| INT8 | 1 | About 2.8 TB |
| INT4 | 0.5 | About 1.4 TB |
These are ideal values. Actual occupancy also includes KV cache and activations. Quantization introduces accuracy loss, and the magnitude depends on the quantization method (such as AWQ or GPTQ), the representativeness of calibration data, and the task type. Before deployment, evaluate accuracy on your own business data rather than looking only at the theoretical compression ratio.
3. Cloud Storage and Network
Object storage and file storage can host TB-scale weight files. Storage services from platforms such as Baidu AI Cloud can serve as locations for weights. During loading, distributed cache prewarming can distribute weights to compute nodes in advance and reduce cold-start wait. On the network side, high-speed protocols such as RoCE can help reduce All-to-All latency. Availability and performance depend on the actual specifications and test results of the chosen cloud service.
4. Inference Engine Configuration
Inference engines optimized for MoE typically support dynamic batching and PagedAttention. Each has its applicable conditions:
- Dynamic batching improves GPU utilization but increases queuing time per request, making it suitable for throughput-first scenarios.
- PagedAttention manages KV cache in pages and can reduce memory fragmentation, making it suitable for long-context and high-concurrency scenarios. It requires engine and hardware support and adds some management overhead.
- Load-balancing strategies can adjust expert replicas or request scheduling when local GPU memory or compute pressure is too high.
5. API Call Example
The code below calls a hypothetical OpenAI-compatible interface. The endpoint address and model name must be replaced with the actual deployed values. This interface is not an official Baidu AI Cloud interface, and the code does not represent any performance or availability commitment. The required environment is requests 2.31 or later. Read the key from environment variables; do not write it into code or logs.
import osimport requestsendpoint_url = os.environ["MODEL_ENDPOINT_URL"]api_key = os.environ["MODEL_API_KEY"]payload = {"model": "ultra-large-moe-v1", # Replace with the actual deployed model name"messages": [{"role": "system", "content": "You are a professional data analysis Agent."},{"role": "user", "content": "Please plan the execution steps for retrieving yesterday's sales and generating a daily report."},],"temperature": 0.3, # A low temperature helps the Agent produce more stable output"max_tokens": 1024,}headers = {"Content-Type": "application/json","Authorization": f"Bearer {api_key}",}try:response = requests.post(endpoint_url, json=payload, headers=headers, timeout=(5, 60))response.raise_for_status()content = response.json()["choices"][0]["message"]["content"]print(content)except requests.exceptions.Timeout:print("Request timed out. Check the network connection or service load.")
