Architecture, Quantization, and Deployment of Trillion-Parameter MoE Models
This article explores the practical aspects of deploying trillion-parameter Mixture of Experts (MoE) models. It covers the core principles of sparse MoE architectures, engineering trade-offs for long-context and sparse attention, low-bit quantization techniques such as microscaling formats and QAT, and provides a step-by-step guide for local deployment resource planning.
1. Problem Background
As the parameter scale of large models continues to grow, the training and inference costs of dense models rise rapidly. Mixture of Experts (MoE) models mitigate this contradiction with a design that maintains a large total parameter count but activates only a small fraction of parameters per step. When evaluating these models, three key metrics must be considered simultaneously: the total parameter count determines the storage and loading costs of weights; the activated parameter count determines the compute volume per step; and the context length determines attention computation and cache usage. The combination of these three determines whether a model can run on specific hardware and at what speed.
2. Basic Principles of Sparse MoE Architecture
The sparse MoE model has three core components.
1. Expert Routing. Each Transformer layer contains a large number of feedforward expert networks. A router calculates a score for each expert for every token, selecting only the top-k highest-scoring experts to participate in the computation while keeping the rest idle. Below is a schematic Top-k selection function:
def top_k_route(probs, k):"""probs is the expert probability after softmax, returns selected expert indices and normalized weights (schematic)"""ranked = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)[:k]total = sum(probs[i] for i in ranked)return [(i, probs[i] / total) for i in ranked]# Assume 8 experts, select 2 per tokenprint(top_k_route([0.05, 0.30, 0.02, 0.25, 0.10, 0.08, 0.15, 0.05], k=2))# [(1, 0.5454...), (3, 0.4545...)]
2. Shared Experts. Some designs retain a small number of shared experts that apply to all tokens, designed to carry general language and pattern knowledge. The remaining routed experts handle specific domains more finely. The division of labor between these two types of experts reduces knowledge redundancy.
3. Activation Ratio. The activation ratio is the proportion of parameters actually involved in the computation for each token relative to the total parameters. Consider this example: if the total parameters are 3 trillion and 100 billion parameters are activated per token, the activation ratio is approximately 3.3%. This means the compute volume for a single inference step is close to that of a 100-billion dense model, but the storage cost is dictated by the trillion parameters. This disparity is the most critical aspect to plan for during deployment.
3. Engineering Trade-offs for Long Context and Sparse Attention
The computation of standard attention grows quadratically with sequence length, and the footprint of the KV cache grows linearly. Using a sample calculation to illustrate the magnitude: assume the model has 90 layers, 8 KV heads per layer, a head dimension of 128, and a 2-byte cache storage. The KV cache per token is approximately:
2 (K and V) × 8 (KV heads) × 128 (head dimension) × 2 (bytes) × 90 (layers) ≈ 0.37 MB
When the context reaches 1 million tokens, the cache for a single request is about 370GB, which exceeds the total VRAM of many single machines.
Sparse attention aims to let each query focus only on a subset of positions in the historical sequence, thereby reducing computation and memory access. Common approaches include sliding windows, block selection, and lightweight scoring to select key positions. In engineering practice, you can replace full attention in only some layers while retaining a global view in others. The smaller the scope of changes, the easier it is to verify behavioral differences from the original model using regression tests.
It is important to note that sparse attention saves computation and bandwidth but does not automatically reduce the loss of semantic information in long contexts. Before going live, you should test tasks such as retrieval, cross-paragraph reasoning, and long document summarization separately.
4. Low-Bit Quantization
Weight quantization compresses each parameter from 16 bits to around 4 bits, reducing the weight volume to roughly a quarter. Using 8-bit activations can further reduce memory access and communication overhead.
Microscaling Formats. Per-tensor quantization uses a single scaling factor for the entire tensor, which leads to noticeable precision loss when numerical ranges vary significantly. The microscaling floating-point format allows a group of adjacent elements (e.g., 32) to share a scaling factor, better preserving local numerical ranges with minimal additional storage overhead.
Quantization-Aware Training. Direct post-training quantization at low bit-widths is prone to accumulating errors. Quantization-Aware Training (QAT) simulates low-bit computation during the training process, allowing weights to gradually adapt to quantization errors, typically resulting in more stable precision than post-training quantization.
To calculate the average storage bit-width per parameter, you can use the following formula:
Average bit-width = Element bit-width + Scaling factor bit-width ÷ Elements per group
For example, a 4-bit element combined with an 8-bit scaling factor shared among 32 elements yields an average bit-width of 4 + 8/32 = 4.25 bits.
def weight_memory_gb(total_params, bits_per_param):"""Estimate the VRAM occupied by the weights themselves (GB), excluding KV cache, activations, and communication buffers"""return total_params * bits_per_param / 8 / 1e9print(round(weight_memory_gb(3e12, 4.25))) # Approximately 1594 GB
Before deployment, confirm whether the inference framework and hardware natively support the chosen format. If not, an online conversion will be required during loading, which increases startup time and may introduce additional errors.
5. Resource Planning for Local Deployment
To deploy a trillion-level model, resources can be estimated using the following steps.
Step 1: Estimate Weight Size. Use the formula from the previous section. At 3 trillion parameters and 4.25 bits, the weights are approximately 1.6TB.
Step 2: Reserve KV Cache. Calculate based on the target concurrency and context length. Higher concurrency requires more cache.
Step 3: Reserve Runtime Overhead. Activations, communication buffers, and memory fragmentation typically require extra space. Empirically, a 10% to 20% margin can be reserved, though the exact proportion should be determined through testing.
Step 4: Choose a Parallelism Strategy.
- Expert Parallelism: Distributes different experts across different devices, suitable for MoE structures, but requires monitoring the load balance of experts across devices.
- Tensor Parallelism: Splits individual matrix operations, demanding high inter-device interconnect bandwidth.
- Pipeline Parallelism: Splits the model by layers, allowing capacity to scale as the number of devices increases, but requires careful management of pipeline bubbles.
