Trillion-Parameter MoE Models: Architecture and Engineering Trade-Offs
As large language models reach the trillion-parameter scale, Mixture-of-Experts architectures make this growth more practical by separating total model capacity from the parameters activated for each token. This article compares DeepSeek-V3 and Kimi K2, covering expert sparsity, attention heads, MLA, training optimizers, memory requirements, distributed inference, and practical deployment checks. It also highlights why communication, load balancing, and training stability must be evaluated alongside theoretical compute savings.
Large language models have entered the trillion-parameter era. Dense models use all of their parameters during every inference step, making their compute and operating costs difficult to sustain. As a result, many teams have turned to the Mixture-of-Experts (MoE) architecture.
MoE separates the model’s total parameter count from the number of parameters actually used for each token. The model can be extremely large, while each token is processed by only a small subset of its parameters.
Several trillion-parameter models released publicly over the past year share similar structural choices. DeepSeek-V3 published its technical report in December 2024. It has 671B total parameters, activates approximately 37B parameters per token, uses Multi-head Latent Attention (MLA), and includes 256 routed experts. Kimi K2, released by Moonshot AI with a technical report in July 2025, follows the general design direction of DeepSeek-V3 with several changes: approximately 1.04T total parameters, around 32B activated parameters per token, 384 experts, and a reduction in attention heads from 128 to 64. Its training setup uses the Muon optimizer together with QK-Clip.
The figures below are taken from these two reports. Always verify the latest official materials before using them for production planning.
| Item | DeepSeek-V3 | Kimi K2 |
|---|---|---|
| Total parameters | 671B | Approximately 1.04T |
| Activated parameters per token | Approximately 37B | Approximately 32B |
| Routed experts (selected per token) | 256 (8) | 384 (8) |
| Shared experts | 1 | 1 |
| Attention heads | 128 | 64 |
| Attention mechanism | MLA | MLA |
| Pretraining tokens | Approximately 14.8T | Approximately 15.5T |
| Optimizer | AdamW | Muon with QK-Clip |
1. Sparsity and Attention Head Count: Two Separate Trade-Offs
Sparsity can be measured as the proportion of routed experts activated for each token. DeepSeek-V3 selects 8 experts from 256, for a ratio of approximately 3.1%. Kimi K2 selects 8 from 384, reducing the ratio to approximately 2.1%. K2 is therefore more sparse.
Increasing the number of experts allows each expert to specialize in a narrower area of knowledge and increases the model’s total capacity. The costs are also significant:
- The router requires more computation.
- Every MoE layer must perform cross-device data redistribution, typically through all-to-all communication. As experts become more distributed, communication volume increases.
- The weights of all experts must remain resident in GPU memory.
- Fewer tokens may be assigned to each expert, potentially reducing matrix multiplication efficiency.
The effect of attention head count should be considered separately. Reducing the number of heads lowers the cost of computing attention scores and weighted sums. The benefit is most noticeable in long-context workloads. However, fewer heads also mean that each layer can model fewer types of relationships in parallel. The impact on representation quality must be determined through evaluation rather than inferred from compute cost alone.
2. MLA Compresses the KV Cache, Not the Number of Heads
MLA, or Multi-head Latent Attention, first compresses Keys and Values into lower-dimensional latent vectors and caches those vectors. In DeepSeek-V3, for example, each layer caches a 512-dimensional compressed latent vector for every token, plus a 64-dimensional decoupled RoPE key used for positional encoding. Together, this requires 576 elements.
With standard multi-head attention, 128 heads with 128 dimensions per head would require 32,768 elements for K and V combined. The difference is approximately 57 times.
This leads to a common misconception: KV Cache size is determined primarily by the latent vector dimensions and has relatively little to do with the number of attention heads. Reducing the number of attention heads from 128 to 64 mainly lowers attention computation. Its effect on memory usage is limited.
MLA also introduces costs. During attention computation, the latent representation must be reconstructed or incorporated through an absorption-style computation. The inference engine therefore needs suitable operator support. Before deployment, measure throughput and latency on the target inference framework rather than assuming that theoretical savings will translate directly into production gains.
3. Training: Muon, QK-Clip, and Token Efficiency
Token efficiency describes how much useful information a model can learn from a fixed amount of data. Because high-quality training data is limited, improving token efficiency has become a major objective.
The Muon optimizer targets weight matrices and approximately orthogonalizes momentum, typically using Newton–Schulz iterations. Compared with AdamW, its updates for matrix parameters are more evenly distributed, which can help accelerate convergence. The Kimi K2 report presents comparisons showing improved token efficiency under its experimental settings. However, these comparisons were largely conducted by the model team itself. The results may not hold at a different model scale, dataset size, or learning-rate configuration, so they should be validated with small-scale experiments.
Muon can cause attention logits to grow too quickly, which may lead to training divergence. QK-Clip is designed to address this issue. During training, it monitors the maximum QK logit for each attention head. When a value exceeds a threshold, the corresponding (W_q) and (W_k) weights are proportionally scaled down after the parameter update.
QK-Clip modifies the weights; it does not clip the dot product during the forward pass. Both the threshold and the scope of the operation are training hyperparameters that must be adjusted for the specific setup.
Synthetic data is another way to expand the available token pool. For example, intermediate checkpoints or smaller models can be used to generate and rewrite data. This can increase the volume of training material, but it may also amplify errors and distribution bias. Before synthetic data enters training, it should undergo deduplication, quality filtering, and factual spot checks.
4. Engineering Implications: Estimate Before Deployment
For a 1.04T-parameter model, estimating weight memory requires only two values: the parameter count and the number of bytes per parameter.
- With BF16 at 2 bytes per parameter, the weights require approximately 2.08 TB.
- With FP8 at 1 byte per parameter, the weights require approximately 1.04 TB.
A node with eight 80 GB GPUs provides 640 GB in total. Even the FP8 weights alone would require at least two such nodes. Additional capacity is needed for the KV Cache, activations, and communication buffers, so the actual node count is typically higher.
This estimate assumes that all parameters remain resident in GPU memory. Quantization and parameter offloading can change the result.
Distributed execution strategies should be selected according to the actual bottleneck. Within a node, tensor parallelism can use high-bandwidth NVLink communication. Cross-node expert parallelism introduces substantial all-to-all traffic and typically depends on RDMA networking, such as InfiniBand or RoCE. When network bandwidth is insufficient, GPUs spend more time waiting than computing.
Before deployment, use the following checklist:
- Verify model specifications: Confirm the total parameter count, activated parameter count, number of experts, experts selected per token, attention head count, and context length against the official model card.
- Estimate memory usage: Calculate the requirements for weights, KV Cache, activations, and communication buffers separately at the target precision and concurrency level.
- Benchmark communication: Measure the proportion of each step spent on all-to-all communication. If it is high, consider higher-bandwidth networking or reducing cross-node expert placement.
- Check load balancing: Track token distribution across experts. Significant imbalance can cause throughput to be limited by the busiest expert.
- Plot KV Cache growth: Measure memory usage at different batch sizes and context lengths to identify the service capacity limit.
- Monitor training stability: Record the maximum logit for each attention head, loss spikes, and gradient norms. Roll back promptly when anomalies appear.
5. Conclusion
Trillion-parameter MoE models such as DeepSeek-V3 and Kimi K2 offer useful reference points for expert sparsity, MLA-based compression, and training stability. Each technique involves trade-offs, and no single design is suitable for every workload.
For developers, three recommendations are immediately actionable:
- Use the official model card and technical report when selecting a model. Do not base capacity planning on figures from secondary articles.
- Benchmark communication and memory before deployment. In expert parallelism, the primary bottleneck is often the network rather than GPU compute.
- During training or fine-tuning, continuously monitor attention logits and loss so that stability problems are detected as early as possible.
