Sparse MoE Architecture in LLMs: Comparing Technical Routes and Optimization Trade-offs
This article compares two sparse Mixture-of-Experts (MoE) design routes used in large language models. It examines how expert count, attention head configuration, and sparsity affect model capacity, inference cost, and training stability. It also offers a framework for judging whether an MoE architecture's design choices are sound.
1. Why “Small Differences” Still Matter
Chinese large language models are converging on a common technical direction. Mainstream models almost all adopt the sparse Mixture-of-Experts (MoE) architecture, and they share highly similar base components, including attention mechanisms, normalization strategies, and positional encoding. From a purely technical lineage perspective, many of these models come from the same design roots.
Architectural similarity does not mean engineering equivalence, however. On the same technical foundation, different teams have made very different trade-offs on core issues: sparsity scaling, inference cost control, and training stability. These trade-offs may look like differences in parameter tables, but they reflect different answers to a basic question: in which direction should model capacity be expanded? This article analyzes three areas: architecture parameters, attention mechanisms, and sparsity strategies. It aims to help readers build a systematic framework for evaluating whether an MoE model’s design is reasonable.
2. Architecture Parameter Comparison: Tuning the Same Foundation Differently
2.1 Shared Architectural Features
Current mainstream sparse MoE models commonly use the following design combination:
- Multi-head Latent Attention (MLA): Compresses the key-value cache through low-rank compression, significantly reducing memory usage in long-context scenarios.
- Fine-grained expert segmentation with shared expert isolation: Improves expert specialization while preserving a pathway for general knowledge.
- Auxiliary load-balancing loss: Prevents routing collapse among experts and keeps training stable.
Take a typical MoE model at the trillion-parameter scale as an example. Its hidden dimension is usually set around 7000, its expert hidden dimension is about 2048, and it uses a decoder-only Transformer stack. This combination has become a widely recognized cost-effective baseline for sparse architectures in the industry.
2.2 Key Differences: Expert Count and Attention Head Configuration
Built on this baseline, models diverge mainly on two parameters:
| Architecture parameter | Route A (expert-scaling) | Route B (attention-lean) |
|---|---|---|
| Total parameters | About 1T | About 671B |
| Activated parameters | About 32B | About 37B |
| Number of experts | 384 | 256 |
| Number of attention heads | 64 | 128 |
| Hidden dimension | 7168 | 7168 |
| Expert hidden dimension | 2048 | 2048 |
The core logic of Route A is based on scaling law analysis, which suggests that continued increases in sparsity can bring significant performance gains. The team therefore raised the number of experts from 256 to 384, a 50% increase, and reduced the number of attention heads from 128 to 64. This combination of more experts and fewer heads was not arbitrary. More experts raise the capacity ceiling of the model, while fewer attention heads directly reduce inference-time computation and KV cache pressure.
The core logic of Route B is more conservative. It keeps a moderate sparsity level with 256 experts and retains 128 attention heads to preserve representation resolution. Its activated parameters are slightly higher (37B versus 32B), so each forward pass requires more FLOPs. However, it may have advantages in attention expressiveness and long-range dependency modeling.
2.3 Engineering Implications of the Parameter Differences
The two routes should not be judged simply as better or worse. They should be analyzed against deployment scenarios:
- The expert-scaling route suits high-concurrency, low-latency online inference. With 384 experts and 64 attention heads, each token activates a smaller share of experts (higher sparsity), which allows larger batch sizes on the same hardware. However, a larger number of experts makes training the routing network harder and requires more refined load-balancing strategies.
- The attention-lean route may be more robust for long-context understanding and complex reasoning. Its 128 attention heads provide finer-grained multi-head representation, which benefits tasks that require precise attention allocation, such as multi-hop reasoning and code understanding. However, larger activated parameters mean higher compute cost per inference.
3. Trade-offs in Attention Mechanisms
3.1 Compression Gains and Expressiveness Loss in MLA
Multi-head Latent Attention (MLA) compresses the key-value cache into a low-rank latent space, which can reduce long-context memory usage by several times. Compression always loses some information, though. The low-rank bottleneck limits the bandwidth of information exchange among attention heads.
Under this constraint, the choice of attention head count becomes especially important:
- With more heads (such as 128), each head has a smaller dimension, so the damage that MLA’s low-rank compression causes to overall expressiveness is spread across heads.
- With fewer heads (such as 64), each head has a larger dimension, so the compression gains are concentrated. However, once the low-rank bottleneck turns into an information bottleneck, the model may struggle to capture fine-grained relationships between tokens.
Route A chooses 64 heads. In effect, it bets that the capacity gained from sparse experts can offset the drop in attention resolution. Route B keeps 128 heads and seeks a balance between attention precision and controllable inference cost.
3.2 Quantifying Inference Overhead
From an inference cost perspective, the number of attention heads directly affects KV cache size and attention computation. At the same sequence length, moving from 128 heads to 64 heads can theoretically halve the KV cache footprint, and it reduces attention computation accordingly. For inference services that handle very long contexts (such as 128K tokens and above), this difference appears directly in how many concurrent requests one accelerator can serve and in time-to-first-token latency.
The inference bottleneck in MoE models often lies not in the attention layers but in expert routing and expert computation, however. Increasing the number of experts from 256 to 384 raises the top-k selection complexity of the routing network. It also increases the overhead of loading and swapping expert parameters. Therefore, whether the attention savings from fewer heads can offset the routing overhead from more experts depends on the specific hardware configuration and how well the inference framework is optimized.
4. Scaling Law Basis for Sparsity Strategy
4.1 The Non-linear Relationship Between Sparsity and Performance
Scaling law research shows that, with total parameters fixed, increasing sparsity (that is, reducing the share of activated parameters) continues to improve model performance, but the gains diminish over time. Going from 256 to 384 experts lowers the activation ratio from about 5.5% (37B / 671B, a rough estimate)
