Architecture Analysis of Trillion-Parameter MoE Models: From Incremental Attention to Ultra-Sparse Expert Routing

I. Background: Three Core Engineering Challenges in Ultra-Large-Scale MoE Models

The parameter scale of large language models has transitioned from the hundred-billion to the trillion level. Sparse activation schemes, represented by the Mixture-of-Experts (MoE) architecture, have become the mainstream choice for balancing model capacity with inference costs. However, when total parameters exceed one trillion and active parameters reach the hundred-billion level, engineering teams consistently face three intractable issues:

1. Attention computation efficiency bottlenecks in ultra-long contexts: When the context window expands to the million-token scale, the quadratic complexity of standard Softmax attention becomes unacceptable. While linear attention reduces complexity to linear time, numerical precision issues prevent it from fully utilizing hardware matrix multiplication acceleration units.
2. Training stability issues with massive expert routing: When the number of routing experts reaches hundreds or even nearly a thousand, load imbalance triggers routing collapse. Traditional auxiliary loss methods struggle with convergence and are sensitive to hyperparameters in large-scale training.
3. Gradient propagation and representation degradation in deep Transformers: As layers are stacked to extreme depths, the information transmission efficiency in the residual path degrades. Cross-layer information sharing mechanisms become critical to breaking through depth bottlenecks.

This article systematically analyzes architectural designs and engineering practices addressing these three core problems, based on cutting-edge industry solutions.

II. Numerical Dilemmas of Linear Attention and Bounded Decay Parameterization

2.1 Root Cause: Numerical Overflow of Cumulative Decay Factors

Linear attention and its variants (such as Delta Net-like structures) rely on cumulative decay factors for intra-chunk attention weights during chunked parallel computation. Assuming the decay factor is a per-position log-decay value, the cumulative scaling coefficient of the i-th position relative to the chunk’s starting position is the exponential sum of the decay factors at each position.

The problem arises when decay factors use unbounded parameterization (such as negative Softplus): the reciprocal of the cumulative decay factor is highly prone to numerical overflow or underflow under limited precision (like BF16). This prevents diagonal blocks (where queries and keys reside in the same chunk) from directly utilizing dense matrix multiplication. They must fall back to an explicit per-position pairing computation path, severely degrading computational throughput.

2.2 Solution: Bounded Scaling Sigmoid Parameterization

The core idea is to change the generation method of the log-decay from an unbounded function to a bounded scaling Sigmoid form. Specifically, by fixing a lower bound constant (e.g., -5), the decay value at each position is constrained within a safe interval:

g_t = g_min * sigmoid(linear_projection(h_t))
where g_min = -5

The benefit of this modification is direct: within any fixed-length chunk (e.g., 16 tokens), the reciprocal of the cumulative scaling factor is locked within a numerically safe range. Diagonal blocks no longer require special computation branches. All causal blocks—whether diagonal or non-diagonal—can uniformly invoke dense Tensor Core matrix multiplication, significantly boosting throughput.

2.3 Engineering Reusability

The greatest advantage of this solution is the minimal engineering modification required. It does not alter the overall computation graph of the attention mechanism. By merely adjusting the generation function of the decay parameters, it resolves the implementation efficiency issues of linear attention operators on mainstream deep learning frameworks and CUDA/CUTLASS kernels. Furthermore, because the decay has a lower bound, the model natively supports direct extrapolation for million-level contexts without Positional Encoding (NoPE), eliminating the need for extrapolation strategies like Rotary Position Embedding (RoPE).

III. Load Balancing in Ultra-Sparse MoE: Quantile Balancing Mechanism

3.1 Limitations of Traditional Auxiliary Loss Methods

In MoE architectures, the gating network assigns tokens to Top-K experts. When the number of experts reaches hundreds, a lack of effective constraints causes a few experts to absorb excessive tokens, while others remain under-trained—a phenomenon known as “routing collapse.” The industry commonly introduces auxiliary load balancing losses, but this approach has clear flaws:

• Auxiliary losses compete with the main task loss for gradients, making weight tuning difficult.
• In large-scale training, the convergence behavior of auxiliary losses is unstable, easily leading to training oscillations.
• Under extreme sparsity (where the proportion of activated experts is extremely low), the regulatory strength of auxiliary losses is difficult to control precisely.

3.2 Core Design of Quantile Balancing

Quantile Balancing (QB) provides an auxiliary-loss-free dynamic load balancing strategy. Its fundamental idea is not to “penalize” imbalance via an additional loss term, but to directly introduce dynamic bias adjustments based on quantiles during the routing scoring phase.

Specifically, a dynamic bias term is maintained for each expert’s routing score, updated based on recent quantile statistics of token allocations. For experts receiving excess allocations, their bias is decreased; for under-allocated experts, the bias is increased. This process occurs entirely within the routing logic, producing no extra loss terms and without interfering with the main task’s gradient propagation.

3.3 Stability Safeguards

Alongside Quantile Balancing, the architecture incorporates the SiTU-GLU activation function and a stabilized latent space MoE structure. Compared to traditional GLU variants, SiTU-GLU provides smoother gradient behavior in the gating path. The stabilized latent space structure prevents the variance of expert outputs from cumulatively amplifying over training steps through normalization and projection constraints on expert inputs. Together with the QB mechanism, these ensure training stability under extreme sparsity conditions (nearly 900 routing experts with a very low activation ratio).

IV. Block Attention Residuals: Breaking the Deep Representation Bottleneck

4.1 Information Transmission Bottlenecks in Deep Transformers

Standard Transformer residual connections provide an identity mapping path for each layer. However, as the number of layers increases, the information in the residual stream is gradually “diluted” by subsequent layer transformations. Attention mechanisms compute independently at each layer, lacking direct attention information sharing channels between layers. This leads to diminishing marginal returns in representation capacity for the latter half of extremely deep models.

4.2 Mechanism of Block Attention Residuals

The core idea of Block Attention Residuals (AttnRes) is to pass the output of attention modules to subsequent layers as cross-layer residuals at a block granularity. Specifically, the model is divided into several layer groups (blocks). The attention output within each block is aggregated and injected into the attention computation of subsequent blocks in a residual form.

This design yields two benefits:

• Improved representation capacity: Subsequent layers can directly access the attention patterns of preceding blocks without indirect transmission through feed-forward networks, effectively breaking the deep representation bottleneck.
• Reduced memory and communication overhead: Because sharing occurs at block granularity rather than full per-layer connections, the memory footprint and communication volume of cross-layer information transfer are kept within acceptable limits.

4.3 Synergy with Other Technologies

Block Attention Residuals complement the aforementioned incremental attention mechanism. Incremental attention solves the computational efficiency of single-layer attention in long contexts, while Block Attention Residuals solve the information transmission efficiency in multi-layer stacking. Together, they enable the model to maintain efficient training and inference under million-token contexts and extremely deep network structures.

V. Multi-Stage Pretraining and Agentic Post-Training Strategies

5.1 Progressive Context Length Extension

This architecture employs a multi-stage pretraining strategy. The context window starts at 8K, sequentially expanding to 64K, 256K, and ultimately 1M tokens. The data composition and learning rate scheduling at each stage are adapted to the current context length. This progressive extension strategy avoids the training instability that arises from jumping directly from short to ultra-long contexts.

5.2 Million-Token Level Agentic RL Post-Training

After pretraining, the model undergoes million-token level agentic reinforcement learning post-training. The goal of this stage is to enhance the model’s performance in long-horizon reasoning tasks—including multi-step tool invocation, long-document understanding, and cross-paragraph information integration. By expanding task trajectories in the reinforcement learning environment to the million-token scale, the model learns to maintain reasoning coherence and goal alignment within ultra-long contexts.

VI. Engineering Practice Recommendations and Conclusion

Synthesizing the above analysis, engineering teams planning to build or optimize ultra-large-scale MoE models should focus on the following:

1. Prioritize numerical safety in linear attention over theoretical operator complexity: Prioritize “small change, big gain” optimizations like bounded decay parameterization to quickly unlock hardware compute power.
2. Internalize load balancing into routing logic: Avoid auxiliary losses that compete with the main task. Solutions like Quantile Balancing are more scalable in ultra-large-scale scenarios.
3. Control the granularity of cross-layer information sharing: Block-level residuals strike a good balance between representation gains and memory overhead, making them a pragmatic choice for deep Transformer architectures.
4. Match context extension with post-training strategies: Multi-stage pretraining combined with long-context reinforcement learning is an effective path to unlocking ultra-long context capabilities.

The engineering implementation of trillion-parameter MoE models is fundamentally a continuous trade-off among numerical stability, computational efficiency, and representation capacity. The technical solutions discussed above provide a proven set of problem-solving approaches worth referencing for similar model architecture scales.