Architectural Evolution of Trillion-Parameter Sparse Multimodal Models: From Attention Mechanisms to Expert Routing in Engineering Practice

1. Background: Crossing the Trillion-Parameter Threshold

As large model parameter counts move from the hundreds of billions toward the trillions, conventional dense architectures are reaching the practical limits of training cost, inference latency, and memory consumption. Mixture-of-Experts (MoE) architectures decouple total parameter count from per-forward-pass computation through sparse activation, making them the mainstream choice for ultra-large-scale models. However, when total parameters reach 2.8 trillion, activated parameters exceed 100 billion, and the number of routed experts approaches 900, several problems that are negligible at smaller scales become sharply amplified:

• Numerical overflow in linear attention operators: In ultra-long context scenarios, cumulative multiplication of decay factors can easily overflow or underflow under finite precision.
• Load imbalance in expert routing: Under extreme sparsity, a small number of experts become over-activated while most remain idle for long periods, sharply degrading training efficiency.
• Representation bottlenecks in deep Transformers: Gradient attenuation and information loss caused by increasing layer depth make it difficult for deeper networks to effectively utilize shallow features.

This article addresses these three core challenges from three dimensions—attention mechanisms, expert routing, and cross-layer information sharing—and systematically examines key architectural designs and engineering practices for trillion-parameter sparse multimodal models.

2. Numerical Stability in Linear Attention: Lower-Bound Decay Parameterization

2.1 Root Cause

Linear attention and its variants (such as Delta Net-class structures) improve throughput through parallel computation within chunks. When computing inside a chunk, the reciprocal of the cumulative decay factor Γ must be introduced for normalization. The problem is that when the decay factor uses unbounded parameterization (such as negative Softplus), the reciprocal of Γ can easily overflow under BF16 precision—especially when sequence lengths reach millions of tokens, where cumulative decay spans an extremely wide dynamic range.

This leads to a direct engineering consequence: diagonal tiles cannot participate in the standard matrix multiplication acceleration path and must instead take an explicit position-pair computation branch. This not only slows overall throughput but also increases the complexity of CUDA/CUTLASS operator implementations.

2.2 Design Approach for Lower-Bound Decay

The core idea is elegantly simple: replace unbounded logarithmic decay with a lower-bounded scaled Sigmoid parameterization. Specifically, the decay lower bound is fixed to a safe constant (such as -5), ensuring that the reciprocal scaling factor within any 16-token tile remains locked within a numerically safe range.

The engineering value of this change includes:

• Eliminates the special branch for diagonal tiles: All causal tiles (diagonal and non-diagonal) can directly invoke dense Tensor Core GEMM, unifying the operator implementation path.
• Significant throughput improvement: Diagonal tiles no longer require a separate position-pair computation path, making the overall computation graph more regular.
• Native support for positional-encoding-free extrapolation: Because decay parameterization itself provides position awareness, the model can directly extrapolate to million-token contexts without explicit positional encodings.

2.3 Reuse Recommendation

The engineering change required is minimal—essentially just replacing the parameterization form of an activation function—but it delivers significant efficiency gains for linear attention operator implementations. For any team that needs to deploy linear attention variants in ultra-long context scenarios, this is a low-cost, high-return optimization.

3. Load Balancing in Extreme Sparse MoE: Quantile-Based Dynamic Balancing

3.1 Limitations of Conventional Approaches

Load balancing in MoE training typically relies on auxiliary losses to encourage uniform expert utilization. However, in extreme sparse scenarios with nearly 900 experts and very low activation ratios, auxiliary losses face a dilemma:

• If the weight is too small: balancing is insufficient, and routing collapse (a few experts handling the vast majority of tokens) becomes a high risk.
• If the weight is too large: it interferes with the main task gradients and damages model expressiveness.

Furthermore, auxiliary losses are inherently soft constraints that cannot guarantee actual per-step load balance across experts.

3.2 Quantile Balancing Mechanism

The core idea of quantile balancing is to replace soft constraints with hard constraints. During the routing decision stage, a quantile threshold is dynamically selected based on the score distribution of each expert within the current batch to determine the routing target for each token, thereby achieving load balance without introducing any additional loss term.

Its advantages include:

• No auxiliary loss: Does not interfere with main task gradients, keeping the training objective purer.
• Dynamic adaptivity: The quantile threshold adjusts dynamically throughout training, adapting to load distribution changes at different stages.
• Rapid balancing: The routing decision itself serves as the balancing operation, requiring no additional gradient backpropagation path.

3.3 Engineering Considerations

In distributed training environments, quantile computation requires cross-device synchronization. To reduce communication overhead, a strategy of local statistics combined with periodic global synchronization is typically adopted. Additionally, quantile thresholds need appropriate smoothing mechanisms to avoid routing oscillations caused by sharp fluctuations in early training.

4. Cross-Layer Sharing for Deep Representations: Block Attention Residuals

4.1 The Nature of the Depth Bottleneck

In standard Transformers, each layer’s output is passed layer by layer through residual connections. As depth increases, deep networks face two problems:

• Gradient attenuation: Gradients decay significantly after passing through dozens of residual paths during backpropagation.
• Representation homogenization: The difference between deep and shallow features shrinks, reducing the network’s effective depth.

In trillion-parameter models, these problems are further amplified—because models need sufficient depth to support complex reasoning capabilities, but increased depth does not linearly translate into performance gains.

4.2 The Block Attention Residual Mechanism

The core design of block attention residuals is to share attention layer outputs across layers at the block level. Specifically, every several layers form a block, and the attention outputs of each layer within the block are aggregated through residual paths into a block-level representation, which is then passed to subsequent blocks.

This design delivers three benefits:

• Breaks the deep representation bottleneck: Block-level aggregation allows shallow information to influence deep outputs through shorter paths, mitigating gradient attenuation.
• Reduces memory overhead: Attention outputs within a block can share storage, reducing memory usage for intermediate activations.
• Reduces communication volume: In distributed training, block-level aggregation reduces both the frequency and volume of cross-device communication.

4.3 Synergy with Other Techniques

Block attention residuals complement the aforementioned lower-bound decay parameterization and quantile balancing mechanisms: the first addresses numerical stability in attention, the second addresses load balancing in expert routing, and the third addresses representation efficiency in deep networks. Together, they form the technical foundation for the training stability of trillion-parameter sparse models.

5. Multi-Stage Pretraining and Agentic Post-Training

5.1 Progressive Context Length Extension

Ultra-long context training should not be done in a single leap. In practice, a multi-stage pretraining strategy is adopted, progressively extending context length from 8K to 64K, then 256K, and finally to 1M tokens. At each stage, the model needs to adapt to new attention spans, and progressive extension significantly reduces training instability.

5.2 Million-Token Agentic Reinforcement Learning

After pretraining is complete, million-token-level agentic reinforcement learning post-training further improves model performance on long-range reasoning and agent tasks. The key challenges at this stage include:

• Credit assignment in ultra-long trajectories: In million-token interaction trajectories, how to attribute final rewards to early decision steps.
• Computational efficiency: RL training under long contexts demands extremely high computational resources, requiring sparse activation and efficient attention operators to control costs.

6. Summary and Engineering Takeaways

Trillion-parameter sparse multimodal model architecture design is fundamentally about finding the optimal balance among model capacity, computational efficiency, and training stability. The three key techniques discussed in this article—lower-bound decay parameterization, quantile balancing, and block attention residuals—correspond respectively to the engineering challenges of attention numerical stability, expert load balancing, and deep representation efficiency, and offer high reuse value.

For developers exploring ultra-large-scale sparse models, the following practical principles are recommended:

1. Prioritize numerical safety: Identify and eliminate numerical overflow risks at the operator level as early as possible to avoid unrecoverable loss spikes during training.
2. Hard constraints over soft constraints: For systemic constraint problems such as load balancing, hard constraint mechanisms are often more reliable than auxiliary losses.
3. Progressive extension: Whether for context length or model scale, progressive strategies consistently outperform all-at-once approaches.
4. Architectural co-design: Attention, routing, and cross-layer connections are not independent modules; they need to be jointly optimized to achieve maximum benefit.