Evolution and Engineering Practices of Ultra-Scale Multimodal MoE Architectures

    As large language models expand toward multimodal inputs and trillion-scale parameter counts, Mixture-of-Experts architectures and million-token context processing have become central engineering challenges. This article examines the general technical principles behind these systems, including attention optimization, MoE load balancing, staged context expansion, long-horizon reinforcement learning, and the cloud infrastructure required for stable training and inference.

    Evolution and Engineering Practices of Ultra-Scale Multimodal MoE Architectures

    As large language models evolve toward multimodal inputs and extremely large parameter counts, Mixture-of-Experts (MoE) architectures and long-context processing have become major areas of research and engineering.

    The industry has recently introduced several open-source multimodal MoE models with total parameter counts exceeding one trillion and hundreds of billions of activated parameters. These models natively support inputs such as images while processing contexts containing up to millions of tokens.

    Rather than focusing on any specific product, this article analyzes the core innovations behind ultra-scale multimodal MoE models from a general architectural perspective. It covers attention optimization, MoE load balancing, training strategies, and cloud infrastructure considerations to provide practical guidance for large-model engineering.

    1. Engineering Breakthroughs in Long-Context Processing and Attention

    When processing contexts containing millions of tokens, the conventional Transformer attention mechanism faces severe challenges in memory consumption and computational complexity. To improve the efficiency of long-sequence computation, modern large models increasingly incorporate linear attention mechanisms. In real-world deployments, however, numerical stability and hardware utilization remain critical bottlenecks.

    1.1 KDA Lower-Bound Decay Parameterization and Numerical Stability

    Traditional linear attention mechanisms, such as DeltaNet, can encounter severe numerical overflow or underflow during in-chunk computation because of the inverse of the accumulated decay factor Γ. These issues are particularly pronounced in limited-precision environments such as BF16.

    As a result, diagonal blocks often need to follow an explicit positional path instead of taking advantage of matrix multiplication acceleration on GPU Tensor Cores. This substantially reduces overall throughput.

    One approach to addressing this problem is lower-bound decay parameterization. It replaces the unbounded Negative-Softplus function used for log decay with a Scaled Sigmoid function that has a defined lower bound.

    By setting a fixed minimum decay value, such as -5, the inverse scaling factor within any 16-token tile can be constrained to a safe numerical range. This relatively small engineering change delivers significant benefits: it eliminates the need for a special computation branch for diagonal blocks, allowing all causal attention blocks, including diagonal and non-diagonal blocks, to directly use dense Tensor Core GEMM operators.

    The result is higher computational throughput at the CUDA and CUTLASS levels. It also naturally supports direct extrapolation to million-token contexts without positional encoding.

    1.2 Attention Residuals and Deep Representation Bottlenecks

    As Transformer networks become deeper, they can suffer from representation degradation and information loss. Block-level attention residuals have been introduced to address this bottleneck.

    This mechanism shares attention information across different layers, enabling information to flow between layers. It helps reduce feature smoothing in deep networks while also lowering memory usage and inter-layer communication overhead during training and inference.

    As a result, it provides a practical path for scaling large models under constrained memory conditions.

    2. Improving Training Stability in Massive-Expert MoE Architectures

    In MoE architectures with trillions of total parameters, efficiently and reliably routing tokens among hundreds of experts presents another major challenge. Under extreme sparsity, models can experience severe load imbalance across experts, as well as gradient instability and even gradient explosions.

    2.1 Stable LatentMoE and Activation Function Design

    To balance model capacity and computational efficiency, modern large models often use Stable LatentMoE architectures with hundreds of routed experts.

    In this setting, activation function selection is important for training stability. Conventional ReLU or GELU activations may perform poorly in extremely deep networks. As a result, SiTU-GLU, which combines characteristics of SiLU and gated linear units (GLU), is increasingly used.

    SiTU-GLU provides a smooth nonlinear transformation while enhancing representational capacity through gating. This creates a more stable foundation for activating and training a large number of experts.

    2.2 Fast Dynamic Load Balancing Without an Auxiliary Loss

    Traditional MoE load-balancing methods typically rely on an auxiliary loss. When the number of experts reaches several hundred, however, the auxiliary loss can complicate the optimization objective, create conflicts between optimization goals, and still fail to achieve truly dynamic balancing.

    To address this issue, the industry has proposed a fast dynamic load-balancing strategy based on quantile balancing. This method removes the auxiliary loss entirely and instead monitors statistical properties of expert-load distributions, such as quantiles, to adjust routing decisions dynamically.

    Because it does not depend on an additional loss function, the approach can respond quickly and adjust workloads across a large number of experts on a millisecond timescale. It helps prevent expert collapse and the “winner-takes-all” effect, supporting stable pretraining at trillion-parameter scale.

    3. Pretraining and Post-Training for Long-Horizon Reasoning

    Architectural innovation must be combined with appropriate training strategies to achieve its full potential.

    During pretraining, staged context expansion has become a widely adopted approach. Models may progressively expand from an initial 8K context to 64K, then 256K, and eventually 1M tokens.

    This gradual progression helps control computational costs and reduces the risk of catastrophic forgetting, in which features learned from short-text training fail to transfer effectively to long-context scenarios.

    During post-training, million-token-scale Agentic RL has become important for improving long-horizon reasoning. By simulating complex tool use, multi-step planning, and reflection within extremely long contexts, models can learn to maintain logical consistency and track state across extended tasks.

    This shift from passive generation to active long-horizon reasoning represents an important step toward more general-purpose AI systems.

    4. Cloud Engineering Practices and Infrastructure Considerations

    Training and serving ultra-scale multimodal MoE models impose demanding requirements on the underlying computing infrastructure. For enterprise development, relying on a capable cloud-native computing platform is often necessary.

    Several areas require particular attention:

    1. High-performance clusters and network topology: Training trillion-parameter MoE models requires extremely high-bandwidth All-to-All communication. Large-scale high-performance computing clusters provided by cloud vendors such as Baidu AI Cloud commonly use lossless Ethernet or InfiniBand networks. These configurations can meet the low-latency cross-node communication requirements of LatentMoE architectures.

    2. Distributed storage and checkpoint recovery: The throughput of large multimodal datasets and the rapid storage of checkpoints depend on enterprise-grade object storage and parallel file systems. During transitions between different pretraining stages, efficient storage I/O can significantly reduce idle time.

    3. Containerization and elastic scheduling: Container platforms can provide fine-grained GPU scheduling. Combined with elastic fault-tolerance mechanisms, they help maintain stable training for months at thousand-GPU scale.

    Conclusion

    Every small architectural improvement in ultra-scale multimodal MoE models, from KDA lower-bound decay parameterization to QB dynamic load balancing, reflects a deeper integration of mathematical principles and low-level engineering.

    These advances span hardware-level optimization of attention operators, stable training for trillion-parameter MoE systems, and Agentic RL over million-token contexts. Together, they extend the performance boundaries of large models and provide useful direction for developers and enterprises implementing large-model systems in production.