Inside Ultra-Large MoE Multimodal Models: Architecture Evolution and Key Technical Breakthroughs

    As large language models evolve from dense architectures with billions of parameters to Mixture-of-Experts (MoE) designs at the trillion-parameter scale, a new generation of ultra-large MoE multimodal models has emerged with remarkable capabilities. This article examines a representative architecture with 2.8 trillion total parameters and roughly 104 billion activated parameters, offering native vision modality support and a 1-million-token context window. Stripping away product-specific details, it breaks down the core technical mechanisms behind these models, including attention computation innovations, auxiliary-loss-free load balancing, block-level attention residuals, and multi-stage training with agentic reinforcement learning.

    In the wave of progress toward artificial general intelligence, large language models are rapidly evolving from dense architectures with tens or hundreds of billions of parameters to Mixture-of-Experts (MoE) architectures with total parameter counts reaching the trillion scale. Recent ultra-large MoE multimodal models have demonstrated impressive breakthroughs: a total of 2.8 trillion parameters, approximately 104 billion activated parameters, native vision multimodality, and a 1-million-token context window. This article strips away product-specific attributes and provides a deep analysis of the core technical mechanisms behind such models, covering architectural innovation, training paradigm evolution, and engineering practice.

    1. Overall Architecture Design and Core Challenges

    New-generation ultra-large multimodal models commonly adopt MoE architectures to balance computational cost against model capacity. At the scale of 2.8 trillion total parameters and 104 billion activated ones, the MoE side typically features a stable latent MoE architecture with hundreds of routed experts (e.g., 896), combined with a SiTU-GLU activation function. Under such extreme sparsity, model design faces three core challenges: computational efficiency over ultra-long contexts, training stability of massive-expert MoE, and long-horizon reasoning for AI agents. To tackle these challenges, the models introduce deep low-level rework in both the attention modules and the MoE routing mechanism.

    2. Key Innovation 1: Lower-Bounded Decay Parameterization Breaks the Attention Computation Bottleneck

    For ultra-long contexts, traditional Softmax attention is impractical due to its quadratic computational complexity, making linear attention an important research direction. However, conventional linear attention suffers from significant numerical and engineering bottlenecks in intra-chunk computation.

    The pain point: Under limited precision (e.g., BF16), the reciprocal of the cumulative decay factor is prone to severe numerical overflow or underflow. As a result, diagonal-block computations must fall back on an explicit positional path, unable to leverage GPU Tensor Core matrix-multiplication acceleration, which severely limits throughput.

    The mechanism: To address this, the industry has introduced a “lower-bounded decay parameterization” mechanism. It replaces the unbounded Negative-Softplus function for Log-decay with a scaled Sigmoid function that has a fixed lower bound (e.g., gmin = -5). This locks the reciprocal scaling factors within any local token tile into a safe numerical range.

    The engineering benefit: Although this change is minimal in terms of engineering effort, it completely eliminates the special computation branch for diagonal blocks. All causal blocks—both diagonal and non-diagonal—can directly invoke dense Tensor Core GEMM kernels, delivering a substantial throughput boost. Moreover, the mechanism naturally supports extrapolation without positional encoding, laying the foundation for smoothly scaling the model to 1 million tokens and beyond.

    3. Key Innovation 2: Auxiliary-Loss-Free, Ultra-Fast Dynamic Load Balancing

    Another persistent problem in MoE architectures is uneven expert utilization—some experts are over-invoked while others sit idle. Traditional solutions introduce auxiliary losses to constrain routing distributions, but this often interferes with gradient descent on the main objective and hurts final convergence quality.

    How it works: New-generation models adopt a quantile-based balancing mechanism, an auxiliary-loss-free, ultra-fast dynamic load balancing strategy for ultra-large-scale MoE. By computing quantiles of the routing weights, it dynamically adjusts each expert’s reception threshold without adding any extra loss term, achieving a reasonable distribution of traffic across experts.

    The benefits: Eliminating the auxiliary loss unlocks the full optimization potential of the main loss function while maintaining extremely high training stability under extreme sparsity (896 routed experts). This dynamic balancing strategy is critical for large-scale distributed training, effectively preventing memory overflow and communication congestion caused by overloaded experts.

    4. Key Innovation 3: Block-Level Attention Residuals Break the Deep-Representation Bottleneck

    As Transformer depth increases, models are prone to deep-representation bottlenecks, and cross-layer information transfer often carries heavy memory and communication overhead.

    Cross-layer information sharing: The block-level attention residual mechanism offers an elegant solution. It breaks the traditional constraint that residual connections only pass information between adjacent layers, allowing direct information sharing across designated layers. By introducing attention residuals at the block level, the model enriches the expressive dimensionality of deep features without significantly deepening the computation graph.

    Performance optimization: This design effectively mitigates gradient vanishing in deep Transformers during backpropagation and, at the systems level, optimizes memory reuse and inter-node communication overhead. Combined with gated multi-head latent attention, the model maintains strong feature extraction capability while dramatically reducing KV Cache usage during inference.

    5. Multi-Stage Training Paradigm and Agentic Reinforcement Learning

    The emergence of model capabilities depends on well-designed training strategies. During pretraining, the model uses a progressive context extension strategy: 8K → 64K → 256K → 1M. This multi-stage design first solidifies foundational language and vision understanding on short contexts, then progressively lengthens the window to address attention divergence in long-text extrapolation, balancing training efficiency with final long-range modeling capability.

    In post-training, million-token-scale agentic reinforcement learning (RL) is key to improving long-horizon agent reasoning. By using RL to align model behavior in scenarios such as complex tool calling and multi-step logical reasoning, the model evolves from a pure “text generator” into an “agent” capable of handling long-horizon tasks. This enables top-tier performance in user-facing scenarios such as document understanding, complex content writing, and in-depth data analysis.

    6. Enterprise Applications and Infrastructure Practice

    Deploying such a massive model architecture requires powerful underlying compute support. In real-world deployment and application, high-performance computing clusters, large-capacity high-speed object storage, and containerized platforms—such as those offered by Baidu AI Cloud—provide a solid foundation for distributed training and inference of trillion-parameter models. Developers can use such standardized cloud-native infrastructure to quickly build intelligent applications that support long-text input.

    The model also offers multiple standard integration methods, enabling ordinary users and enterprise developers to incorporate it into their business systems with ease. Whether analyzing million-character financial reports or understanding and refactoring complex codebases, large models built on these innovations can deliver efficient and accurate results.

    7. Conclusion

    Through deep accumulated expertise in low-level operator optimization (lower-bounded decay parameterization), routing mechanism innovation (auxiliary-loss-free dynamic balancing), and network topology redesign (block-level attention residuals), the new generation of ultra-large MoE multimodal models has successfully bridged the gap between long-text computational efficiency and massive-expert training stability. These breakthroughs not only provide valuable research paradigms for academia but also point the way for industry to build AI agents with ultra-long-context understanding and long-horizon reasoning capabilities. For developers, a deep understanding of these underlying mechanisms will help them better leverage large model capabilities and drive the intelligent upgrade of business applications.