Scaling to Trillions: Architecture Evolution and Key Technologies of Large-Scale MoE Multimodal Models
As large language models scale toward trillions of parameters, native multimodality, and million-token contexts, Mixture-of-Experts (MoE) has become the dominant industry architecture. This article breaks down the key technical advances behind a trillion-parameter multimodal MoE model—from the lower-bounded decay parameterization that unlocks linear attention efficiency, to auxiliary-loss-free load balancing via quantile-based expert routing, and multi-stage context expansion combined with agentic reinforcement learning. It offers developers practical insights into architecture design, training stability, and engineering trade-offs.
The rapid evolution of large language models is being driven by three parallel trends: ever-larger parameter counts, native multimodal input support, and ultra-long context windows. Among the architectures enabling this shift, Mixture-of-Experts (MoE) has emerged as the industry’s preferred choice due to its strong balance between computational efficiency and model capacity. However, as total parameters surpass the hundred-billion mark and approach the trillion scale, two fundamental challenges come to the fore: maintaining training stability under extreme sparsity, and breaking through the bottlenecks of long-context computation and expert load balancing. Drawing on recent technical reports from state-of-the-art large-scale MoE multimodal models, this article provides an in-depth analysis of architecture design, attention mechanism optimization, and training paradigms, offering hands-on engineering perspectives for developers.
I. Overall Architecture: Trillion-Parameter MoE with Native Multimodality
At the architectural level, next-generation large models typically adopt a sparse MoE design with total parameters reaching the trillion scale (e.g., 2.8 trillion), while keeping activated parameters in the hundred-billion range (e.g., 104 billion). This architecture natively supports multimodal inputs such as vision, and can handle context windows of up to 1 million tokens.
On the MoE side, the model is equipped with a stable latent MoE architecture housing hundreds of routing experts (e.g., 896). To enhance the expressive capacity of expert networks, activation functions such as SiTU-GLU are introduced to strengthen nonlinear feature extraction. In addition, ensuring training stability under extreme sparsity requires both carefully designed parameter initialization strategies and robust backend compute infrastructure. For instance, large-scale distributed training on clusters with thousands or tens of thousands of GPUs—such as those provided by Baidu AI Cloud—relies on efficient containerized scheduling and distributed training frameworks to support stable pre-training of models at this scale.
II. Core Innovation: Deep Optimization of Attention Mechanisms
The attention mechanism remains the cornerstone of Transformer architectures. To handle million-token contexts, the industry has turned to variants such as linear attention—but these introduce new engineering challenges at the hardware level.
1. Lower-Bounded Decay Parameterization to Eliminate Numerical Underflow
Traditional linear attention mechanisms compute cumulative decay factors within each chunk. Due to the reciprocal operations involved, these computations are highly susceptible to numerical overflow or underflow under limited precision formats such as BF16. As a result, diagonal blocks were forced to take an explicit position-aligned computation path, which prevented them from fully utilizing the matrix-multiplication acceleration of underlying tensor cores—severely hampering throughput.
The new architecture introduces a “lower-bounded decay parameterization” technique, which modifies the log-decay from an unbounded Negative-Softplus to a bounded Scaled Sigmoid. By fixing a minimum decay lower bound (for example, set to −5), the reciprocal scaling factor within any 16-token block is locked into a numerically safe range. This seemingly minor engineering change completely eliminates the special computation branch for diagonal blocks, allowing all causal blocks (both diagonal and off-diagonal) to directly invoke dense matrix multiplication operators. This significantly improves the computation efficiency of linear attention and, as a natural byproduct, supports ultra-long context extrapolation without positional encoding.
2. Gated Multi-Head Latent Attention and Block Attention Residuals
Beyond parameterization, the architecture also integrates gated multi-head latent attention, which uses gating mechanisms to dynamically control information flow across different attention heads, effectively reducing the memory footprint of KV caches. In parallel, a block attention residual mechanism is introduced to enable cross-layer information sharing, breaking through the representational bottlenecks common in deep Transformer networks. These residual connections not only optimize gradient backpropagation paths but also substantially reduce memory and communication overhead in the deeper layers of the network.
III. Training Stability: Load Balancing Without Auxiliary Losses
In a large-scale MoE model with nearly a thousand experts, uneven expert load distribution is a primary cause of training collapse. The conventional remedy has been to introduce auxiliary losses that penalize load imbalance—but these often degrade performance on the primary task.
To address this challenge, the industry has proposed an auxiliary-loss-free, high-speed dynamic load balancing strategy: the quantile-balanced mechanism. This approach discards the traditional auxiliary loss function entirely. Instead, it dynamically adjusts the acceptance probability of each expert by statistically analyzing the quantiles of routing weights. This quantile-driven adjustment achieves efficient and balanced load distribution across experts without interfering with the main gradient propagation path. It not only ensures training stability under extreme sparsity but also avoids the performance penalties associated with auxiliary losses—a key breakthrough for large-scale MoE training.
IV. Training Paradigm: Multi-Stage Pre-Training and Agentic Reinforcement Learning
The model’s strong performance is not solely attributed to architectural innovation; a well-designed training paradigm plays an equally important role.
1. Multi-Stage Context Extension Pre-Training
To achieve a smooth transition from short texts to million-token long documents, the model adopts a multi-stage pre-training strategy: starting at 8K tokens, then progressively advancing to 64K, 256K, and finally reaching 1M. This gradual extension allows the model to adapt to the positional encoding and dependency patterns of increasingly long sequences, avoiding the computational waste and convergence difficulties that would arise from training directly on ultra-long contexts.
2. Agentic Reinforcement Learning for Post-Training
Following pre-training, the model undergoes a post-training phase incorporating million-token-level agentic reinforcement learning (RL). This stage targets the core challenge of long-horizon reasoning. By designing complex agent tasks, the model engages in multi-turn interactions, tool invocation, and long-range planning, significantly improving its logical reasoning and task decomposition capabilities in complex business scenarios. For practical deployment, developers can combine cloud-native components—such as message queue and logging services—to build an end-to-end evaluation and data-feedback loop that continuously refines model performance.
V. Summary and Engineering Takeaways
The new generation of hundred-billion-parameter multimodal MoE models delivers a compelling architectural blueprint. From lower-bounded decay parameterization that accelerates underlying operators, to auxiliary-loss-free quantile-based balancing that ensures stable training across thousands of experts, and to multi-stage context extension with agentic RL as a training innovation—each technical breakthrough removes a critical barrier to large-scale model deployment.
For developers and enterprise users, understanding these underlying technical decisions is valuable for assessing model capability boundaries and making sound architectural choices when building applications on top of these models. As compute platforms and distributed training frameworks continue to evolve, ultra-long-context and multimodal agent systems are well positioned to unlock significant business value across a growing range of vertical industries.
