2.8T-Parameter MoE Multimodal Model: Architecture Innovations and Training Practices
This article analyzes the general technical principles behind a 2.8-trillion-parameter MoE multimodal model with about 104 billion activated parameters, native visual support, and up to 1M-token context. It covers Stable LatentMoE with 896 routed experts, quantile-balance routing, KDA lower-bounded decay for linear attention, block attention residuals, progressive context extension from 8K to 1M tokens, and Agentic RL post-training. Readers gain an engineering-oriented view of how these techniques address training stability, numerical precision, long-context efficiency, and long-range agent reasoning.
As large language models move toward ultra-large parameter counts and multimodal capabilities, mixture-of-experts (MoE) architectures have become a key path for breaking through compute bottlenecks. A recent technical report on a new ultra-large MoE multimodal model has drawn broad industry attention. The model has 2.8 trillion total parameters, about 104 billion activated parameters, native visual multimodality, and support for contexts of up to 1 million tokens. This article strips away product-specific attributes and examines the underlying general technical principles and engineering value across four dimensions: core architecture innovation, attention mechanism optimization, MoE routing strategy, and training paradigm.
1. Core Architecture Challenges and MoE Evolution
As model scale expands rapidly, traditional dense Transformer architectures face unsustainable computational costs. MoE architectures use sparse activation to expand parameter capacity without linearly increasing inference computation. However, when the number of experts reaches hundreds or even nearly a thousand, training stability, routing collapse, and communication overhead become critical engineering problems.
On the MoE side, the model adopts a Stable LatentMoE architecture with a large system of 896 routed experts. To ensure training stability under extreme sparsity, it introduces the SiTU-GLU activation function and a quantile-balance (QB) mechanism. Traditional MoE training often relies on auxiliary loss functions to enforce load balancing, but this can damage final model performance. As an auxiliary-loss-free dynamic load balancing strategy for ultra-large-scale MoE, the QB mechanism can dynamically adjust expert routing distribution without interfering with the main gradient descent path, fundamentally alleviating the “winner-takes-all” problem in massive-expert scenarios.
2. A Breakthrough in Linear Attention: KDA Lower-Bounded Decay Parameterization
When handling ultra-long contexts at the 1-million-token level, the computational complexity of traditional Softmax attention grows quadratically, prompting the industry to explore linear attention mechanisms. However, traditional Linear Attention or Delta Net has severe numerical precision bottlenecks in chunk-level computation.
2.1 Numerical Overflow Pain Points Under BF16
Model training commonly uses BF16 precision to accelerate computation. In block-level computation for Linear Attn, the inverse of the cumulative decay factor Γ may suffer severe numerical underflow or overflow under limited precision. As a result, diagonal blocks had to use an explicit position-pair path and could not fully utilize GPU Tensor Core matrix multiplication acceleration units, greatly slowing throughput.
2.2 Scaled Sigmoid and Bounded Decay
To solve this problem, the model proposes KDA lower-bounded decay parameterization. Its core idea is to change the log-decay from an unbounded Negative-Softplus mapping to a bounded Scaled Sigmoid function: a fixed decay lower bound parameter gmin is used (for example, set to -5). This change strictly confines the reciprocal scaling factor within any 16-token tile to a safe numerical range.
2.3 Operator-Level Benefits
By eliminating the special computation branch for diagonal blocks, the model allows all Causal Tiles—whether diagonal or non-diagonal—to directly call Dense Tensor Core GEMM for accelerated computation. This not only substantially improves operator implementation efficiency and throughput on CUDA/CUTLASS, but is also naturally compatible with positional-encoding-free techniques, making direct extrapolation to 1M context possible. For developers training on high-performance compute platforms or self-built GPU clusters, this kind of optimization that fully exploits lower-level hardware characteristics is key to improving cluster utilization.
3. Cross-Layer Information Sharing and Breaking Deep Representation Bottlenecks
After deep Transformer models are stacked to a certain number of layers, they often face deep representation degradation: fine-grained information from lower layers is gradually lost as it propagates upward. To break this bottleneck, the model introduces attention residual mechanisms, especially Block Attention Residuals.
AttnRes establishes cross-layer information-sharing channels at specific block levels, allowing upper attention modules to directly access hidden-state representations from lower or shallower layers. This design effectively preserves fine-grained feature information and improves the accuracy of long-range reasoning. At the engineering level, it also optimizes memory footprint and inter-node communication overhead. Combined with gated multi-head latent attention, the model greatly enhances the richness of feature representation without adding extra computational redundancy.
4. Multi-Stage Pretraining and Agentic RL Post-Training
Training for ultra-long context cannot be done in one step; pretraining directly on 1M tokens would make convergence difficult. The model adopts a progressive context extension strategy: 8K -> 64K -> 256K -> 1M. This multi-stage pretraining allows the model to first learn semantic composition in short texts and then gradually adapt to global dependencies in long texts, ensuring training efficiency and final quality.
In the post-training stage, the model adds million-token-level Agentic RL. Traditional RLHF mainly targets alignment for single-turn question answering, while Agentic RL focuses on complex tool invocation, multi-step planning, and long-range reasoning tasks. By building massive agent task environments, the model learns through self-evolution how to maintain goal consistency across extremely long interaction histories, effectively addressing the core challenge of long-range reasoning for agents.
5. Summary and Engineering Implications
In summary, the success of this new trillion-parameter MoE model lies not only in stacking parameter scale, but also in deeply reconstructing low-level operators, routing mechanisms, and training paradigms. KDA parameterization solves the parallelization challenge of linear attention on Tensor Cores; the QB mechanism and Stable LatentMoE ensure training stability for massive experts; and progressive context extension plus Agentic RL give the model the ability to handle ultra-long and complex tasks.
For enterprises and developers, these technical innovations provide highly valuable paths for reuse. In practical large-model engineering, whether teams rely on large-scale GPU clusters or self-built compute infrastructure, deeply understanding and borrowing these low-level optimization strategies can help improve training stability and inference efficiency, and support the real-world deployment of general AI technologies in complex business scenarios.
