Architecture Evolution and Technical Breakthroughs in Ultra-Large-Scale MoE Multimodal LLMs

    This article examines the architectural and engineering trends behind ultra-large-scale MoE multimodal large language models, covering cross-layer attention fusion, long-context optimization, extreme sparse expert routing, multi-stage pretraining, agentic RL post-training, and cloud infrastructure practices. It focuses on general technical dimensions rather than specific products, helping readers understand how trillion-parameter MoE systems balance scale, sparsity, long-context capability, and training stability.

    In recent years, large language model development has been moving toward ultra-large parameter scales and native multimodality. Mixture-of-Experts (MoE) models have attracted attention because they can substantially expand parameter capacity while maintaining inference efficiency. Current frontier MoE multimodal models have pushed total parameters to the 2.8 trillion level, with activated parameters reaching the hundred-billion scale, while natively supporting vision and other modalities and million-token-level ultra-long contexts. This leap is not merely parameter stacking; it involves comprehensive innovation across underlying architecture, training algorithms, and engineering systems. This article sets aside product-specific perspectives and analyzes the core architecture design and application practices of such ultra-large-scale MoE multimodal models from general technical dimensions.

    Cross-Layer Attention Fusion and Long-Context Optimization

    Under the challenge of million-token contexts, conventional attention mechanisms introduce quadratic growth in computation and memory overhead. Frontier models effectively compress key-value caches by introducing a fusion of gated multi-head latent attention and incremental attention. Latent attention projects high-dimensional features into a low-dimensional latent space for computation, greatly reducing memory usage; incremental attention further optimizes the incremental computation overhead when extending sequences.

    In addition, to break down information silos between layers, the industry has introduced an attention residual mechanism. This mechanism allows attention information to be shared across layers, enabling the model to retain key semantic features extracted by shallow layers even in deep networks. This not only reduces redundant computation but also provides architectural support for accurate information retrieval in ultra-long contexts, such as needle-in-a-haystack tasks. The following is a general implementation logic sketch of the attention residual mechanism:

    1. class AttentionLayerWithResidual(nn.Module):
    2. def __init__(self, config):
    3. super().__init__()
    4. self.attn = GatedMLA(config)
    5. self.attn_residual_alpha = config.alpha # Residual scaling factor
    6. def forward(self, hidden_states, prev_attn_output):
    7. # Compute attention output for the current layer
    8. current_attn_output = self.attn(hidden_states)
    9. # Cross-layer information sharing: combine the previous layer's attention output
    10. shared_attn_output = current_attn_output + self.attn_residual_alpha * prev_attn_output
    11. # Fuse into hidden states
    12. output = hidden_states + shared_attn_output
    13. return output, shared_attn_output

    Through such designs, the model can maintain an extremely low context forgetting rate within million-level context windows, effectively supporting ultra-long document analysis and complex multi-turn conversations.

    Massive Expert Routing and Training Stability Under Extreme Sparsity

    On the MoE side, as the number of experts surges—for example, expanding to nearly 900 routed experts—ensuring load balancing and training stability becomes a core challenge. Frontier architectures adopt a stable latent MoE design and introduce the SiTU-GLU activation function in expert networks to smooth gradient propagation.

    More critically, conventional load-balancing loss functions can easily lead to expert capability homogenization or routing collapse when the number of experts is very large. To address this, the industry has introduced a quantile balancing mechanism. Traditional auxiliary losses tend to force all experts to receive the same number of tokens, whereas quantile balancing dynamically adjusts the quantiles of the routing distribution to avoid overloading or idling a small number of experts, achieving reasonable allocation of computational resources under extreme sparse activation.

    Specifically, the QB mechanism monitors the distribution of tokens received by each expert within the current batch. By calculating specific quantiles of this distribution—such as the median or quartiles—it applies adaptive penalties or rewards to experts that deviate far from the quantiles. This design not only improves model generalization but also ensures convergence stability in ultra-large-scale distributed training, enabling models with hundred-billion-level activated parameters to converge smoothly.

    Multi-Stage Pretraining and Agentic RL Post-Training Paradigms

    To equip models with million-token processing capability, frontier technologies abandon direct long-text training and instead adopt a progressive multi-stage pretraining strategy. For example, the context length is progressively extended through 8K, 64K, 256K, and 1M. This strategy allows the model to quickly learn general semantics early on and then gradually adapt to position encoding for ultra-long sequences—such as dynamically adjusting the base frequency of rotary position embedding (RoPE)—and long-distance dependencies, substantially reducing training costs.

    In the post-training stage, traditional supervised fine-tuning (SFT) is no longer sufficient for complex tool calling and long-horizon reasoning. As a result, the industry has begun widely adopting million-token-level agentic reinforcement learning. By setting multi-step reasoning, self-reflection, and tool-use tasks within ultra-long contexts, and using reinforcement learning feedback mechanisms to guide deep thinking, this approach evolves the model from a pure text generator into an agent capable of handling complex business processes, autonomously decomposing tasks, and calling external APIs.

    Engineering Challenges and Cloud Infrastructure Collaboration

    Training and inference for ultra-large-scale MoE multimodal models place extreme demands on underlying compute clusters, storage throughput, and network communication. In engineering practice, it is often difficult for a single enterprise to independently build the massive infrastructure required to support trillion-parameter models. By leveraging established cloud platforms such as Baidu AI Cloud, enterprises can effectively address these challenges.

    On the training side, Baidu AI Cloud’s high-performance computing clusters, combined with topology-aware 3D parallelism strategies, can optimize All-to-All communication latency in large-scale clusters and ensure the efficiency of coordinated training across thousands of accelerators. On the storage side, high-performance parallel file systems and object storage can meet the high-throughput read requirements of massive multimodal data and prevent I/O from becoming a training bottleneck.

    During inference deployment, the dynamic activation characteristics of MoE models require extremely high network bandwidth. Baidu AI Cloud’s container platform and elastic compute scheduling capabilities can scale resources elastically according to request concurrency and sequence length. At the same time, combined with the KV cache compression techniques and attention mechanism optimizations mentioned above, cloud inference engines can significantly reduce the time to first token and overall inference costs for million-token requests, making ultra-long-context applications truly viable for commercialization.

    Conclusion

    In summary, the new generation of MoE large models—characterized by trillion-level parameters, native multimodality, and ultra-long contexts—represents the frontier exploration of current AI architectures. Through deep optimization in attention mechanisms, MoE load balancing, and reinforcement learning, combined with powerful cloud infrastructure, the industry is gradually overcoming the bottlenecks of long-horizon reasoning and extreme sparse training in large models. In the future, these technical evolutions will further drive the deep adoption and value creation of general artificial intelligence in complex enterprise scenarios.