Architectural Evolution of Trillion-Parameter MoE LLMs and Breakthroughs in Ultra-Long-Context Training

1. Introduction: Core Engineering Challenges for Ultra-Large MoE Models

As large language models evolve rapidly toward multimodal capabilities and ultra-long context windows, traditional dense models are hitting ceilings in both compute scaling and inference cost. The Mixture-of-Experts (MoE) architecture, with its parameter decoupling and sparse activation properties, has become the mainstream choice for building trillion-parameter models. However, when total parameter counts reach the trillion level and native support for million-token contexts is required, engineering teams face three major challenges.

First, ultra-long contexts cause attention computation complexity to surge and GPU memory consumption to explode. Second, massive numbers of routed experts under extreme sparsity can lead to training collapse and load imbalance. Third, long-horizon agentic reasoning imposes stringent requirements on a model’s ability to preserve cross-layer information and maintain logical planning capabilities.

This article dissects an industry-leading MoE multimodal LLM technical solution, exploring how it systematically addresses these engineering and algorithmic challenges through fused attention modules, dynamic load balancing strategies, and a multi-stage training paradigm.

2. Core Architecture Design: Dual Innovations in Sparse Routing and Attention Mechanisms

This technical solution defines an ultra-large MoE architecture with 2.8 trillion total parameters and 104 billion activated parameters. To support efficient training and inference at this scale, the underlying operators have been deeply restructured.

2.1 Fused Attention: Synergy Between KDA and Gated Latent Attention

When processing million-token contexts, the quadratic complexity of traditional Softmax attention becomes a critical bottleneck. While linear attention can effectively reduce computational complexity, it suffers from extremely poor parallel efficiency on underlying hardware such as CUDA and Tensor Cores, and is highly prone to numerical overflow or underflow under limited precision such as BF16.

To address this, the solution fuses KDA (Kernelized Delta Attention) with gated multi-head latent attention. The core breakthrough lies in the lower-bounded decay parameterization of KDA. In traditional linear attention, diagonal block computation requires special code paths because the reciprocal of the cumulative decay factor is unbounded, preventing full utilization of Tensor Core matrix multiplication acceleration. By reconstructing the log-decay from an unbounded negative-softplus formulation to a lower-bounded scaled sigmoid, the reciprocal scaling factor within any 16-token tile is locked into a safe numerical range.

Engineering benefit: this small algorithmic change eliminates the special handling path for diagonal blocks, allowing all causal computation blocks to directly invoke high-throughput dense Tensor Core GEMM operators. This not only substantially improves linear attention throughput, but also natively supports million-token context extrapolation without positional encoding (NoPE).

2.2 Block Attention Residuals: Breaking Through Deep Representation Bottlenecks

In very deep Transformer networks, information tends to degrade or get lost during cross-layer propagation. This solution introduces a block attention residual mechanism. By establishing residual connections between specific deep network blocks, it enables efficient cross-layer information sharing.

Engineering benefit: Block Attention Residuals not only effectively alleviate deep network representation bottlenecks and improve the model’s ability to capture long-range dependencies, but also reduce memory consumption and cross-layer communication overhead during deep Transformer training. It is an excellent practice that balances model performance and engineering cost.

3. MoE-Side Engineering Optimization: Ensuring Training Stability Under Extreme Sparsity

In the Stable LatentMoE architecture with nearly a thousand routed experts, ensuring balanced token distribution across experts is key to training stability. Traditional approaches typically introduce auxiliary losses to constrain routing distributions, but this not only harms convergence on the primary task, but also struggles to adapt to dynamically changing data distributions.

3.1 Quantile Balancing (QB): Auxiliary-Loss-Free Rapid Dynamic Balancing

This solution adopts a quantile-based balancing mechanism. The QB strategy abandons traditional auxiliary losses and instead dynamically monitors and adjusts expert load distribution at runtime. Its core logic identifies overloaded and idle experts through statistical quantiles, then smoothly guides token traffic to idle experts through lightweight bias adjustments.

The following pseudocode illustrates a common engineering implementation of auxiliary-loss-free dynamic load balancing:

# Pseudocode: Quantile Balancing dynamic routing adjustment logic
def dynamic_routing_balance(router_logits, expert_capacity):
    # Compute the raw probability of each token over all experts
    probs = softmax(router_logits, dim=-1)
    # Compute the expected load of each expert for the current step
    expected_load = probs.sum(dim=0)
    # Compute load quantiles to identify overloaded and idle experts
    median_load = torch.median(expected_load)
    overload_mask = expected_load > (median_load * 1.2) # overload threshold
    underload_mask = expected_load < (median_load * 0.8) # idle threshold
    # Dynamically adjust bias terms without gradient computation,
    # keeping the primary task loss unaffected
    bias_adjust = torch.zeros_like(expected_load)
    bias_adjust[overload_mask] = -0.1 # reduce probability of selecting overloaded experts
    bias_adjust[underload_mask] = 0.1  # increase probability of selecting idle experts
    # Apply adjusted logits for Top-K routing
    adjusted_logits = router_logits + bias_adjust
    return top_k_routing(adjusted_logits, k=expert_capacity)

3.2 Introduction of the SiTU-GLU Activation Function

Within the expert networks, the solution adopts the SiTU-GLU activation function. Compared to traditional ReLU or GeLU, the gated linear unit variant provides smoother gradient surfaces when processing ultra-large sparse features, further ensuring training stability under extreme sparsity conditions.

4. Training Paradigm and Post-Training Strategy: From Multi-Stage Pretraining to Agentic RL

4.1 Progressive Multi-Stage Pretraining

To balance training efficiency and long-context adaptability, the pretraining phase adopts a progressive context extension strategy of 8K -> 64K -> 256K -> 1M.

In the early stages (8K/64K), the model focuses on general knowledge absorption and foundational logical capability building, iterating rapidly on high-throughput compute clusters. In the later stages (256K/1M), the model progressively incorporates long-text and multimodal data, using positional encoding interpolation or NoPE extrapolation techniques to gradually adapt to ultra-long sequence dependency capture. This avoids the massive compute waste of directly training on long contexts from the start.

4.2 Agentic RL Post-Training

Targeting the pain points of long-horizon agentic reasoning, the post-training phase adds agentic reinforcement learning at the million-token level. Traditional RLHF primarily aligns with human preferences, but in agent scenarios, models need to continuously invoke tools, process multiple rounds of environmental feedback, and maintain long-term goal consistency without drift.

By introducing Agentic RL, the model continuously learns through trial and error in multi-step tool invocation, code execution, and state tracking, effectively overcoming the core challenges of instruction forgetting in long contexts and intermediate-step hallucination.

5. Enterprise Implementation Insights: Underlying Compute and Cloud-Native Architecture Support

Training and inference for such massive MoE models cannot be achieved without robust underlying AI infrastructure. For enterprise users looking to reproduce or apply such architectures, close attention should be paid to network communication and storage throughput capabilities of the underlying compute platform.

5.1 Heterogeneous Compute and High-Speed Interconnects

All-to-All communication in MoE architectures can easily become a bottleneck. When deploying such models, organizations should select container platforms and bare-metal compute clusters that support high-speed interconnects to minimize communication latency for cross-node expert routing.

5.2 Multimodal Data Pipelines

Native visual multimodal support requires massive volumes of high-quality image-text paired data. Enterprises can leverage high-concurrency read capabilities of object storage, combined with message queues, to build asynchronous data preprocessing pipelines that ensure efficient multimodal data injection into GPUs.

5.3 Observability and Training Fault Tolerance

For training interruption and recovery of ultra-large models, a comprehensive logging service and monitoring alerting system is essential. Through real-time monitoring of loss curves and expert load distribution, combined with high-availability storage snapshots, minute-level training checkpoint recovery can be achieved.

6. Conclusion

The breakthroughs of next-generation ultra-large MoE multimodal models lie not only in parameter scale, but in the deep synergy between underlying algorithms and engineering operators. From the extreme efficiency gains of lower-bounded decay parameterization on Tensor Core computation, to auxiliary-loss-free dynamic routing balance, to the fusion of progressive long-context training and Agentic RL, this series of technical solutions provides a highly valuable engineering paradigm for pushing the boundaries of LLM capabilities. As AI infrastructure continues to evolve, we have reason to expect more efficient, stable, and long-horizon reasoning-capable general-purpose agents to accelerate toward real-world deployment.