Architecture Optimization and Training Practices for Ultra-Long Context MoE LLMs
This article explores the architectural optimizations and training practices for ultra-long context, extreme sparse Mixture-of-Experts (MoE) large language models. It addresses three major challenges: attention computation and memory overhead from million-token contexts, routing collapse and load imbalance in extreme sparse MoE, and state consistency in agentic long-range reasoning. We detail a hierarchical decoupled architecture integrating lower-bounded decaying linear attention, quantile-balanced auxiliary-loss-free routing, and block-level attention residuals. Furthermore, it covers a multi-stage context extension curriculum and million-token agentic reinforcement learning, providing actionable insights for engineering implementation and comprehensive evaluation.
Architecture Optimization and Training Practices for Ultra-Long Context MoE LLMs
I. Problem Background: The Triple Challenge of Ultra-Long Context, Multimodality, and Extreme Sparse MoE
When large models simultaneously move toward multimodality, million-token contexts, and ultra-large scale Mixture-of-Experts (MoE), the system encounters three types of bottlenecks. First, the attention computation and memory overhead brought by ultra-long sequences grow non-linearly, making traditional full attention economically unscalable to million tokens in both training and inference. Second, although extreme sparse MoE significantly improves parameter capacity and activation efficiency, routing collapse, expert load imbalance, and training instability amplify as the number of experts increases. Third, agentic long-range reasoning requires the model to maintain state consistency, tool-calling accuracy, and credit assignment capabilities over ultra-long trajectories, which cannot be covered by relying solely on short-context supervision.
A public technical solution proposes a general approach: using an MoE multimodal large model as the backbone, adopting an extreme sparse design with approximately 2.8 trillion total parameters, 104 billion activated parameters, and 896 routed experts. On the attention side, it fuses lower-bounded decaying incremental linear attention with gated multi-head latent attention, and achieves cross-layer information sharing through block-level attention residuals. On the MoE side, it adopts a stable latent mixture-of-experts architecture, combined with gated SiLU-like activation and a quantile balance mechanism. On the training side, it uses a multi-stage context expansion of 8K→64K→256K→1M, overlaid with million-token level agentic reinforcement learning post-training. Its core is not merely stacking parameters at a single point, but a collaborative design centered around “long-context computational efficiency, massive expert training stability, and long-range reasoning capability.”
II. Overall Architecture: Hierarchical Decoupling for Scalability
From an engineering perspective, this architecture can be decomposed into four layers:
- Representation Layer: Multimodal encoding and unified token space, allowing vision, text, and other modalities to share long-context positions and extrapolation strategies.
- Attention Layer: A mix of local and long-range. Incremental linear attention handles efficient historical compression, gated latent attention handles selective aggregation of key information, and block-level attention residuals handle cross-layer representation reuse.
- Expert Layer: Stable latent mixture-of-experts, reducing communication and computational overhead through latent space routing, and maintaining expert utilization through quantile balancing.
- Training Layer: Multi-stage context curriculum learning and agentic reinforcement learning, first allowing the model to learn to “read long,” and then to “use long.”
The value of this hierarchical decoupling is that attention efficiency, expert balance, and long-range reasoning can be tuned separately, preventing instability in one module from propagating to the entire system. For example, during the long-context extension phase, the expert routing strategy can be fixed to adjust only position extrapolation and the attention window; during the agentic post-training phase, most experts can be frozen, updating only the routing bias and policy head to reduce training costs.
III. Key Technology 1: Parameterization of Lower-Bounded Decaying Linear Attention
3.1 The Pain Point: Numerical Overflow of Cumulative Decay Factors
Linear attention and its incremental variants often need to accumulate the decay factor Γ when computing within chunks. If the decay parameter is unbounded, the reciprocal of Γ may severely overflow or underflow under limited precision (such as BF16), forcing diagonal blocks to take an explicit position-aligned path, making them unable to share the high-performance matrix multiplication calls used by ordinary causal blocks. The result is that non-diagonal blocks use dense matrix multiplication while diagonal blocks take a special branch, complicating kernel implementation, reducing throughput, and making long-context extrapolation unstable.
3.2 Mechanism: Bounded Scaling Sigmoid
The core of lower-bounded decay parameterization is to change the unbounded log-decay to a bounded scaling Sigmoid:
g_min = -5.0log_decay = g_min * sigmoid(e_A * h * z)gamma = exp(log_decay)inv_gamma = 1 / gamma # Due to the lower bound, the inverse is restricted to a safe range
Where e_A is a learnable scaling parameter, and h and z are the hidden states of the current token. Since log_decay is restricted around [g_min, 0), the inverse scaling factor within any 16-token tile falls within a safe numerical range, allowing diagonal blocks and ordinary blocks to uniformly take the dense matrix multiplication path.
3.3 Benefits and Considerations
Benefits include: eliminating the special branch for diagonal blocks to improve kernel throughput; improving BF16 training stability; and naturally supporting direct extrapolation of million-token contexts without positional encoding. Engineering considerations: the lower bound should not be too small, otherwise long-range memory decays too quickly; nor too large, otherwise the reciprocal might still approach the precision boundary. It is recommended to perform numerical sweeps based on tile length and precision, and apply clipping protection to inv_gamma during mixed-precision training.
IV. Key Technology 2: Quantile-Balanced Auxiliary-Loss-Free Routing
4.1 The Pain Point: Load Imbalance with Massive Experts
If MoE routing relies solely on Top-k scores, a “the strong get stronger” phenomenon often occurs: a few experts are overly selected while many remain idle, reducing training efficiency. Traditional auxiliary losses can alleviate the balancing issue but interfere with the main task gradients and are highly sensitive to hyperparameters in massive expert scenarios.
4.2 Mechanism: Dynamic Quantile Threshold
The idea of quantile balancing is to compute the distribution of expert routing scores at each step, take the target quantile as a dynamic threshold so that approximately the target proportion of tokens enter the corresponding expert, and then superimpose capacity constraints and tie-breaking randomization. The pseudo-code is as follows:
def quantile_balance(scores, capacity, q):threshold = quantile(scores, q)mask = scores >= threshold# Capacity constraint: re-route or drop tokens exceeding capacitymask = apply_capacity(mask, capacity)# Tie-breaking randomization to avoid fixed expert biasmask = random_tie_break(mask, scores)return mask
This mechanism does not rely on auxiliary losses, can quickly respond to load changes, and is suitable for 896-level experts and extreme sparse activation. In engineering, attention should be paid to the communication overhead of global quantile statistics; hierarchical quantiles or sliding window approximations can be adopted. Additionally, monitor the expert utilization variance, token drop rate, and re-routing rate to avoid over-balancing leading to main task loss.
V. Key Technology 3: Block-Level Attention Residuals
Deep Transformers suffer from a deep representation bottleneck: as the number of layers increases, information in the residual stream is easily diluted, leading to insufficient cross-layer feature reuse. Block-level attention residual groups several layers into a block, aggregating residual information between blocks to achieve cross-layer sharing with lower memory and communication overhead. The schematic is as follows:
for block in blocks:y = block(x)x = y + alpha * pool_block_residual(x, block.residual_cache)
The benefits are: mitigating deep gradient and representation degradation; reducing the memory pressure of saving full residuals layer by layer; and reducing cross-device communication under expert and sequence parallelism. During implementation, control the block size and aggregation weight to avoid additional latency introduced by residual pooling; it is recommended to focus on validating its gains for retrieval, long document Q&A, and tool-calling trajectories during the long-context training phase.
VI. Multi-Stage Pre-training and Agentic Reinforcement Learning Post-Training
Context expansion adopts a curriculum pre-training of 8K→64K→256K→1M: the short-context phase lays the foundation, while the long-context phase gradually increases the sequence length and the proportion of long document data, accompanied by extrapolation without positional encoding and attention window scheduling. The key is that the data ratio must change with length: the long-context phase needs to increase samples like cross-paragraph retrieval, long code dependencies, and multi-turn tool trajectories; otherwise, the model will only “read long” rather than “perform well.”
Post-training adopts million-token level agentic reinforcement learning: using long-range task trajectories as the sampling unit, designing process rewards and outcome rewards to handle long-range credit assignment. Engineering recommendations: increase trajectory length in stages; use a mix of reward models and rule-based rewards; perform segmented advantage estimation for ultra-long trajectories; and monitor policy entropy, tool-calling success rate, and trajectory repetition rate. This phase is strongly correlated with long-context attention efficiency; if inference costs are too high, sampling efficiency will become a bottleneck.
VII. Engineering Implementation and Evaluation Recommendations
On general cloud platforms, you can use object storage to save checkpoints and training data, container platforms for elastic scheduling, and logging and monitoring services to observe training metrics. Baidu AI Cloud also provides similar general services that can be combined as needed. For parallelism strategies, it is recommended to mix expert parallelism, tensor parallelism, and sequence parallelism, prioritizing the optimization of all-to-all communication and expert load balancing. Monitoring metrics should include: expert utilization variance, token drop rate, attention kernel throughput, long-context retrieval accuracy, agentic task success rate, and long-range consistency.
During evaluation, do not just look at the average score of public benchmarks; verify across dimensions: whether short-context base capabilities regress, whether long-context extrapolation is stable, whether training converges under extreme sparsity, and whether agentic long-range tasks have reproducible gains. Only by simultaneously incorporating attention numerical stability, routing balance, and long-range credit assignment into the evaluation can ultra-long context MoE large models transition from “leading in parameter scale” to “engineering usability.”
