Long-Context MoE LLMs: Key Technical Innovations and Engineering Practices
This article breaks down the core technical innovations behind a recently released large language model featuring a 2.8-trillion-parameter MoE architecture, 104B activated parameters, native visual multimodality, and a 1M-token context window. It examines how bounded-decay parameterization (KDA) enables numerically stable linear attention on Tensor Cores, how loss-free quantile balancing solves load imbalance in near-thousand-expert MoE training, and how attention residuals and multi-stage long-context training strategies push the frontier of ultra-large-scale model engineering.
In recent years, the capability boundaries of large language models have been pushed to new heights. Two core dimensions of this evolution stand out: the extreme extension of context windows, which have grown from a few thousand tokens to the million-token scale, and the scaling of model parameters through Mixture-of-Experts (MoE) architectures that dramatically expand model capacity while keeping compute costs under control. However, where these two trends converge, a series of fundamental engineering challenges emerge: How can linear attention fully exploit Tensor Core compute while remaining numerically stable? How can training stay efficient and stable under extreme sparsity with trillions of parameters and nearly a thousand experts? And how can we break through the representational bottleneck of very deep Transformers?
A recently released technical report presents a complete set of solutions to these problems. The model adopts an MoE architecture with 2.8 trillion total parameters and 104 billion activated parameters, natively supports visual multimodal input, and offers a context window of up to 1 million tokens. The innovations disclosed in its technical report not only underpin the model’s top-tier performance but also provide highly valuable engineering paradigms for the industry. This article sets aside the specific product context and focuses on the core technical innovations, analyzing their key design paths and reusability.
1. Long Context Meets Sparse Experts: Where Two Core Challenges Intersect
When a model’s context length grows from 8K to 1M, computational complexity and memory pressure grow exponentially. Traditional softmax attention, with its O(n²) complexity, struggles with ultra-long sequences. Linear attention reduces complexity to O(n) by decomposing the attention matrix into kernel-function form, making it a mainstream direction for ultra-long contexts. However, efficiently implementing linear attention on real hardware is far from trivial.
Meanwhile, MoE architectures enable explosive model scaling through conditional computation. But the surge in expert count introduces new problems: the “dead expert” phenomenon caused by routing imbalance, communication bottlenecks in expert parallelism, and training stability maintenance—all of which hinder further scaling. When both challenges must be solved within a single model, isolated optimizations are no longer sufficient; co-design at the architectural level becomes essential.
2. Attention Mechanism Innovation: Reconciling Numerical Stability with Hardware Efficiency
1. The Tensor Core Dilemma of Linear Attention
In standard linear attention implementations (such as Delta Net), the sequence is typically divided into fixed-size chunks to exploit the matrix-multiplication acceleration of Tensor Cores. Non-diagonal blocks (cross-chunk information exchange) can be computed in parallel via matrix multiplication. However, diagonal blocks (intra-chunk self-attention) require accumulating the reciprocal of decay factors for position-by-position correction, which easily triggers numerical underflow or overflow under BF16 precision.
This numerical instability prevents diagonal blocks from directly calling dense GEMM (General Matrix Multiply) kernels like non-diagonal blocks do, forcing them to fall back to explicit position-pair computation paths and severely sacrificing efficiency. This is the root cause of limited long-context inference performance in many linear attention variants.
2. Bounded Lower-Bound Decay Parameterization (KDA)
The core breakthrough lies in fundamentally restructuring how decay factors are parameterized. Traditional approaches use an unbounded negative-Softplus function to parameterize the decay rate in log space, meaning the reciprocal of the decay factor can approach infinity in extreme cases and trigger numerical issues.
The new approach (KDA) adopts a “bounded lower-bound” parameterization strategy. By mapping the decay rate computation into an interval with an explicit lower bound—for example, using a scaled and shifted Sigmoid function—it ensures that after accumulation over any 16-token chunk, the reciprocal scaling factor always stays within a numerically safe range for the hardware. This seemingly minor change yields enormous benefits: it eliminates the special computation branch for diagonal blocks, allowing all causal tiles (both diagonal and non-diagonal) to uniformly invoke highly optimized dense Tensor Core GEMM kernels. This not only greatly improves throughput but also enables the model to achieve stable direct extrapolation at 1M context length even without positional encoding (NoPE), clearing a critical obstacle for efficient training and inference on ultra-long sequences.
3. Gated Multi-Head Latent Attention and Attention Residuals
Beyond low-level numerical optimization of linear attention, the model also integrates multiple attention mechanisms. On one hand, it draws on the design of Gated Multi-head Latent Attention (Gated MLA), using low-rank compression and gating mechanisms to improve the information extraction capability of attention heads. Even more notable is the introduction of the Attention Residual (AttnRes) mechanism. In deep Transformer networks, gradients must traverse dozens or even hundreds of layers, and information faces decay risk during layer-by-layer propagation. Attention residuals share or pass attention information across layers, providing a more direct “highway” for deep networks, effectively alleviating the depth representation bottleneck while also helping reduce memory and communication overhead.
3. The Sparse Art of MoE Architecture: Synergy of Stability and Balance
1. Stable LatentMoE Architecture
For expert architecture design, the model adopts an architecture called Stable LatentMoE. Its core idea is to introduce a “latent representation” as an intermediate layer for routing decisions. Compared to routing directly on token representations, this compress-then-match mechanism more stably captures the core semantic features of tokens, improving routing accuracy and consistency. Combined with the SiTU-GLU activation function, the architecture maintains efficient state flow under extreme sparsity (only about 3.7% of parameters activated).
2. Loss-Free Quantile Balancing
In ultra-large-scale MoE with nearly a thousand routed experts, load balancing is decisive for training success. Traditional load balancing strategies typically rely on adding penalty terms to auxiliary loss functions, which introduces new hyperparameters and may interfere with the optimization of the main task objective.
The proposed Quantile Balancing (QB) mechanism completely abandons reliance on auxiliary losses. Its core logic is elegant: it assigns experts based on the distribution of routing scores across all tokens rather than the absolute score of each token. By setting a quantile threshold, the mechanism dynamically adjusts the acceptance threshold for each expert, ensuring a naturally balanced distribution of token counts between high-load and low-load experts. This non-parametric dynamic balancing strategy is extremely lightweight to implement (minimal code changes), yet it fundamentally solves the load imbalance problem of ultra-large-scale MoE and ensures stable training progress.
4. Training Strategy: Multi-Stage Progression and Agentic Reinforcement Learning
The success of ultra-long context relies not only on architectural innovation but also on carefully designed training strategies. The model was not trained directly at 1M context; instead, it followed a multi-stage pre-training curriculum of 8K → 64K → 256K → 1M. This progressive approach allows the model to quickly learn fundamental language and knowledge capabilities on short contexts before gradually extending to long-sequence modeling, effectively reducing the convergence difficulty of ultra-long-context training.
On top of this, the model undergoes a million-token-scale Agentic RL post-training stage. Traditional RL stages are typically constrained by context length, making it difficult to handle complex tasks requiring multi-turn tool calls or long-horizon planning. By performing reinforcement learning in million-token-scale environments, the model learns to effectively retrieve, integrate, and reason over information within ultra-long contexts—a key reason its long-horizon agent task execution far surpasses that of comparable models.
5. Conclusion: Lessons from a Technical Paradigm
From an engineering practice perspective, the insights from this technical report are profound. It demonstrates that in pursuing extreme model performance, low-level operator numerical stability (such as KDA), system-level load balancing strategies (such as QB), and architectural designs for cross-layer information flow (such as AttnRes) are of equal importance. The KDA solution achieves massive performance gains with minimal engineering changes, showcasing clever thinking in hardware-software co-design. The quantile balancing mechanism represents an evolution of MoE load balancing from “heuristic penalties” toward “data-driven dynamic balancing.” And attention residuals provide a new dimensional reference for the architectural design of ultra-deep networks.
Facing future challenges of even larger models and longer contexts, these technical solutions undoubtedly provide the entire industry with a highly valuable “operating-system-level” optimization reference. For developers, deeply understanding and absorbing these innovations will help build more efficient and stable large-scale model applications in practice. Technological evolution is never a single-point breakthrough—it is a comprehensive victory of systems engineering.
