Deep Dive into Core Technologies for Ultra-Long Context MoE LLMs

1. Background: Why Long Context and Sparse MoE Are Essential for LLMs

The competitive landscape for Large Language Models (LLMs) is shifting from sheer parameter scale to “effective capability density.” Real-world business scenarios—such as AI Agents, code repository comprehension, ultra-long document analysis, and multimodal report interpretation—require models to process hundreds of thousands or even millions of tokens in a single inference pass. Conversely, the inference cost of dense architectures scales almost linearly with parameter size, exacerbating the conflict between “capability” and “cost.” Mixture of Experts (MoE) has emerged as the mainstream technical approach to balance these demands through a sparse paradigm: maintaining a massive total parameter count while activating only a small subset of experts per token.

Recently, a publicly released technical report on a trillion-parameter multimodal MoE model sparked widespread discussion. The model features approximately 2.8 trillion total parameters with about 104 billion activated parameters per token, natively supporting visual multimodality and a 1-million-token context. It systematically addresses three major challenges across two dimensions (architecture and training systems): ultra-long context computational efficiency, training stability for massive expert MoEs, and long-horizon Agent reasoning. Rather than focusing on a specific product, this article treats it as a “technical sample” to abstract reusable design paradigms and engineering experiences.

Overall architecture overview (based on public technical reports):

• Overall Paradigm: MoE multimodal LLM with unified modeling for vision and text; pre-training context progressed through four stages: 8K → 64K → 256K → 1M.
• Attention Side: Incremental linear attention (with lower-bound decay parameterization) + Gated Multi-head Latent Attention (Gated MLA) + Block Attention Residuals.
• MoE Side: 896 routed experts per layer, combined with latent expert stabilization, SiTU-GLU gating activation function, and auxiliary-loss-free Quantile Balancing.
• Post-training: Million-token-level Agentic RL, aligned with long-horizon Agent task formats.

2. Three Core Challenges: Where the Difficulty Lies

2.1 Computational and Memory Bottlenecks at the Million-Token Scale

Standard Transformer self-attention computation grows quadratically with sequence length. Expanding context from 8K to 1M increases the sequence length by 128x, which inflates attention computation by roughly 16,000x. Even with IO-aware chunked attention optimizations, this only reduces memory access overhead without changing the magnitude of quadratic complexity. Furthermore, the KV Cache expands linearly with sequence length; a million-token cache can easily consume the majority of VRAM on a single accelerator card. The conclusion is clear: making million-token context a “usable and affordable” capability requires paradigm-level transformations to the attention mechanism itself—linearization and KV compression are both indispensable, rather than simply scaling hardware.

2.2 Training Stability for Massive Expert MoEs

The benefit of MoE comes from sparse activation, but when the number of experts scales to the hundreds, routing issues are drastically amplified. Load imbalance leads to some experts being overtrained while others are undertrained; routing distributions fluctuate violently in the early stages of training; and gradient noise rises with sparsity. The traditional solution is to append an auxiliary balancing loss to the training objective, but this inherently conflicts with the main objective—if the loss weight is too high, language capabilities suffer; if too low, balancing fails. In the extreme sparse scenario of 896 experts per layer, this conflict becomes particularly acute, necessitating a balancing mechanism that does not “pollute” the main objective.

2.3 Long-Horizon Reasoning in Agent Scenarios

Real-world Agent tasks often involve dozens of rounds of tool calls and continuous decision-making spanning hundreds of thousands of tokens. Supervised fine-tuning struggles to cover error accumulation and credit assignment over long trajectories: the model must both “remember and locate” information within ultra-long contexts and “think far ahead and correct itself” during multi-step decisions. This places systematic demands on pre-trained context length, post-training task formats, and the throughput and stability of the training framework.

3. Breakdown of Key Technologies

3.1 Lower-Bound Decay Parameterization for Linear Attention: Embedding Numerical Safety into the Formula

Pain Point: Delta-rule-based linear attention achieves constant-cost long-sequence modeling via a “state matrix + per-position decay” approach. Its decay factor is typically parameterized as an unbounded negative value (e.g., negative softplus). In chunked parallel implementations commonly used during training, intra-chunk computation requires taking the reciprocal of the cumulative decay factor, which easily overflows or underflows under BF16 finite precision. To bypass numerical issues, engineering implementations had to reserve a slow path for the intra-chunk diagonal region, explicitly calculating by “position pairs.” This prevented that portion of computation from utilizing matrix multiplication acceleration units, significantly dragging down operator throughput.

Mechanism: The idea behind lower-bound decay parameterization is straightforward—change the value range of the logarithmic decay from unbounded to having a lower bound:

g_t = g_min * sigmoid(a_t), where g_min = -5

# Traditional parameterization: unbounded value range, reciprocal of cumulative decay easily overflows
g_t = -softplus(a_t)            # g_t ∈ (-∞, 0)
# Lower-bound decay parameterization: value range locked in a safe interval
g_t = g_min * sigmoid(a_t)      # g_t ∈ [g_min, 0), g_min = -5

Thus, the reciprocal scaling factor of the cumulative decay within any 16-token sub-block is locked within a numerically safe range. The intra-chunk diagonal and non-diagonal regions can uniformly utilize the dense matrix multiplication path, completely eliminating special branches.

Benefits: First, all causal tiles are unified into dense matrix multiplication, massively boosting operator throughput. Second, because the decay mechanism itself carries the positional information, the model can discard explicit positional encoding (NoPE) and extrapolate directly on million-token contexts. This experience is highly reusable: numerical stability should not rely on post-hoc precision patching, but should guarantee a safe value range during the parameterization design phase, requiring minimal engineering changes.

3.2 Gated Multi-head Latent Attention + Block Attention Residuals: Dual Optimization of Memory and Depth

Multi-head Latent Attention (MLA) compresses the KV Cache by projecting keys and values into a low-rank latent space, serving as an effective method to reduce memory footprint during long-context inference. Introducing a gating factor on top of this allows each attention head to learn to suppress irrelevant information, enhancing the stability of attention distributions over long sequences. This is particularly crucial for real-world scenarios where “only a small amount of key information exists within a million tokens.”

Block Attention Residuals extend the residual concept from “adjacent layers” to “cross-layer sharing,” establishing direct connection channels for attention information between distant layers. Its value is twofold: first, it alleviates the representation bottleneck that emerges in deep Transformers as depth increases, allowing information to “take a shortcut” directly to deeper layers; second, reusing attention computation results across layers reduces the memory footprint of repeated activations and communication overhead under tensor parallelism. For ultra-large models with dozens or hundreds of layers, this structural-level optimization is continuously amplified in the depth direction.

3.3 Auxiliary-Loss-Free Quantile Balancing: From “Loss Constraints” to “Dynamic Control”

Faced with the extreme sparse scenario of 896 routed experts per layer, new solutions generally abandon auxiliary losses in favor of direct control over the routing process. The core idea is to maintain a dynamic routing bias for each expert. During training, the quantile of the overall expert load serves as a dynamic baseline—experts with loads higher than the target quantile have their biases decreased, while those lower have their biases increased, gradually converging the load distribution to the target shape.

# Auxiliary-loss-free load balancing (conceptual illustration)
for expert in experts:
    if load[expert] > quantile_target:
        bias[expert] -= step
    else:
        bias[expert] += step

Because the bias only affects “who is selected” and does not enter the loss or gradient computation, the main task objective remains completely unpolluted, and the balancing intensity can be adjusted in real-time as training progresses. Combined with novel gating activation functions like SiTU-GLU to improve gradient flow and functional differentiation among experts, this combination allows an MoE with hundreds of experts to remain stable throughout the pre-training process. For any team planning to train large-scale sparse models, this is a cleaner engineering choice than auxiliary losses.

3.4 Multi-Stage Context Expansion + Agentic RL: Capabilities Are Trained via a “Curriculum”

Million-token context is not achieved in one step. Mainstream approaches use curriculum-based multi-stage pre-training: starting from 8K and progressively expanding to 64K, 256K, and finally 1M. At each stage, the data mixture (proportions of long documents, code repositories, multi-turn conversations) and positional encoding strategies are adjusted simultaneously. Because the aforementioned linear attention natively supports NoPE extrapolation, the engineering resistance to context expansion is significantly reduced.

Building upon pre-training, Agentic RL post-training scales task sizes to the million-token level. The model learns by trial and error in real-world task formats like multi-turn tool calls and long document analysis, with reward signals covering both final results and critical intermediate steps, thereby acquiring long-horizon credit assignment and self-correction capabilities. This validates an important judgment: long-context capabilities and Agent capabilities must be designed as a whole—the former is the carrier of the latter, and the latter is the value outlet of the former.

3.5 Mutual Enabling of Native Multimodality and Long Context

With unified modeling for vision and text, images, charts, and scanned documents are tokenized and enter the same context window. For real-world scenarios like “an entire financial report + accompanying charts” or “hundreds of pages of contracts,” multimodality and long context mutually enable each other: unified modalities reduce information splicing overhead, while the ultra-long window allows “full-text understanding” to replace “fragment stitching.” Tasks such as file comprehension, content drafting, and data analysis benefit simultaneously.

4. Reusable Engineering Insights

1. Shift numerical safety to the parameterization layer: Using bounded functions like sigmoid to lock activation/decay value ranges is more reliable than post-hoc reliance on loss scaling or gradient clipping, and comes at almost zero cost.
2. Align operator design with hardware: In chunked parallel implementations, avoid special branches that cannot utilize matrix multiplication units; the uniformity of tile granularity directly determines the throughput ceiling.
3. Move balancing mechanisms from loss to control: Auxiliary losses “fight” with the main objective; dynamic biases and quantile control are cleaner alternatives.
4. Train capabilities via a curriculum: Context length and Agent capabilities should progress in stages, with data mixture and training objectives adjusted synchronously, which is more critical than tuning a single hyperparameter.
5. Align post-training with real workflows: RL task design should closely mimic real-world formats like multi-turn tool calls and long document processing, rather than static Q&A.

5. Practical Implications for Developers and Enterprises

For most teams, the training paradigm of trillion-level models serves more as a “lens to understand cost structures”: sparse activation dictates the marginal cost structure of inference, KV compression dictates the memory threshold for long-context services, and Agentic RL dictates the reliability ceiling of models in automated workflows. When evaluating any long-context model service, it is recommended to focus on three dimensions: first-token latency and throughput stability on long texts, accuracy in multimodal document comprehension, and the success rate and controllability of Agent tasks.

Regarding implementation paths, developing a proprietary foundation model is not a necessary option. Cloud platforms—such as Baidu AI Cloud—provide general capabilities including LLM services, long-text comprehension, and Agent orchestration. Coupled with infrastructure like object storage, message queues, logging services, and monitoring alerts, developers can focus their efforts on areas that generate differentiated value, such as business data governance, prompt and workflow design, and evaluation system construction.

6. Conclusion

Ultra-long context computational efficiency, massive expert training stability, and long-horizon Agent reasoning correspond to collaborative innovations across three levels: attention mechanisms, training systems, and post-training paradigms. Lower-bound decay parameterization embeds numerical safety directly into formulas; auxiliary-loss-free balancing returns control from the loss function to the routing system; curriculum-based expansion and Agentic RL “feed” capabilities to the model in stages. This combination demonstrates that LLM competition has shifted from comparing single-point capabilities to competing in holistic engineering capabilities across architecture, systems, and training methodologies. Based on public evaluations, the benefits of this technical route on long-text and Agent tasks have been validated. For practitioners, the truly valuable takeaway is digesting these transferable design principles, rather than fixating on any specific model version.