Ultra-Long Context and Ultra-Large-Scale MoE LLMs: Key Techniques Explained

1. Background: The “Long” and “Sparse” Tracks of LLM Evolution

In recent years, competition in large model capabilities has shifted from simply stacking more parameters toward two more engineering-driven tracks: continuously extending context length, and improving compute efficiency through sparsification. The application-side drivers are straightforward:

• Long-document and multimodal understanding: Contract review, deep paper reading, and financial report analysis often involve inputs of hundreds of thousands or even millions of tokens, frequently accompanied by charts, scanned documents, and other multimodal information.
• Repository-level code tasks: Cross-file refactoring and whole-repo defect localization require the model to “read” an entire repository rather than a single file fragment.
• Long-horizon agent tasks: Agents need to maintain consistent task state and memory across multiple rounds of tool calls, and the historical trajectory itself expands into an ultra-long sequence.

At the same time, the bottlenecks of dense architectures are becoming increasingly obvious: the compute of standard attention grows quadratically with sequence length, and the KV cache expands linearly with context. If all parameters were activated densely, the inference cost of trillion-parameter models would be unacceptable. As a result, “hybrid attention + ultra-large-scale sparse MoE” has become a widely recognized direction for the industry. Recently published representative technical reports show that leading models can already achieve: trillion-level total parameters, only tens of billions of activated parameters per token, routing experts scaled to hundreds or even nearly a thousand, and native support for visual multimodality and million-token context. This article takes this class of technical solutions as its subject and breaks down the core challenges and key designs.

2. Architecture Overview: Sparse Activation and Hybrid Attention Combined

The overall approach of this class of solutions can be summarized as “decoupling capacity from cost, and layering precision with efficiency“:

1. Ultra-large-scale MoE decouples capacity from compute. The total parameter count determines the upper bound of the model’s knowledge capacity, while the number of parameters activated per token determines the actual compute overhead. Through a design with a massive number of routed experts plus a small number of shared experts, the model can keep per-token compute at the tens-of-billions activation level while maintaining trillion-level total capacity.
2. Native multimodal pretraining. Visual tokens and text tokens are jointly modeled in a unified sequence from the pretraining stage, rather than attaching a visual encoder after training is complete. This gives tasks such as chart understanding and screenshot Q&A native capability.
3. Layered hybrid attention. Some layers use gated multi-head latent attention, which projects the keys and values of multiple heads into a low-dimensional latent space, greatly compressing the KV cache while retaining precise retrieval capability. Other layers use incremental linear-like attention, which maintains historical information in a fixed-size matrix state and achieves approximately linear compute and memory-access complexity. Interleaving the two balances “accurate lookup” with “affordable compute.”
4. Cross-layer attention residuals. Cross-layer direct connections for attention outputs are introduced in deep networks to alleviate the deep-representation bottleneck while reducing memory and communication overhead.
5. Multi-stage training paradigm. Pretraining context is advanced in stages: 8K → 64K → 256K → 1M, followed by agent-oriented reinforcement learning post-training to address long-horizon reasoning and tool use.

The following sections break down the most reusable techniques one by one.

3. Key Techniques Broken Down

3.1 Bounded-Decay Parameterization for Incremental Attention: Making Linear Attention Fully Utilize Hardware

The pain point: Incremental linear-like attention compresses historical information into a fixed-size matrix state and relies on a decay factor to control the “forgetting rate.” Training commonly adopts a chunked parallel strategy: parallel computation within chunks and recursive updates across chunks. The problem is that intra-chunk computation requires the reciprocal of the cumulative decay factor for retrospective correction. If the log-decay uses a traditional unbounded parameterization (such as a softplus-based negative mapping), its reciprocal can be exponentially amplified under limited precision (e.g., BF16), causing overflow or underflow and numerical breakdown. The engineering compromise is that the intra-chunk (diagonal-block) path degrades to an explicit per-position-pair computation, unable to invoke the hardware’s dense matrix multiplication acceleration units, becoming the throughput bottleneck of the entire operator.

The solution: Change the log-decay to a bounded scaled Sigmoid parameterization, locking its value range into a safe interval:

import torch
# Traditional parameterization: log-decay has no lower bound; cumulative reciprocal may overflow
# log_decay = -torch.nn.functional.softplus(w * x)      # values in (−∞, 0)
# Bounded decay parameterization: value range locked to [g_min, 0], numerically safe
g_min = -5.0
log_decay = g_min * torch.sigmoid(scale * x)
decay = torch.exp(log_decay)   # within any 16-token chunk, the reciprocal of cumulative decay stays in a safe range

Benefits: First, diagonal blocks and non-diagonal blocks uniformly follow the dense matrix multiplication (GEMM) path, eliminating special computation branches and improving both hardware utilization and code maintainability. Second, this parameterization is naturally compatible with training without explicit positional encoding, providing a stability foundation for length extrapolation to million-level context. Third, the engineering change is minimal—a classic operator-level innovation of “small change, large payoff”—with high reuse value for teams building their own linear attention operators.

3.2 Gated Multi-Head Latent Attention: Balancing KV Cache and Expressiveness

The dominant memory cost in long-context inference is the KV cache. The idea of multi-head latent attention is to project the keys and values of each head into a low-dimensional latent space, so that at inference time only the latent vectors need to be cached and restored on demand, significantly reducing memory usage and memory-access pressure at long sequence lengths. Building on this, a gating mechanism is introduced to perform learnable soft selection over feature channels, which on the one hand enhances the model’s expressive freedom and on the other hand stabilizes the activation distribution, which is especially friendly to training very deep networks. When interleaved in layers with incremental attention, the latent attention layers handle retrieval-type tasks that need precise “look-back” to the original text, while the linear layers handle low-cost global information flow, giving a clear division of labor.

3.3 Cross-Layer Attention Residuals: Reducing Both Deep-Network Representation and Communication Burden

As layer count increases, each attention layer in a standard Transformer “rewrites” the residual stream, and critical information captured by shallow layers (such as entity references and local structure) is easily diluted layer by layer, forming a deep-representation bottleneck. Cross-layer attention residuals allow attention outputs to be shared directly across layers, similar to the idea of dense connections: shallow-layer information can reach deep layers without being reconstructed at every step. This yields three benefits: smoother gradient flow and stable scaling of depth; reduced redundant computation; and simultaneous reductions in cross-GPU communication and activation memory overhead.

3.4 Auxiliary-Loss-Free Dynamic Load Balancing: A Stabilizer for Near-Thousand-Expert Scale

The pain point: MoE relies on a router to dispatch tokens to experts. When the number of experts scales to hundreds or even nearly a thousand, “routing collapse” becomes very likely: a few popular experts are over-selected while the rest sit idle, making training both unstable and inefficient. The traditional approach introduces an auxiliary load-balancing loss, but it interferes with the main training objective and is hyperparameter-sensitive—the larger the expert scale, the harder it is to tune.

The solution: Adopt quantile-based dynamic load balancing. Expert historical loads are tracked online, mapped to quantile biases, and added to the routing scores, so that more heavily loaded experts become harder to select. The key design is that the bias only affects “who gets selected,” while the gating weights are still determined by the original routing scores, so the main loss is not polluted:

# Pseudocode: quantile-based dynamic load balancing (auxiliary-loss-free)
load_ema = ema_update(expert_token_count, momentum=0.9)  # track expert load online
bias = quantile_transform(load_ema)   # higher load → larger routing bias
scores = router(x) + bias
topk_experts = topk(scores, k=3)      # Top-3 routing
weights = softmax(router(x)[topk_experts])  # gating weights determined by original scores

Combined with an improved gating activation function (a gated linear unit variant), routing score normalization, and expert capacity control, training can remain stable under extreme sparsity.

4. Training Paradigm: Multi-Stage Context Extension and Agent Reinforcement Learning

4.1 Staged Context Extension

Ultra-long context cannot be achieved in one step; it must be advanced in stages:

Stage	Context length	Training focus
1	8K	Establish general capability and a stable baseline
2	64K	Increase long-document data ratio; switch sequence parallelism strategy
3	256K	Introduce context parallelism; strengthen retrieval and multi-hop reasoning data
4	1M	Validate extreme-length extrapolation; add long agent trajectory data

Each stage requires synchronized adjustments to data construction (long documents, code repositories, multi-turn dialogue trajectories), parallelism strategy (sequence parallelism, context parallelism), and learning rate scheduling. Effective context should be validated with “needle-in-a-haystack”-style retrieval tests and multi-hop aggregation tasks, rather than judging by the nominal length alone.

4.2 Agent-Oriented Reinforcement Learning Post-Training

Supervised fine-tuning struggles to teach long-horizon decision-making: agent tasks have sparse rewards and trajectories that often span dozens of rounds. Reinforcement learning post-training interacts with verifiable environments (tool calls, code execution, multi-turn Q&A) and optimizes the entire trajectory using task outcomes as rewards. This forms a positive feedback loop with long-context capability: long trajectories themselves are high-quality long-sequence training data, while long-context capability in turn supports a longer task horizon.

5. Engineering Takeaways and a Developer Perspective

• Evaluate “effective context”: A nominal 1M does not mean reliability everywhere. Evaluation sets should be designed around retrieval fidelity, multi-hop reasoning, and long-horizon consistency.
• Inference-side engineering: Chunked prefill, prefix cache reuse, KV quantization, and operator fusion are key to reducing the cost of long-context serving. Sparse activation does not equal linear cost, and serious capacity planning is still required.
• Infrastructure selection: Training and inference pipelines depend on general-purpose capabilities such as object storage, container platforms, GPU compute scheduling, and logging/monitoring/alerting. Mainstream cloud platforms offer products in these categories, which can be combined as needed to build data processing, training, and online serving pipelines.

6. Conclusion

Looking across the technical lineage of this generation of ultra-long-context large models, three main tracks can be distilled: the operator layer pursues numerical stability and hardware friendliness (bounded decay parameterization); the architecture layer decouples capacity from cost through hybrid attention plus massive expert sparsification; and the training layer fills in long-horizon capability through multi-stage extension and agent reinforcement learning. For developers, understanding the motivations and trade-offs behind these designs is more valuable than memorizing specific numbers—they form the basic vocabulary for selecting, evaluating, and building the next generation of large models.