MoE Large Models: Architecture Deep Dive on Ultra-Long Context and Sparse Training Stability

As large model parameter counts and context windows expand simultaneously, the core engineering challenge has shifted from “can we train it” to “can we train it efficiently, stably, and reproducibly.” An open-source MoE multimodal model released in July 2026, featuring 2.8 trillion total parameters, 104 billion activated parameters, native vision multimodality, and a 1-million-token context window, offers a systematic reference point. This article does not evaluate or recommend any specific product. Instead, it treats the model as a public technical sample and extracts architectural ideas and engineering methods that can be transferred to general large model development.

1. Architecture Overview: Two Main Lines of Sparse Activation and Long Context

Based on the public technical report, the architecture can be summarized as pursuing long-context efficiency on the attention side while maintaining stability under extreme sparsity on the MoE side.

• Scale and activation: The model has 2.8 trillion total parameters and approximately 104 billion activated parameters per forward pass, with an activation ratio of about 3.7%. This is a typical high-sparsity MoE design. The model’s knowledge capacity and per-token computation cost are decoupled, with scaling concentrated in the expert pool rather than the dense path.
• Attention module: It combines Kernelized Delta Attention (KDA) with Gated Multi-head Latent Attention (Gated MLA), and uses attention residuals (AttnRes) for cross-layer information sharing. These three correspond respectively to parallel efficiency in linear attention, memory efficiency in latent-space attention, and representation reuse in deep networks.
• MoE side: It uses a Stable LatentMoE architecture with 896 routed experts, combined with the SiTU-GLU activation function and a Quantile Balancing (QB) mechanism. The goal is to maintain training stability under extreme sparsity.
• Training paradigm: Pretraining progresses through multi-stage context extension from 8K to 64K to 256K to 1M tokens. Post-training adds Agentic RL at the million-token level, targeting long-horizon agentic reasoning.

The design intent is clear: long context is not achieved through a single positional encoding trick. It results from the coordinated interaction of attention operators, routing mechanisms, and training curricula.

2. Innovation 1: KDA Lower-Bound Decay Parameterization

2.1 The Problem: Why Diagonal Tiles Cannot Use Tensor Cores

Traditional linear attention and Delta Net structures need to introduce a cumulative decay factor Γ when computing within chunks. The problem is that the reciprocal of Γ can easily overflow or underflow in limited precision such as BF16. As a result, the diagonal tiles within a chunk cannot use the standard matrix multiplication path and must fall back to explicit position-pair computation. The outcome is that non-diagonal tiles benefit from Tensor Core throughput while diagonal tiles become a performance bottleneck, complicating operator implementation and creating uneven performance.

2.2 The Mechanism: Converting Unbounded Decay into Bounded Decay

KDA changes the log-decay from the unbounded negative softplus to a scaled sigmoid form with a lower bound. By fixing a decay lower bound such as gmin = -5, the decay factor is constrained within a safe range. The direct effect is that the reciprocal scaling factor within any 16-token tile is locked into a numerically safe interval, eliminating the need for a special computation branch for diagonal tiles.

The core difference can be expressed in pseudocode:

# Conventional approach: unbounded decay, reciprocal may overflow
log_decay = -softplus(score)              # No lower bound
inv_gamma = 1.0 / cumsum(exp(log_decay))  # May overflow/underflow in BF16
# KDA approach: bounded decay, numerically safe
log_decay = gmin * sigmoid(score)         # gmin = -5, bounded range
inv_gamma = 1.0 / cumsum(exp(log_decay))  # Scaling factor constrained to a safe range

2.3 Benefits: Unified Path and Natural Extrapolation

After eliminating the special branch for diagonal tiles, all causal tiles—both diagonal and non-diagonal—can directly call dense Tensor Core GEMM. This significantly reduces implementation complexity in CUDA or CUTLASS and improves throughput. More importantly, this parameterization naturally fits NoPE (no positional encoding) settings, enabling the 1M context to be extrapolated directly without relying on interpolation or other additional techniques. For engineering teams, this is a minimal-change, high-impact reusable technique: it does not alter model semantics, only the numerical parameterization, yet improves both operator efficiency and extrapolation capability.

3. Innovation 2: Quantile Balancing (QB) — Load Balancing Without Auxiliary Loss

3.1 The Problem: Routing Collapse in Very Large Expert Pools

The most typical failure mode in MoE training is routing collapse: a small number of experts become over-activated while most experts remain underutilized. The common industry solution is to introduce an auxiliary load-balancing loss. However, auxiliary loss can conflict with the main task gradient, is sensitive to weight tuning, and may fail to keep up with training dynamics when the expert pool is as large as 896 experts.

3.2 The Mechanism: Dynamic Thresholds Based on Quantiles

The QB mechanism does not rely on auxiliary loss. Instead, it dynamically adjusts expert capacity thresholds based on quantiles of the routing scores, pushing the actual assignment of each expert toward the target load. It can be understood as admission by quantile ranking rather than truncation by fixed capacity. The advantage is that the balancing signal comes from the routing distribution itself and does not contaminate the main loss. At the same time, the thresholds adapt to training dynamics, making the method suitable for very large expert pools.

3.3 Engineering Value

Eliminating auxiliary loss means a cleaner training objective and reduced tuning cost from multi-objective trade-offs. Dynamic balancing ensures that expert utilization does not skew severely under extreme sparsity. For teams building their own MoE models, QB offers an engineering alternative to auxiliary loss, especially when the expert count reaches several hundred.

4. Innovation 3: Block Attention Residuals (Block AttnRes)

A classic problem in deep Transformers is depth representation bottleneck: as layers deepen, the incremental information in the residual stream tends to saturate, and deeper layers contribute diminishing marginal value to the final representation. Block AttnRes addresses this by enabling cross-layer information sharing at the block level, allowing attention modules to selectively reuse representations from earlier blocks rather than relying only on single-path residual accumulation.

The value lies in two aspects. First, it alleviates the depth representation bottleneck so that deeper attention computation has genuinely new information to work with. Second, it reduces memory and communication overhead—cross-layer sharing is organized at block granularity, avoiding the bandwidth pressure of full layer-by-layer transfer. In training scenarios with hundreds of billions of activated parameters and tens of thousands of parallel accelerators, reducing communication overhead can matter more for overall throughput than optimizing compute.

5. Training Paradigm: Multi-Stage Context Extension and Agentic RL

Pretraining uses a multi-stage curriculum of 8K to 64K to 256K to 1M tokens, which is a mainstream best practice for long-context training. The model first converges on basic language abilities with short contexts, then gradually extends the window, avoiding wasted computation and amplified numerical instability from starting with very long sequences. Post-training adds Agentic RL at the million-token level, advancing long context from being able to read long inputs to being able to make multi-step decisions in long-horizon tasks. This step is critical for agent scenarios—failures in long-horizon reasoning often stem not from insufficient single-step capability but from problems in cross-step state maintenance and credit assignment, which require reinforcement learning over long sequences to align.

6. Reusable Techniques and Implementation Priorities

The reusability of the three techniques can be summarized as follows:

Technique	Reusability	Applicable Scenarios	Change Cost
KDA lower-bound decay parameterization	High	Linear attention operator implementation, long-context extrapolation	Minimal
Quantile Balancing (QB)	High	MoE training with very large expert pools	Medium
Block AttnRes	Medium-high	Deep Transformer representation and communication optimization	Medium

For implementation, the priority should be operators first, then routing, then structure. KDA parameterization is a small numerical-level change and can be validated on existing linear attention implementations for throughput gains. QB should be introduced before scaling up the expert count to avoid having to remedy routing collapse after it appears. Block AttnRes is best evaluated as a structural optimization when the model depth continues to increase.

7. Conclusion

The real value of this technical sample is not any single metric. It demonstrates a co-design paradigm for long context and high-sparsity MoE: the attention side uses bounded parameterization to gain operator efficiency and extrapolation capability; the MoE side uses auxiliary-loss-free dynamic balancing for training stability; the structural side uses block-level cross-layer sharing for deeper representation and communication efficiency; and the training side uses multi-stage curricula and long-horizon RL to turn these capabilities into practical performance. For teams working on ultra-long context or large-scale MoE models, all three techniques offer directly transferable engineering value.