Architecture and Training Insights for Trillion-Parameter Sparse MoE Multimodal Models
A deep dive into the architecture design and training practices behind trillion-parameter sparse MoE multimodal models, covering hybrid attention mechanisms, numerically stable linear attention, auxiliary-loss-free load balancing across hundreds of experts, and multi-stage long-context pretraining with agentic RL.
1. Background: Why Ultra-Large Sparse Models Became the Mainstream Path
Recent progress in large models has followed two clear trajectories: first, context length has grown from a few thousand tokens toward the million-token scale; second, parameter scale has continued expanding along a sparse route characterized by “extremely large total parameters, extremely few activated parameters.” Based on publicly released technical reports, a representative configuration looks like this: total parameters reach the trillion level, each token activates only about one thousandth of the parameters, and the model natively supports visual multimodal input with a million-token context window. This design strikes a strong balance between inference cost and model capability, and it is worth analyzing in depth.
The core advantage of the sparse route is that, with enough total capacity, the model can “memorize” more world knowledge, while the limited per-token activation keeps inference compute in the same range as small-to-medium dense models. However, this route introduces three major engineering challenges:
- Attention computation efficiency under ultra-long context;
- Training stability of large-scale MoE with hundreds or even thousands of experts;
- Building long-horizon, multi-step reasoning capabilities for Agent scenarios.
This article examines the corresponding architectural innovations and training strategies for each of these three challenges.
2. The Attention Side: Hybrid Attention and Cross-Layer Residuals
2.1 Gated Multi-Head Latent Attention (Gated MLA)
One of the main bottlenecks in ultra-long-context reasoning is the memory footprint of the KV cache. Multi-Head Latent Attention compresses the cache significantly by projecting Keys and Values into a low-dimensional latent space, making million-token context feasible within limited memory. Adding a gating mechanism on top of this allows dynamic modulation of information flow across different latent channels, mitigating the information loss caused by latent compression. A simplified sketch:
# Gated MLA pseudocodelatent_kv = down_proj(kv) # Compress into low-dimensional latent spacekv = up_proj(latent_kv) # Restore on demandkv = kv * sigmoid(gate) # Gate the information flow
2.2 KDA and Lower-Bound Decay Parameterization
Linear attention operators (such as Delta-style update rules) replace full quadratic attention computation with incremental updates of a state matrix, theoretically reducing long-sequence inference complexity to linear. However, their implementation contains a well-known numerical stability trap: in chunked parallel computation, the reciprocal of the accumulated decay factors within a chunk easily overflows or underflows in low-precision formats like BF16. This forces implementers to route diagonal blocks through a special per-position computation branch, losing the dense matrix-multiply acceleration of Tensor Cores.
An effective fix is to parameterize the decay factor with a lower bound: replace the originally unbounded negative activation (such as negative Softplus) with a scaled Sigmoid form:
decay = g_min * Sigmoid(θ · z) # g_min is a fixed negative lower bound, e.g., -5
This change carries substantial engineering value:
- With any fixed chunk length (e.g., a 16-token tile), the reciprocal scaling factors are locked within a safe numerical range, eliminating overflow risk in low precision;
- Diagonal blocks no longer require a special branch—all causal blocks (diagonal and off-diagonal) can uniformly use the dense GEMM path, significantly improving hardware throughput;
- Since the decay itself handles distance-based forgetting, the model can naturally support million-token context extrapolation without positional encoding (NoPE), simplifying positional encoding design.
Mixing this type of linear attention operator with Gated MLA across layers is a common architectural choice in current long-context models: shallow layers and local dependencies use linear attention for efficiency, while key layers use latent attention to preserve precise retrieval capability.
2.3 Block-Level Attention Residual (Block AttnRes)
As model depth grows to hundreds of layers, deep Transformers commonly face a representation bottleneck: after accumulating through many residual layers, layer representations gradually converge and gradient propagation becomes less efficient. Block-level attention residual establishes an additional cross-layer information-sharing path between adjacent attention blocks, with benefits in two areas:
- Mitigating deep representation degradation: fine-grained features captured by lower layers can bypass several layers and reach deeper layers directly, preserving representation diversity;
- Reducing memory and communication overhead: under hybrid pipeline and tensor parallel deployment, the residual path design can reduce stored activations and cross-device communication volume.
3. The MoE Side: Stability Design at 896-Expert Scale
3.1 Quantile-Based Balancing (QB): Auxiliary-Loss-Free Load Balancing
Traditional MoE training relies on an auxiliary balancing loss (Auxiliary Loss) to prevent routing collapse onto a few experts. However, the auxiliary loss conflicts with the main loss in gradient direction: setting it too strong hurts language modeling quality, while setting it too weak makes balancing ineffective—a frustrating hyperparameter in ultra-large-scale MoE training.
The quantile-based balancing mechanism offers an alternative that requires no auxiliary loss: by tracking quantiles of each expert’s load, it dynamically adjusts routing bias or expert capacity thresholds, keeping the load distribution within a reasonable range throughout training. Its advantages:
- No extra loss term is introduced, so gradients of the main task are undisturbed;
- Balancing strength adapts automatically to the actual load distribution, with no manual weight tuning;
- Training remains stable even with hundreds of experts and extreme sparsity (e.g., only 8 experts activated per token).
3.2 Activation Functions and Expert Routing
Inside the expert networks, adopting SiTU-style gated activation functions in place of traditional SwiGLU yields smoother activation distributions at the same parameter budget. Combined with the stabilized routing mechanism, this further reduces loss oscillation in later training stages. Each such detail contributes modestly on its own, but together they determine the engineering feasibility of “training for weeks on a thousand-GPU cluster without crashing.”
4. Training Recipe: Multi-Stage Context Extension and Agentic Post-Training
4.1 Staged Pretraining for Context Length
Pretraining directly on million-token context is both inefficient and unstable; the industry-standard approach is multi-stage progression:
- 8K stage: the bulk of knowledge injection, accounting for the majority of training tokens;
- 64K / 256K stages: gradually raise the RoPE base frequency or adjust decay parameters, completing length adaptation with a small amount of data;
- 1M stage: final extension using long documents and repository-level code data, while validating length generalization for retrieval and reasoning.
Combined with the natural length extrapolation of linear attention and the cache compression of MLA, the training cost of the 1M-context stage remains manageable.
4.2 Million-Token Agentic RL
For long-horizon agent reasoning, the post-training stage introduces reinforcement learning rewarded by multi-step task success rates. The difficulty lies in that a single rollout may span dozens of tool calls and reasoning steps, with trajectories reaching the million-token scale. Engineering challenges include credit assignment over long trajectories, densification of partial rewards, and efficient parallel sampling of ultra-long sequences. This kind of training markedly improves performance on multi-step planning, long-document analysis, and code engineering tasks, and it has become a key differentiator for top open-source models on benchmarks.
5. Reusable Lessons
For teams working on large model development, the following three takeaways are the most transferable:
- Lower-bound parameterization of linear decay: a minimal change—simply replacing the activation form—allows linear attention operators to fully leverage Tensor Cores, offering excellent engineering ROI;
- Auxiliary-loss-free dynamic load balancing: worth prioritizing in ultra-large-scale MoE training, as it balances load distribution without sacrificing main-loss quality;
- Block-level cross-layer residuals: a low-cost way to mitigate representation bottlenecks in deep networks while reducing memory and communication pressure in distributed training.
Overall, the technical reports of this generation of ultra-large sparse multimodal models lay out a clear roadmap: hybrid attention solves long-context efficiency, parameterization tricks and loss-free balancing solve ultra-large MoE stability, and agentic reinforcement learning completes agent capabilities. These methods are all general-purpose and independent of specific hardware or ecosystems, offering direct reference value for teams building multimodal, long-text, agent-based applications for everyday users—such as document understanding, content writing, and data analysis scenarios.
