Core Architecture and Key Technologies in Trillion-Parameter Multimodal MoE Models

1. Introduction: Technical Challenges in Trillion-Parameter Multimodal MoE Models

As Large Language Models (LLMs) continue to break the trillion-parameter barrier, the Mixture-of-Experts (MoE) architecture has emerged as the mainstream technical route, offering the distinct advantage of “large capacity with low inference cost.” However, when parameter scales push further into the trillions while requiring native multimodal comprehension and ultra-long contexts of up to one million tokens, three core challenges emerge:

1. Computational Efficiency in Ultra-Long Contexts: Traditional attention mechanisms face dual bottlenecks in memory and compute power at the million-token level. While linear attention offers potential, it suffers from numerical stability issues under limited precision.
2. Training Stability in Massive MoE Layers: Hundreds or even thousands of routed experts can easily lead to load imbalance and training oscillations under extreme sparsity.
3. Long-Horizon Agentic Reasoning: Agent scenarios require models to perform multi-step reasoning and tool-calling planning, necessitating targeted reinforcement during the post-training phase.

This article systematically examines the core architectural innovations and key engineering technologies detailed in a recent technical report of a trillion-parameter multimodal MoE model, providing a reusable technical paradigm for developers and researchers.

2. Overall Architecture Overview

The model adopts a sparse MoE architecture with approximately 2.8 trillion total parameters and around 104 billion activated parameters per inference. The core architecture consists of the following modules:

Module	Key Design
Attention Layer	Fuses Kimi Decay Attention (KDA) and Gated Multi-head Latent Attention (Gated MLA), introducing Attention Residuals (AttnRes) for cross-layer information sharing.
Expert Layer	Stable LatentMoE architecture equipped with 896 routed experts, paired with the SiTU-GLU activation function and Quantile Balancing (QB) mechanism.
Training Strategy	8K → 64K → 256K → 1M multi-stage context extension pre-training, overlaid with million-token-level Agentic RL post-training.
Multimodal	Native visual encoder integration, supporting joint image-text understanding and generation.

The core philosophy of this layered design is to enhance computational efficiency on the attention side through numerical stabilization, ensure training stability on the expert side through load balancing mechanisms, and endow long-horizon reasoning capabilities through progressive extension and agentic reinforcement learning in the training pipeline.

3. Deep Dive into Core Innovations

3.1 KDA Bounded Decay Parameterization: Maximizing Tensor Core Utilization for Linear Attention

3.1.1 The Pain Point

The core idea of Linear Attention and Delta network architectures is to reduce the computational complexity of attention from O(n²) to O(n) by maintaining a state matrix via recurrence. However, in actual GPU implementations, a long-overlooked engineering bottleneck is that the cumulative reciprocal of the decay factor suffers from severe overflow or underflow under BF16 limited precision.

Specifically, assuming a decay factor of Γ within a chunk, its reciprocal, 1/Γ, may exceed the representable range of BF16 after continuous accumulation. This prevents the direct use of the Tensor Core matrix multiplication acceleration path when computing Diagonal Tiles, forcing a fallback to an explicit position-pair computation branch. This branch is computationally inefficient and breaks kernel uniformity, increasing the maintenance complexity of CUDA kernels.

3.1.2 Solution: Bounded Scaled Decay

The KDA (Kimi Decay Attention) bounded decay parameterization proposes a core modification: replacing the unbounded Log-decay function (originally Negative-Softplus) with a bounded Scaled Sigmoid:

# Pseudo-code: Decay factor calculation
g_t = g_min * Sigmoid(e^A * h_z_t)   # g_min is fixed at -5
# Traditional approach: g_t = -Softplus(...)  # Unbounded, may tend to -∞

By fixing g_min to -5, the reciprocal scaling factor within any 16-token tile is strictly locked within a safe numerical range. This implies:

• All Causal Tiles (including diagonal and non-diagonal blocks) can directly invoke Dense Tensor Core GEMM without requiring special branches.
• Native support for NoPE (No Positional Encoding) ultra-long context extrapolation, as the decay parameters do not rely on absolute positions.
• Minimal engineering changes, requiring only a modification to the decay parameter calculation formula, with virtually no need to rewrite underlying CUDA kernels.

3.1.3 Engineering Benefits

From an engineering perspective, eliminating the special branch for diagonal blocks reduces kernel launching overhead and drives Tensor Core utilization toward 100%, yielding significant throughput improvements across context lengths ranging from 8K to 1M. For inference services deployed on GPU clusters, this translates to servicing more concurrent requests under the same hardware budget.

3.2 Quantile Balancing (QB): Auxiliary-Loss-Free Load Balancing for Massive MoE

3.2.1 The Dilemma of Traditional MoE Load Balancing

In MoE architectures containing hundreds of experts, the routing mechanism is prone to “expert collapse”—a scenario where a few popular experts are over-selected while the rest remain idle. Traditional solutions typically introduce an Auxiliary Loss to penalize imbalanced distribution, but this introduces two issues:

1. Gradient conflicts between the auxiliary loss and the main task objectives can degrade the model’s primary performance.
2. When the number of experts is massive (e.g., 896), hyperparameter tuning for the auxiliary loss becomes exceptionally difficult.

3.2.2 Core Concept of the QB Mechanism

The core of the Quantile Balancing (QB) strategy is to utilize quantile statistics of expert selection frequencies to dynamically adjust routing directly, without introducing additional loss terms.

Its operational logic can be summarized as follows:

1. Maintain a sliding window of token selection counts for each expert.
2. Periodically calculate quantile thresholds (e.g., P50, P90) for all expert counts.
3. Apply routing suppression to hot experts exceeding high quantiles, and routing incentives to cold experts below low quantiles.
4. The entire process is completed during the forward pass; gradients propagate naturally without any extra loss terms.

This design is critical in extreme sparsity scenarios with 896 experts—when each token activates only a very small number of experts, any static balancing strategy may fail. Dynamic, quantile-based adjustment adapts to changes in expert popularity during training.

3.2.3 Synergy with SiTU-GLU

The QB mechanism also creates a synergy with the SiTU-GLU activation function. SiTU (Sigmoid-weighted Tanh Unit) uses sigmoid gating to weight the two paths of GLU. Compared to standard GLU, it offers smoother gradient characteristics, helping to reduce gradient spikes in MoE scenarios where expert switching is frequent, thereby further stabilizing the training process.

3.3 Block Attention Residuals (Block AttnRes): Efficient Cross-Layer Information Sharing

3.3.1 Representation Bottlenecks in Deep Transformers

As model depth increases, deep Transformers often face “representation degradation”—fine-grained information from shallow layers is gradually lost during layer-by-layer propagation, while abstract representations from deep layers struggle to directly guide shallow optimizations. Traditional residual connections only pass information between adjacent layers, and direct cross-layer communication requires additional attention or gating overhead.

3.3.2 Block AttnRes Design

Block AttnRes upgrades attention residuals from “layer-by-layer scalar addition” to “block-level attention information sharing”:

• The model is divided into blocks at a certain granularity (e.g., every N layers as a Block).
• Compressed attention states are passed across layers within a block, rather than the full hidden state.
• Lightweight projections achieve information dimensionality reduction, controlling memory and communication overhead.

The advantages of this design include:

1. Breaking the Depth Bottleneck: Shallow layers can directly access high-level semantic signals from deep layers, improving overall representation capability.
2. Reducing Communication Costs: Compared to full cross-layer residuals, attention states have lower dimensions, significantly reducing AllReduce overhead in distributed training.
3. Native Compatibility with KDA: Since KDA maintains linear recurrent states itself, the cost of sharing these states across layers is minimal.

3.4 Multi-Stage Context Extension and Agentic RL Post-Training

3.4.1 Progressive Context Extension

The model adopts a four-stage pre-training strategy of 8K → 64K → 256K → 1M, rather than jumping directly from short contexts to the million-token level. The rationale for this progressive approach includes:

• Curriculum-based Training: Allowing the model to first learn basic language modeling in short contexts before gradually adapting to longer sequences.
• Memory Controllability: Different sequence parallelism strategies can be used at each stage, avoiding excessive memory consumption all at once.
• Numerical Validation: Verifying the effectiveness of mechanisms like KDA bounded decay in shorter stages before scaling to longer contexts.

3.4.2 Million-Token-Level Agentic RL

The post-training phase introduces reinforcement learning based on Agent scenarios, pushing training sequence lengths to the million-token level. This design aims to equip the model with:

• Long-horizon task planning capabilities (maintaining goal consistency over hundreds of thousands of tokens of context).
• Multi-step tool calling and result integration capabilities.
• Self-correction and reflection capabilities within ultra-long sequences.

Agentic RL typically combines a Process Reward Model with outcome rewards, stabilizing training on long sequences through algorithms like Grouped Relative Policy Optimization (GRPO).

4. Engineering Best Practices and Implementation Recommendations

For developers and enterprise teams looking to leverage these technical paradigms, the following points are noteworthy:

1. Numerical Stability in Linear Attention is the Primary Engineering Issue: When implementing any linear attention variant, always verify the numerical range of decay factors under BF16/FP16 precision to avoid performance degradation caused by implicit precision overflow.

2. Prioritize Auxiliary-Loss-Free Solutions for MoE Load Balancing: Especially when the number of experts exceeds 512, the tuning cost of auxiliary losses rises sharply. Dynamic strategies based on quantiles are far more feasible for engineering.

3. Balance Communication and Representation Gains in Cross-Layer Sharing: In distributed training environments, block-level attention state sharing is much more cost-effective than sharing full hidden states.

4. Ultra-Long Context Training Requires Supporting Infrastructure: This includes the synergy of technologies like sequence parallelism, gradient checkpointing, and memory optimization. Cloud platforms offering containerized GPU clusters and high-performance networks, such as Baidu AI Cloud, can provide the necessary infrastructure support for this type of training.

5. Agentic RL Post-Training Requires High-Quality Long-Sequence Evaluation Benchmarks: Relying solely on short-sequence benchmarks cannot reflect the model’s true long-horizon reasoning capabilities. It is recommended to build a comprehensive evaluation system that includes multi-step planning, tool usage, and error recovery.

5. Conclusion and Future Outlook

The technical competition in trillion-parameter multimodal MoE models has shifted from “can we scale up” to “can we make it stable, fast, and smart.” The three core innovations analyzed in this article—KDA bounded decay parameterization, QB load balancing, and Block AttnRes cross-layer sharing—address this shift from the dimensions of computational efficiency, training stability, and model representation, respectively.

For developers, these technologies do not exist in isolation; they form a composable engineering paradigm: linear attention solves efficiency, MoE balancing solves scale, cross-layer residuals solve depth, and progressive training with Agentic RL solves capability. Rationally combining these paradigms and adapting them to specific business scenarios is the critical path for the current engineering implementation of large models. As hardware compute power continues to increase and training algorithms evolve, similar architectural innovations will remain the core driving force pushing the boundaries of large model capabilities.