Technical Architecture of Trillion-Scale Sparse Multimodal Models: From Attention Fusion to Stable Routing

As large model parameter scales expand from hundreds of billions to trillions, relying solely on dense architectures makes it difficult to balance training costs with inference efficiency. Sparse Mixture-of-Experts (MoE) architectures, combined with native multimodal capabilities and ultra-long context lengths, are becoming the mainstream technical route for next-generation foundation models. This article analyzes a MoE multimodal model with 2.8 trillion total parameters and approximately 104 billion activated parameters, systematically breaking down its attention module design, cross-layer information sharing mechanisms, expert routing architecture, and training stability safeguards to help developers understand and implement similar architectures.

1. Overall Architecture: Balancing Capabilities Under Extreme Sparsity

The core design goals of this model can be summarized in three points: massive capacity, extremely low activation ratio, and native multimodality.

• Massive Capacity: With 2.8 trillion total parameters, the model possesses strong knowledge storage and representation capabilities, enabling it to handle complex tasks across multiple languages, domains, and modalities.
• Extremely Low Activation Ratio: Activating approximately 104 billion parameters per forward pass yields an activation ratio of about 3.7%. This ratio ensures that the computation required per token is significantly lower than that of a dense model with equivalent total parameters, which is critical for controlling inference costs.
• Native Multimodality: Visual capabilities are not implemented by attaching external encoders but are jointly modeled with text during the pre-training phase, avoiding representational fragmentation between modalities.
• Million-Token Context: Supporting an ultra-long context of 1 million tokens places higher demands on the memory complexity and information retrieval precision of the attention mechanism.

These three capabilities constrain one another: longer contexts increase attention overhead; higher sparsity makes routing harder to stabilize; and more modalities complicate representation alignment. Therefore, the architectural focus is not on a single breakthrough but on the synergy between modules.

2. Attention Module: A Fusion of Three Mechanisms

In ultra-long context scenarios, standard full attention faces an O(n²) complexity bottleneck. This model employs a fusion approach in the attention layers, combining two types of attention and one cross-layer mechanism.

2.1 Kimi Delta Attention (KDA)

The core idea behind Kimi Delta Attention (KDA) is to preserve the global modeling capability of attention while reducing redundant computations over long sequences through incremental state updates. Intuitively, it maintains a state representation that updates incrementally as the sequence progresses, allowing historical information to participate in subsequent computations in a compressed form rather than fully reviewing all historical tokens each time.

Its engineering value is twofold:

1. Complexity Optimization: Computational and memory overheads on long sequences are significantly lower than with full attention.
2. Information Fidelity: Compared to fixed windows or crude pooling, incremental states retain finer-grained historical dependencies.

2.2 Gated Multi-head Latent Attention (Gated MLA)

Multi-head Latent Attention (MLA) uses low-rank compression to project the key-value cache into a latent space, drastically reducing the memory footprint of the KV Cache—an almost mandatory capability for million-token contexts. A gating mechanism (Gated) is introduced on top of this, applying learnable gating weights to the attention output. This allows the model to dynamically adjust the informational contribution strength of different heads and positions.

The benefits of gating include:

• Suppressing interference from irrelevant historical context in long sequences;
• Improving the signal-to-noise ratio of effective information under sparse activation conditions;
• Providing more flexible representation selection capabilities for different tasks.

2.3 Attention Residual (AttnRes): Cross-Layer Information Sharing

In deep networks, the outputs of attention layers often exhibit inter-layer redundancy. The Attention Residual (AttnRes) mechanism allows attention modules to reuse or aggregate attention representations from preceding layers across layers, rather than relying solely on the current layer’s input.

The significance of this design includes:

• Reducing Redundant Computation: Similar attention patterns do not need to be relearned at every layer.
• Enhancing Gradient Flow: Cross-layer connections mitigate gradient decay during deep network training.
• Improving Representation Consistency: Key information in long contexts can be continuously transmitted across multiple layers.

The combination of these three is not a simple stack but a clear division of labor: KDA handles long-sequence efficiency, Gated MLA handles KV compression and dynamic filtering, and AttnRes handles inter-layer synergy.

3. MoE Side: A Stable Sparse Architecture with 896 Routed Experts

3.1 Stable LatentMoE Architecture

The model’s MoE layer is equipped with 896 routed experts, representing a typical fine-grained expert partition. A higher number of experts increases theoretical capacity, but the routing decision space also grows exponentially, raising the risk of training instability.

The design philosophy of Stable LatentMoE is to perform expert routing and computation in the latent space: token representations are first projected into a low-dimensional latent space, expert selection and transformation are completed there, and the results are then mapped back to the original space. The benefits of this approach include:

• Reducing the parameter and computational overhead of each expert, making a 896-expert scale engineering-feasible;
• Smoothing the routing distribution in the latent space, reducing extreme routing biases;
• Forming a consistent architectural philosophy with the latent compression approach used in attention.

3.2 SiTU-GLU Activation Function

The internals of the experts utilize SiTU-GLU activation. The GLU family of gated structures has been widely validated in Transformers. The SiTU variant maintains gating characteristics while improving gradient behavior in the negative interval, which is beneficial for numerical stability in deep sparse networks. For a sparse model with only hundreds of billions of activated parameters, the smoothness of the activation function directly impacts the quality of routing gradients.

3.3 Quantile Balance (QB) Mechanism

The most typical problem under extreme sparsity is routing collapse: a few experts are over-selected while most remain under-trained, leading to capacity waste and training divergence. The Quantile Balance (QB) mechanism dynamically calibrates the load of each expert using quantile statistics, making token allocation closer to a balanced distribution.

Compared to hard capacity limits, QB offers the following advantages:

• It is based on statistical quantiles rather than fixed thresholds, adapting to distribution changes across different batches;
• It alleviates the expert cold-start problem, ensuring gradient supply for long-tail experts;
• Combined with latent space routing, it reduces the interference of balancing strategies on model expressiveness.

4. Training Stability: The Core Challenge of Sparse Architectures

The training difficulties of trillion-scale sparse models are concentrated in three areas, which, along with the corresponding designs mentioned above, can be summarized in the following table:

Challenge	Manifestation	Corresponding Mechanism
Routing Collapse	Polarization of expert load	QB Quantile Balance
Long Sequence Overhead	Attention memory and compute explosion	KDA + Gated MLA
Deep Gradient Decay	Inefficient inter-layer information transfer	AttnRes Cross-Layer Sharing
Sparse Gradient Noise	Excessive variance in expert updates	Latent Space Routing + SiTU-GLU

A commonly overlooked point is: the higher the sparsity, the lower the proportion of parameters reached in a single update, making the model more sensitive to optimizer states and gradient accumulation strategies. Therefore, in engineering practice, it is usually necessary to combine larger global batches, gradient clipping, and phased warm-ups. These are general best practices rather than unique designs of a specific architecture.

5. Developer Takeaways for Implementation

For teams looking to leverage this architecture, the following points are worth noting:

1. Trade-offs in Attention Fusion Should Be Scenario-Based. The combination of KDA and Gated MLA is suitable for long-context, high-concurrency inference scenarios; if the context length is limited, standard MLA with gating is sufficient for most needs.
2. More Experts Are Not Always Better. The feasibility of 896 experts relies on latent space compression; on mid-to-small-scale clusters, the number of experts should match the VRAM of a single GPU and communication bandwidth, otherwise routing communication overhead will devour the sparse gains.
3. Stability Mechanisms Must Precede Scale Expansion. Routing balancing and activation function selection should be fully validated at mid-to-small scales before being promoted to trillion-scale training, avoiding repeated debugging on massive tasks.
4. Multimodality Should Be Native, Not Stitched. Modeling visual tokens and text tokens in a unified sequence is the only way to ensure that long-context capabilities truly cover mixed image-text inputs.

Conclusion

The technical essence of trillion-scale sparse multimodal models lies in finding the engineering optimum among capacity, efficiency, and stability. KDA and Gated MLA solve the attention costs of long sequences; AttnRes bridges inter-layer information flow; and Stable LatentMoE combined with the QB mechanism ensures routing health under extreme sparsity. Together, these mechanisms form a reusable architectural paradigm. For developers, understanding the design motivation is more valuable than memorizing parameter numbers: behind every mechanism lies a specific computational or training bottleneck.