Architecture Deep Dive: Trillion-Parameter MoE Multimodal Large Models

    This article explores the architectural innovations behind trillion-parameter Mixture-of-Experts (MoE) multimodal large models. It details how these models achieve massive capacity while maintaining low inference costs through sparse activation. Covering critical components such as incremental attention, Gated Multi-head Latent Attention (Gated MLA), Stable Latent MoE with 896 experts, quantile-based load balancing, and Agentic reinforcement learning, the analysis provides developers with practical insights into deploying and optimizing ultra-large sparse models for long-context and agentic applications.

    I. Background: Why Models Must Go “Larger but Sparser”

    Over the past two years, the capability boundaries of large models have been continuously pushed forward, but the scaling of dense architectures has hit an obvious bottleneck. Doubling parameters leads to linear or even super-linear growth in compute and memory costs, while the gains from training data quality are diminishing. A consensus in the industry is that next-generation foundation models must simultaneously satisfy three conditions:

    1. Native Multimodality: Vision and text are trained end-to-end within the same model, rather than simply attaching a vision encoder as an external plug-in.
    2. Ultra-Long Context: The context window scales from the 100K token level to the 1M token level to support repository-level code understanding, long-document analysis, and multi-turn agentic interactions.
    3. Sparse Activation: Balancing total capacity and per-token compute cost through a Mixture-of-Experts (MoE) architecture.

    A recent typical open-source example involves a multimodal large model with total parameters reaching the 2.8 trillion scale, while the actual activated parameters per token are only about 104 billion—an activation rate of less than 4%. This means the model possesses immense knowledge capacity while keeping inference costs at the level of a 100-billion-parameter dense model. This design philosophy of “decoupling capacity from cost” represents the mainstream evolutionary path for current frontier models.

    II. Attention Side: Three Design Innovations for Million-Token Contexts

    The core contradiction of ultra-long contexts is that the computational complexity of standard self-attention grows quadratically with sequence length, and the memory footprint of the KV cache grows linearly. Both become unacceptable at the million-token scale. The model introduces three combined innovations in its attention module that are worth analyzing layer by layer.

    2.1 Incremental Attention (KDA Approach)

    Traditional attention requires every query position to attend to all historical key-value pairs, resulting in severe computational redundancy. The idea behind incremental attention is to compress historical information into a set of recursively updated “state vectors.” When a new token arrives, it only needs to interact with this state on a small scale, rather than traversing the entire history.

    Its formalized expression can be approximately understood as:

    1. S_t = λ_t · S_{t-1} + k_t · v_t^T (Recursive state update)
    2. o_t = q_t · S_t (Output read)

    Here, λ is a dynamically learned forgetting gate. This structure reduces the computational complexity of sequence processing from quadratic to linear, making it particularly suitable for scenarios like long documents and long conversations where historical information is highly redundant.

    2.2 Gated Multi-head Latent Attention (Gated MLA)

    Multi-head Latent Attention (MLA) compresses the KV cache by projecting it into a low-dimensional latent space, an effective means of alleviating memory pressure in long sequences. Gated MLA builds on this by introducing learnable gating factors for each attention head, allowing the model to adaptively regulate the information inflow intensity of each head:

    • For high-value channels, the gate approaches 1, allowing information to pass through completely.
    • For noisy channels, the gate decays, suppressing interference propagation.

    This is especially critical in ultra-long contexts—noise accumulation in long sequences can significantly dilute effective attention. The gating mechanism provides a “soft information filtering” approach with almost no additional inference overhead.

    2.3 Cross-Layer Attention Residual (AttnRes)

    Deep networks suffer from the well-known “representation drift” problem: fine-grained features extracted in shallow layers are easily overwritten and lost during layer-by-layer propagation by high-level semantics. AttnRes establishes cross-layer residual connections along the attention path, allowing critical shallow information (such as variable definitions in code or entity references in long documents) to bypass directly to deeper layers. This serves to:

    • Mitigate gradient vanishing in deep layers and stabilize training;
    • Enhance the model’s “precise localization” capability for long-range dependencies, yielding obvious benefits in “needle-in-a-haystack” long-context evaluations.

    These three designs are not isolated: incremental attention solves “computing fast,” Gated MLA solves “storing economically,” and AttnRes solves “transmitting accurately,” collectively forming a complete solution for million-token contexts.

    III. MoE Side: Stability Engineering at an 896-Expert Scale

    The difficulty of sparsification has never been “how to draw the routing structure,” but rather the stability and load balancing during the training process. When the number of experts scales to 896, traditional auxiliary loss balancing methods face two failure points: first, gradient conflicts between the auxiliary loss and the main loss are amplified at scale, degrading model quality; second, under extreme sparsity (activating only a very few experts per token), individual experts can easily collapse into “dead experts,” causing routing to fall into local optima.

    3.1 Latent Expert Space

    The model adopts the Stable Latent MoE architecture. The core idea is to introduce a low-dimensional latent space for expert routing: tokens are first projected into a latent representation, and expert matching is then completed in this latent space. The benefits of this approach are:

    • Routing decisions are based on smoother continuous representations, reducing the oscillation of discrete routing;
    • The latent space acts as a “shared base” among experts, mitigating representational fragmentation caused by experts operating independently.

    3.2 SiTU-GLU Activation Function

    The choice of activation function directly affects the gradient health of sparse FFNs. SiTU-GLU is an improved variant of the Gated Linear Unit (GLU) family. While maintaining the expressiveness of the GLU family, it suppresses extreme activation values through piecewise smoothing, reducing the risk of variance explosion in expert outputs—an indispensable detail when training trillion-parameter models in bf16 precision.

    3.3 Quantile Balance (QB) Mechanism

    Traditional balancing relies on loss penalties, whereas the QB mechanism adopts a statistical control approach: it monitors the load distribution of all experts in real-time during training. When an expert’s load exceeds a preset quantile (e.g., P90), it dynamically adjusts its routing score. This can be analogous to a “soft rate limiting”:

    1. for expert in experts:
    2. if load(expert) > quantile(all_loads, p):
    3. routing_score[expert] *= decay_factor

    Compared to loss penalties, this method introduces no additional gradient interference, achieving “balance without hurting quality.”

    IV. Training Strategy: Multi-Stage Length Curriculum and Agentic RL

    4.1 The 4-8-32-…-1M Length Curriculum

    Pre-training directly on million-token lengths is both inefficient and hard to converge. The model employs a progressive sequence length curriculum: it first establishes foundational language and multimodal capabilities on short sequences, then progressively scales to 8K, 64K, 256K, and finally 1M. At each stage, hyperparameters like the RoPE base frequency and attention scaling coefficients are adjusted synchronously to ensure the smoothness of positional encoding extrapolation. This methodology is currently an industry best practice for long-context training and can be reused for any in-house model’s length scaling.

    4.2 Agentic Post-Training at the Million-Token Scale

    Pre-training endows the model with “knowledge,” but agentic capabilities require “practice.” In the post-training phase, the model introduces large-scale agentic reinforcement learning (RL): it interacts with real feedback in environments involving multi-turn tool usage and long-horizon task planning, with task trajectories themselves reaching the million-token scale. This solves two limitations of traditional SFT:

    • Imitation learning can only reproduce demonstration trajectories, whereas RL can explore better strategies;
    • Credit assignment in long-horizon tasks requires propagating rewards across dozens of decision turns, which short-context RL cannot cover.

    For developers, this means the model’s robustness in Agent scenarios—such as multi-step reasoning, tool orchestration, and error recovery—is significantly superior to similar models that have only undergone alignment fine-tuning.

    V. Practical Implications for Developers

    Based on the architectural analysis above, when selecting and deploying such sparse multimodal models, organizations should focus on the following points:

    1. Inference deployment must match sparse characteristics: MoE models require expert parallelism and capacity factor tuning; memory planning should be based on total parameters rather than activated parameters.
    2. Long context should be paired with retrieval strategies: Even if the model natively supports 1M context, Retrieval-Augmented Generation (RAG) should still be combined for ultra-large-scale corpora to avoid “attention dilution.”
    3. Budget multimodal input preprocessing: The sequence proportion of visual tokens directly impacts the available context quota; token budget management must be handled at the application layer.
    4. Prioritize long-horizon tasks in Agent scenarios: When evaluating models, construct multi-turn, cross-dependency real-world business tasks rather than just looking at single-turn QA benchmarks.

    VI. Conclusion

    From this architectural design, it is evident that the competitiveness of frontier large models has shifted from simple “parameter stacking” to systemic engineering. On the attention side, incremental computation and latent compression tame quadratic complexity; on the MoE side, latent routing and statistical balancing ensure training stability under extreme sparsity; on the training side, length curriculum and agentic RL bridge the last mile from pre-training to real-world tasks. For developers focused on the open-source ecosystem, these public technical choices serve as a referenceable engineering blueprint—understanding the underlying “problem-solution” mapping is of greater long-term value than memorizing the name of any single model.