Trillion-Parameter MoE Multimodal LLMs: Architecture Design and Training Practices

    This article examines the architecture design and training practices for trillion-parameter multimodal MoE models. It identifies three scaling bottlenecks—compute efficiency, training stability, and long-range reasoning—and then analyzes ultra-sparse MoE design, lower-bounded decay linear attention, gated multi-head latent attention, auxiliary-loss-free quantile balancing, cross-layer attention residuals, staged context extension, and agent reinforcement learning. It concludes with reusable engineering lessons for teams building large sparse models, covering numerical stability, hard routing constraints, phased long-context training, and observability.

    1. Background: Three Bottlenecks in Scaling Ultra-Large Models

    In recent years, large-model evolution has followed three clear directions: continuously growing parameter counts, native multimodality as a standard capability, and context lengths extending toward the million-token level. However, when total parameters reach the trillion level, active parameters are kept at the tens-of-billions level, and the context window expands to 1M tokens, engineering and algorithmic challenges hit three bottlenecks at the same time:

    1. Compute efficiency bottleneck: The compute and memory overhead of conventional full attention grows quadratically with sequence length, making it unsustainable at million-token contexts. Alternatives such as linear attention can reduce complexity to linear, but their implementation efficiency on modern accelerators has long been unsatisfactory.
    2. Training stability bottleneck: As the number of experts expands to hundreds or even nearly a thousand, routing load imbalance and numerical overflow in activation functions are significantly amplified, making training prone to collapse.
    3. Long-range reasoning bottleneck: In agent scenarios such as multi-turn tool calling and cross-document retrieval, the model must preserve information across very long dependency chains and avoid reasoning drift.

    Recent industry technical reports offer a relatively complete set of solutions to these three bottlenecks. This article breaks down their mechanisms and reusable value one by one.

    2. Mixture-of-Experts Architecture: Capacity Expansion under Extreme Sparsity

    The Mixture-of-Experts (MoE) architecture is a mainstream paradigm for expanding model capacity without proportionally increasing inference cost. Typical designs in this direction show two characteristics:

    • Extremely low activation ratio: Total parameters reach the trillion level, while each token activates only about tens of billions of parameters. The sparsity ratio is kept extremely low, bringing inference cost close to that of dense models with tens of billions of parameters while far exceeding their knowledge capacity.
    • A large number of routed experts: The number of routed experts is expanded to hundreds or more. Combined with shared experts and fine-grained expert partitioning, different tokens can be matched precisely to knowledge modules in specialized domains.

    The direct cost of extreme sparsity is training difficulty: the larger the routing space, the more easily it collapses onto a small number of hot experts. To address this, such architectures usually make changes in two places: first, replacing conventional activation functions with gated GLU variants to maintain stable gradient flow under sparse activation; second, introducing a new load-balancing mechanism (see Section 4).

    3. Attention Mechanisms: Combining Linear Attention and Latent Compression

    (1) Lower-Bounded Decay Parameterization: Making Linear Attention Fully Utilize Tensor Cores

    The core idea of linear attention is to replace explicit positional decay with learnable decay factors, thereby compressing historical states into a fixed-size matrix state and achieving O(n) sequence complexity. Traditional implementations, however, have a long-standing engineering pain point: when computing within a chunk, the inverse of the cumulative decay factor may overflow or underflow under limited precision such as BF16. As a result, the diagonal block—the causal part within a tile—must use an explicit position-by-position computation branch and cannot call dense matrix multiplication interfaces, severely reducing operator throughput.

    An effective solution is to parameterize the log-decay with a lower bound. Traditional approaches map log-decay to an unbounded interval, for example through a negative softplus. The modified scheme uses a scaled sigmoid form, as illustrated below:

    1. # Lower-bounded decay parameterization example
    2. g_min = -5.0 # decay lower bound
    3. g_t = g_min * torch.sigmoid(w * z_t + b) # g_t ∈ [-5, 0]

    Because the decay factor is locked within a safe numerical range, the inverse scaling factor within any tile will not enter a dangerous region. This brings two direct benefits:

    1. Elimination of the special diagonal-block branch: All causal tiles, diagonal and non-diagonal, can uniformly use the dense Tensor Core matrix multiplication path, greatly improving operator throughput.
    2. Long-range extrapolation without positional encoding: Because the decay mechanism naturally encodes relative position information, the model can extrapolate context directly to the million-token level without introducing explicit positional encoding.

    This change is minimally invasive in engineering. It only modifies how decay factors are parameterized, yet it fundamentally changes the operator implementation path. It is a typical “low-change, high-return” optimization and is well suited to teams that need to deploy linear attention in their own inference engines.

    (2) Gated Multi-Head Latent Attention: A Second Path for KV Compression

    In addition to linear attention, the architecture also incorporates gated multi-head latent attention. The idea is to compress key-value projections into a low-dimensional latent space, significantly reducing KV Cache memory usage in long contexts, and then use a gating mechanism to control the contribution of each head. The two attention mechanisms are complementary: linear attention handles low-cost state maintenance over very long ranges, while latent attention handles high-precision fine-grained modeling over short and medium ranges. They are interleaved across layers to form a hybrid memory system of “long-range memory + short-range precision.”

    4. Load Balancing: Auxiliary-Loss-Free Quantile Balancing

    Traditional MoE load balancing relies on an auxiliary loss, i.e., an additional regularization term that encourages uniform load. However, when the number of experts reaches nearly a thousand, auxiliary loss has two inherent drawbacks. First, it is difficult to balance gradient competition with the main loss: if the weight is slightly too large, model quality suffers; if it is slightly too small, balancing fails. Second, its balancing effect is “soft,” and hot experts can still appear in extremely sparse scenarios.

    The quantile-based balancing mechanism provides a different path: instead of “persuading” routing through a loss function, it introduces a dynamic adjustment based on quantiles into routing scores, directly constraining the probability of each expert being selected to a target quantile range. As a result, the auxiliary loss term is eliminated, and balancing changes from a “soft constraint” to a “hard guarantee.” Even under conditions of nearly a thousand experts and an extremely low activation ratio, it can maintain stable load throughout training. For teams planning to scale their own MoE models, this “balancing without auxiliary loss” idea is worth attention.

    5. Cross-Layer Attention Residuals: Breaking Representation Bottlenecks in Deep Networks

    When Transformers are stacked to very deep layers, attention outputs passed layer by layer can suffer from information attenuation, and the effective representation capability of deep networks tends to saturate. The block-wise attention residual mechanism works as follows: within blocks composed of several adjacent layers, it establishes cross-layer residual paths for attention outputs, allowing shallow-layer attention results to bypass intermediate layers and participate directly in deep-layer computation.

    Its benefits appear in two aspects: first, it alleviates the representation bottleneck of deep networks and improves gradient propagation; second, it reuses shallow-layer attention computation results and reduces memory and communication overhead caused by repeated computation. In large-scale distributed training, this overhead is not negligible.

    6. Training Paradigm: Multi-Stage Context Extension and Agent Reinforcement Learning

    Ultra-long context capability is not achieved overnight. A mature industry practice is multi-stage progressive extension: first complete basic capability training with an 8K context, then extend sequentially to 64K and 256K, and finally advance to 1M. Each stage uses a data distribution of the corresponding length, together with common techniques such as RoPE base frequency adjustments, to ensure that the model converges stably at the new length before moving to the next stage.

    In the post-training stage, reinforcement learning for agent scenarios becomes a key increment. By constructing trajectory data from multi-turn tool calls and long-horizon task planning, the model learns from million-token interaction histories “which information is worth remembering and which can be discarded,” thereby turning the long-context ability from “fitting more” into “using it well.”

    7. Engineering Practice Lessons

    For teams planning to build or train large-scale sparse models, the above solutions can be distilled into four reusable lessons:

    1. At the operator level, prioritize solving numerical-stability parameterization problems so that attention variants can follow standard matrix multiplication paths. This is more important than implementing more elaborate algorithms.
    2. Load balancing should shift from loss design to hard constraints in the routing mechanism. The larger the expert scale, the more obvious the benefit.
    3. Long-context training must be advanced in stages. Directly training on ultra-long sequences has a very high failure rate.
    4. At the infrastructure level, general cloud capabilities can be used to build training and inference pipelines. For example, object storage can host multi-stage corpora, container platforms can manage distributed training jobs, and logging services and monitoring/alerting systems can track load distribution and loss anomalies during training, moving stability assurance forward to the observability layer.

    8. Conclusion

    This technical approach shows that the next stage of competition in large models has moved from simple parameter stacking to coordinated optimization across architecture innovation, numerical stability, and training systems engineering. The combination of hybrid linear attention and latent compression, auxiliary-loss-free balancing, cross-layer residuals, and multi-stage training all point to one goal: enabling trillion-level capacity models to train stably and infer efficiently at tens-of-billions-level cost. For engineering teams working on large-model R&D, understanding the principles and implementation paths of these mechanisms has more long-term value than tracking the release of any specific model.