Large-Scale MoE Multimodal Large Language Models: Core Techniques

    This article examines the core techniques behind large-scale Mixture-of-Experts (MoE) multimodal large language models, focusing on sparse parameter activation, long-context attention, multimodal alignment, and training strategies. It explains how lower-bounded decay parameterization, gated multi-head latent attention, block-level attention residuals, and auxiliary-loss-free quantile load balancing help address efficiency, stability, and capacity challenges. It also covers multi-stage context expansion, agentic reinforcement learning, and practical engineering considerations for deployment.

    1. Starting Point: Why Sparsity, Long Context, and Multimodality Are Hard to Combine

    Before discussing specific solutions, it is useful to clarify three intertwined contradictions in this type of model. These contradictions largely determine the architectural trade-offs.

    The first is the tension between parameter sparsity and activation cost. To expand model capacity under limited compute, the industry commonly adopts the Mixture-of-Experts (MoE) structure, allowing each token to activate only a small portion of the parameters. For example, a typical configuration may have a total parameter count at the trillion scale, around 2.8 trillion, while only about 104 billion parameters are activated per token. This “large capacity, low activation” design substantially improves parameter efficiency. However, it also introduces problems such as uneven expert routing, training instability, and sharply increased memory and communication overhead.

    The second is the tension between context length and computational complexity. Standard self-attention has quadratic complexity with respect to sequence length. When the context grows from 8K tokens to millions of tokens, a naive implementation becomes impractical. The key challenge is how to preserve the ability to model long-range dependencies while ensuring that operators can run efficiently on mainstream acceleration hardware such as Tensor Cores.

    The third is the tension between multimodal input and language capability. Natively supporting modalities such as vision means the model must process heterogeneous information within a unified representation space, while not sacrificing performance on pure language tasks. This places high demands on pretraining data mixture and alignment strategies.

    2. Overall Architecture: Sparse MoE as the Backbone, Efficient Attention and Cross-Layer Information Sharing as Support

    The overall design of this type of model can be summarized as “sparse MoE as the main backbone, with efficient attention and cross-layer information sharing as the two wings.” The backbone carries massive parameters and knowledge capacity. The attention layers maintain computational efficiency under ultra-long contexts. The cross-layer information pathway mitigates representation degradation in deep networks. Only when these components work together can the model achieve competitive overall performance on open benchmarks.

    The following sections break down the core technologies at three levels: attention, MoE, and training, and highlight their reusable value.

    3. Key Innovations in the Attention Layer

    3.1 Linear Attention and Lower-Bounded Decay Parameterization

    Linear attention and its variants, such as Delta Net-like structures, reduce complexity from quadratic to linear by introducing decay factors. This is a key enabler for million-token context support. However, there is a subtle engineering pain point: when computing in chunks, the reciprocal of the accumulated decay factor Γ within a chunk may overflow or underflow severely under low-precision formats such as BF16.

    The traditional approach uses negative softplus to construct log-decay. Its values have neither an upper bound nor a strict lower bound, making the numerical range difficult to control. Once the reciprocal scaling factor exceeds the safe range, the diagonal tile must fall back to an explicit “position-pair” computation path. In that case, efficient dense matrix multiplication cannot be used, and Tensor Core compute capacity is wasted.

    The idea of lower-bounded decay parameterization is straightforward: replace the unbounded softplus with a scaled sigmoid that has a lower bound, using a fixed lower bound, for example −5, to lock the decay within a safe interval.

    1. # Example: lower-bounded log-decay parameterization
    2. g_min = -5.0 # fixed lower bound to keep values in a safe range
    3. # Traditional: unbounded, prone to overflow/underflow in reciprocal form
    4. # g = -F.softplus(theta)
    5. # Improved: scaled sigmoid, log-decay ∈ (g_min, 0)
    6. g = g_min * torch.sigmoid(theta)
    7. # Within any 16-token tile, the reciprocal of the accumulated decay Γ
    8. # stays within a safe range, allowing both diagonal and off-diagonal tiles
    9. # to use dense Tensor Core GEMM

    The benefits are twofold. First, it eliminates the special branch for diagonal tiles. All causal tiles can uniformly use dense GEMM, which significantly improves throughput. Second, because the decay is constrained, the model is naturally compatible with the NoPE approach, where “NoPE” means no positional encoding. Long-context extrapolation can be performed directly without retraining. This engineering change is small but delivers clear benefits, making it one of the most worth-reusing techniques in this type of architecture.

    3.2 Gated Multi-Head Latent Attention

    To control KV cache size and memory consumption when expanding context, the attention side typically introduces the idea of Multi-Head Latent Attention (MLA). MLA compresses keys and values into a low-dimensional latent space, substantially reducing cache size. On top of this, a gating mechanism allows the model to adaptively adjust the strength of information writing and reading on a per-token basis, further improving representational stability and efficiency for long sequences.

    This is a combined approach: using low-rank compression to reduce cache usage and using gating to enhance representational capacity. It is suitable for models with strong requirements on long-text and long-code scenarios.

    3.3 Block-Level Attention Residuals (Cross-Layer Information Sharing)

    Deep Transformers commonly suffer from the problem that “the deeper the network, the harder it becomes to learn useful representations.” Block-level attention residuals share and aggregate the outputs of attention modules across layers at block granularity in a residual form. On one hand, this breaks the representation bottleneck caused by depth and allows fine-grained information from shallow layers to be transmitted more effectively to deeper layers. On the other hand, the block-level design controls additional memory and communication overhead, avoiding the cost explosion caused by fully connected residuals. Compared with simply stacking layers, this structure achieves a better balance between information fidelity and resource consumption.

    4. MoE Layer: Stability and Balance under Extreme Sparsity

    4.1 Large-Scale Routed Expert Design

    Under extreme sparsity, the number of experts can be made very large. In a typical configuration, the number of routed experts can reach nearly one thousand, for example about 896. More experts mean that each token can “see” a more diverse set of sub-networks, increasing the upper bound of model capacity. However, this also amplifies routing variance, expert load skew, and training stability issues. Therefore, an extremely large expert pool must be paired with fine-grained load balancing and numerical stability strategies.

    4.2 Auxiliary-Loss-Free Quantile Balancing

    Traditional MoE relies on auxiliary loss to constrain expert load. However, auxiliary loss often competes with the main task objective for gradients, which can harm model quality. In addition, its weight is difficult to tune.

    A better approach is auxiliary-loss-free quantile balancing, or Quantile Balancing. This method dynamically adds a bias to routing scores based on the historical load quantiles of each expert. The bias is applied only during training, while inference uses only the original routing logits.

    1. # Example: auxiliary-loss-free quantile balancing
    2. load_q = compute_quantile(expert_load) # compute load quantiles for each expert
    3. bias = adjust_by_quantile(load_q, target="uniform") # adjust toward a uniform target
    4. train_logits = router_logits + bias # add bias during training
    5. # During inference, only the original router_logits are used; no extra loss is introduced

    The advantage of this approach is that it transforms load balancing from a loss term into a dynamic adjustment of routing bias. It does not interfere with the main-task gradient and can quickly converge to a balanced state even at extremely large expert scales. This makes it well suited for large-scale MoE systems that require rapid dynamic load balancing.

    4.3 Gated Activation Function Variants

    Activation functions directly affect representational capability and numerical stability. The industry commonly uses gated activation structures such as SwiGLU. Variants of these structures may be designed with techniques such as smooth truncation and scaling to improve gradient distribution and numerical stability while preserving representational power. These modifications usually do not require changing the main network backbone, making them low-cost and pluggable optimization points.

    5. Training Strategy: Multi-Stage Context Expansion and Agentic Reinforcement Learning

    Ultra-long context is not achieved in a single step. Instead, a multi-stage progressive expansion strategy is used. The model first converges on a shorter context, such as 8K, to stabilize basic capabilities. It then progressively advances to 64K, 256K, and finally up to 1M tokens. This short-to-long, curriculum-style training allows the model to adapt smoothly to positional extrapolation and avoids training divergence or performance collapse caused by directly training on long sequences.

    In the post-training stage, in addition to conventional alignment and preference optimization, this type of model also adds million-token-level agentic reinforcement learning (Agentic RL). This focuses on difficult problems such as long-horizon planning, multi-step tool calling, and long-horizon reasoning consistency. Reinforcement learning helps the model maintain goal consistency in long-horizon tasks and reduces mid-task forgetting and logical drift. This is a key step in moving from “being able to read long text” to “being able to complete long-horizon tasks.”

    6. Engineering Recommendations

    From an engineering perspective, the following general practices can be considered when deploying this type of model:

    • Prioritize numerical stability: For critical paths such as decay factors, attention scores, and MoE routing, use low-precision safe-range constraints in a unified way. For sensitive operators, preserve a controlled high-precision fallback path.

    • Distributed and parallel computing: Extremely large expert pools are naturally suited to combinations of expert parallelism (EP), data parallelism, and tensor parallelism. Particular attention should be paid to all-to-all communication caused by expert routing and to peak memory usage.

    • Observability: Training and inference tasks can be hosted on container platforms, while distributed storage can be used for large-scale corpora and checkpoints. Logging services and monitoring alerts can track loss, expert load, memory usage, and throughput in real time, helping detect skew and anomalies promptly.

    • Evaluation loop: In addition to horizontal comparisons on standard benchmarks, dedicated evaluation sets should be built for target scenarios such as long-context retrieval, long-horizon agent tasks, and multimodal understanding. This helps avoid the problem of “high benchmark scores but failure in real scenarios.”

    7. Conclusion

    The core competitiveness of ultra-large-scale MoE multimodal models comes from the synergy of three layers: attention, MoE, and training strategy. Lower-bounded decay parameterization makes linear attention both efficient on acceleration hardware and extrapolatable to longer contexts. Gated latent attention and block-level residuals balance cache usage, representational power, and depth. Auxiliary-loss-free quantile balancing tames extremely sparse routing. Multi-stage context expansion and agentic reinforcement learning further complete the capability leap.

    For engineering teams, prioritizing techniques that are “small to change but large in benefit,” such as lower-bounded decay, quantile balancing, and block-level residuals, often provides the most obvious stability and efficiency improvements at the lowest cost.