Three Key Techniques for Training Long-Context Multimodal MoE Large Models

    This article examines the main engineering challenges involved in training long-context, multimodal, and ultra-large-scale mixture-of-experts models. It explains three key techniques: scaled sigmoid lower-bounded decay for stable linear attention, auxiliary-loss-free quantile balancing for MoE routing, and block attention residuals for deep representation flow. It also discusses multi-stage context extension, agentic reinforcement learning, and cloud deployment considerations.

    1. Why long-context MoE multimodal models are difficult to train

    Over the past year or two, the capability boundaries of large models have expanded rapidly along three dimensions: context windows have grown from 8K tokens toward 1M tokens, modalities have expanded from pure text to native visual understanding, and model scales have moved from hundreds of billions of parameters toward trillions of parameters.

    When a model simultaneously combines “million-level context + multimodality + ultra-large-scale MoE,” it is no longer facing a single engineering problem. Instead, it must deal with three tightly coupled systemic challenges:

    1. Efficiency and numerical stability of attention computation: Standard attention has quadratic complexity with respect to sequence length, making memory and compute costs prohibitive at million-token scale. Replacing it with linear or recurrent attention introduces new issues such as numerical overflow and operator parallelization.
    2. Training stability with massive expert sets: Once the number of experts reaches hundreds or even thousands, routing load can easily become imbalanced. A small number of “popular experts” become overloaded, while “cold experts” receive little gradient. This wastes parameters and can also hinder training convergence.
    3. Representation degradation in deep networks: When the number of layers grows beyond several dozen, features learned in shallow layers may not be efficiently propagated into deeper layers. Gradients and information can be diluted as they pass through layer after layer.

    The following sections discuss the core techniques and engineering implementation ideas corresponding to these three challenges.

    2. Challenge one: linear attention and lower-bounded decay parameterization

    2.1 From quadratic complexity to linear complexity

    Standard attention explicitly constructs the attention matrix Q·Kᵀ, whose time and memory cost grow quadratically with sequence length L. At million-token scale, this approach is practically infeasible.

    The core idea of linear attention is to use the associativity of matrix multiplication and rewrite (Q·Kᵀ)·V as Q·(Kᵀ·V), reducing the complexity to linear growth with respect to L.

    In engineering practice, this is usually implemented through chunk-wise parallelism: the sequence is divided into fixed-size tiles, such as 16 tokens per tile. Dense matrix multiplication inside each tile is performed using Tensor Cores, while information across tiles is passed through a continuously updated recurrent state.

    2.2 Numerical underflow: the subtle “reciprocal”

    The problem is that incremental linear attention and gated decay mechanisms multiply a decay factor Γ at each time step to ensure that “more distant history receives lower weight.” In chunked computation, to allow all positions within a tile to share the same cross-tile state, the algorithm needs to apply a reciprocal scaling of this decay.

    If the decay is parameterized in logarithmic form as log Γ and is allowed to approach negative infinity, which is common when using unbounded functions such as negative Softplus, then the numerical range of its reciprocal becomes unbounded.

    Under finite-precision formats such as BF16, the reciprocal of the accumulated decay is highly prone to overflow to Inf or underflow to 0. This forces diagonal blocks to fall back to an explicit position-pair computation path, bypassing the fast Tensor Core matrix multiplication path and eliminating most of the performance advantage.

    2.3 Lower-bounded decay parameterization (scaled sigmoid)

    A straightforward solution is to impose a lower bound on the decay and compress its unbounded range into a safe numerical interval. A typical approach is to replace the unbounded negative Softplus with a scaled sigmoid that has a lower bound:

    1. # Illustrative code: lower-bounded log-decay parameterization
    2. # g_min is the lower bound for decay (e.g., -5), ensuring log_decay has a lower bound
    3. g = g_min * sigmoid(theta @ h + a) # g = log Γ, in (g_min, 0)
    4. gamma = exp(g) # Γ in (exp(g_min), 1)
    5. inv_gamma = exp(-g) # upper bound of reciprocal = exp(-g_min), always finite

    With this formulation, the reciprocal scaling factor inside any 16-token tile is constrained to a finite range, such as exp(5) ≈ 148 when g_min = -5. This range is sufficient for BF16.

    More importantly, this removes the special computation branch for diagonal blocks, allowing all blocks to use the same dense matrix multiplication path. As a result, the throughput of linear attention operators can be brought closer to the limit of Tensor Core execution.

    Reuse value: This type of change has low engineering intrusiveness. Often, only an activation function or decay parameterization function needs to be replaced, yet the efficiency of linear attention operators in low-level libraries such as CUDA and CUTLASS can be significantly improved. It is a typical “small change, large benefit” optimization.

    3. Challenge two: load balancing for ultra-large-scale MoE

    3.1 Load skew with hundreds or thousands of experts

    The mixture-of-experts (MoE) architecture uses a gating network to select a small number of experts for each token. This balances “a very large total parameter count” with “controllable compute cost per token.”

    However, when the number of experts reaches hundreds or even thousands, such as a typical configuration with 896 routed experts, and the activated parameter count reaches the tens-of-billions range, a serious issue emerges: routing can easily become highly skewed.

    Without constraints, the gating network tends to send most tokens repeatedly to the same group of experts, causing:

    • Some experts to become overloaded and turn into bottlenecks for communication and computation;
    • Many experts to receive almost no gradient, wasting “nominal parameters”;
    • Load imbalance to be further amplified under cross-device parallelism, slowing overall training.

    3.2 The cost of traditional auxiliary losses

    A classic approach is to add an auxiliary load-balancing loss to the training objective, penalizing load skew. This method is effective, but it introduces two side effects:

    • It requires tuning the loss weight, and poor tuning can interfere with convergence of the main task;
    • The auxiliary loss can create gradient conflict with the main loss, which is especially difficult to control in large-scale training.

    3.3 Auxiliary-loss-free quantile balancing

    A more elegant route is auxiliary-loss-free dynamic balancing: directly use the current quantile distribution of tokens received by each expert to dynamically recalibrate gating thresholds or per-expert capacity, pushing token allocation toward balance.

    Its key advantages are:

    • No additional loss term or weight hyperparameter is required, so the main task gradient is not disturbed;
    • Dynamic adaptation: balancing is adjusted at each training step based on real-time load, rather than relying on a static prior;
    • Compatibility with extreme sparsity: stable balancing can still be maintained when each token activates only a very small number of experts.

    Combining this with a gating-type activation function, such as a SwiGLU-style multiplicative nonlinearity that combines SiLU or GELU with a linear gating term, can make expert routing decisions smoother while preserving expressive power. This further reduces the difficulty of balancing.

    Together, these designs help ensure training stability under extremely sparse conditions, which is a prerequisite for trillion-scale MoE convergence.

    4. Challenge three: attention residuals and deep representation propagation

    4.1 Representation bottlenecks in deep Transformers

    As Transformers become deeper, a phenomenon often appears: representations in deeper layers degenerate into simple copies or minor perturbations of shallow-layer representations. The network becomes nominally deep, but deeper layers do not effectively learn new things.

    The root cause is that information transfer between layers is too sparse. Rich features extracted in shallow layers are difficult to propagate efficiently into deeper layers.

    4.2 Block attention residuals

    The idea behind block attention residuals is to share the output of attention modules across layers and across blocks in a residual form. This allows different layers and blocks to reuse each other’s attention representations instead of computing in isolation.

    This approach provides two benefits:

    1. It breaks the deep representation bottleneck: once information is propagated across layers, deeper layers no longer need to “re-learn” features already captured by shallow layers, improving parameter utilization.
    2. It reduces memory and communication overhead: by organizing computation in a block-wise manner, it avoids the memory peaks and cross-device communication pressure caused by globally dense residual connections.
    1. # Illustration: reuse information through block attention residuals
    2. # Maintain a block-shared residual accumulator instead of passing each layer independently
    3. attn_res_block = None
    4. for layer in layers:
    5. h = layer(h)
    6. h = h + attn_block(h) # attention output of the current layer
    7. if block_boundary(layer): # aggregate and share at block boundaries
    8. attn_res_block = combine(attn_res_block, h)

    5. Training paradigm: multi-stage context extension + agentic reinforcement learning

    5.1 Progressive pretraining from short to long context

    Pretraining directly on million-token sequences is both expensive and inefficient. In practice, progressive context extension is commonly used: first build language and knowledge foundations at 8K context, then gradually move to 64K, 256K, and finally 1M tokens.

    At each stage, the model adapts to longer dependencies step by step. Meanwhile, position encoding and attention states can extrapolate smoothly with the growing context, avoiding sudden distribution shifts on long sequences.

    5.2 Agentic RL post-training at the million-token scale

    Pretraining is good at “understanding,” but it is still insufficient for agent tasks that require multi-step planning, tool calling, and long-horizon reasoning.

    Therefore, post-training needs to introduce agent-oriented reinforcement learning. Moreover, the training scenarios themselves need to cover million-token-level long-horizon tasks. This allows the model to learn retrieval, planning, error correction, and tool orchestration within extremely long contexts, upgrading “long-document reading ability” into “long-horizon reasoning ability.”

    6. Engineering deployment and cloud considerations

    Making these techniques work in practice requires strong coordination across the underlying infrastructure. From an implementation perspective, large-scale training and inference usually rely on several general cloud capabilities:

    • Heterogeneous compute and distributed training scheduling: large-scale GPU or accelerator clusters are typically hosted through container platforms, relying on efficient parallelism strategies such as data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism, as well as collective communication libraries.
    • Parallel storage and data supply: high-throughput object storage and file systems are needed to support massive training corpora and frequent checkpoint read/write operations.
    • Observability: logging and monitoring/alerting systems are used to track loss, expert load, gradient norms, memory usage, and communication bottlenecks, making it easier to identify routing skew or numerical anomalies during training.
    • Stable inference services: inference for long-context and multimodal models requires KV cache management, dynamic batching, and elastic scaling.

    For example, Baidu AI Cloud’s container platform, distributed storage, monitoring/alerting, and other general capabilities can be combined into an infrastructure base for training and serving ultra-large-scale models, allowing teams to focus more on algorithms and data.

    7. Summary: a reusable checklist of three techniques

    At the most practical level, the most valuable engineering techniques in this article are three highly reusable methods:

    Technique Pain point solved Reuse value
    Lower-bounded decay parameterization (Scaled Sigmoid) Numerical overflow/underflow of linear attention in BF16, preventing diagonal blocks from using matrix multiplication Small change, but significantly improves operator efficiency
    Auxiliary-loss-free quantile balancing (QB) Expert load skew in ultra-large-scale MoE; difficulty tuning auxiliary losses Hyperparameter-free, dynamically adaptive, suitable for extreme sparsity
    Block attention residuals (Block AttnRes) Representation degradation in deep Transformers Breaks depth bottlenecks and reduces memory/communication overhead

    Long-context + multimodality + trillion-scale MoE is essentially a balancing act among efficiency, stability, and representational capacity. Understanding the numerical pitfalls of linear attention, the balancing principles of ultra-large-scale routing, and the residual design for deep information propagation captures the core engineering logic of this class of large models.