Large-Scale MoE Multimodal Model Architecture and Training Techniques

    This article examines the architecture and training methodology of a large-scale sparse MoE multimodal model with 2.8 trillion total parameters and 104 billion activated parameters. It covers hybrid attention combining linear and latent attention, load balancing for 896 routed experts without auxiliary losses, a bounded log-decay parameterization that enables dense GEMM computation, cross-layer attention residuals, and multi-stage context extension. The article also distinguishes claims supported by the original report from those that remain unverified.

    1. Why Dense Transformers Are Hard to Scale Further

    In a dense model, every token passes through all parameters. Once the parameter count reaches the trillion scale, compute cost, memory usage, and training stability all become limiting factors. Standard softmax attention scales quadratically with sequence length, which is especially costly in long-context scenarios.

    MoE (Mixture of Experts) decouples the total parameter count from per-token compute. Each token activates only a small number of experts, so the total parameter count can be very large while the cost of a single forward pass stays manageable. This structure introduces new problems: as the number of experts grows, each expert receives fewer tokens, routing becomes prone to imbalance, and training becomes less stable.

    According to the original report, the model has 2.8 trillion total parameters with 104 billion activated parameters, and natively supports visual input and a 1M-token context. The report does not specify how parameters are counted—for example, whether the visual encoder and embedding layers are included. These figures should be verified against the official technical report before being cited.

    2. Hybrid Attention: Combining Linear Attention and Latent Attention

    The original report combines KDA with Gated MLA (Gated Multi-head Latent Attention) and adds attention residuals to share information across layers. KDA, translated in the report as “delta attention,” belongs to the family of linear attention mechanisms.

    The rationale behind this hybrid design is as follows:

    • Linear attention scales approximately linearly with sequence length, making it suitable for long sequences, but it is generally weaker than softmax attention at precise long-range retrieval.
    • Multi-head latent attention compresses keys and values (KV) into low-dimensional latent vectors, reducing the memory footprint of the KV cache during inference.
    • The hybrid structure delegates long-range processing to linear-type layers while retaining some global attention layers, balancing efficiency and retrieval capability.

    The report does not disclose the ratio of linear layers to global layers, nor the KV cache compression ratio. Both numbers are essential when evaluating whether this design fits a given use case.

    3. Sparse MoE: Routed Experts and Load Balancing

    The report states that the model contains 896 routed experts in an architecture named Stable LatentMoE. For the feed-forward activation function, it uses SiTU-GLU, intended to maintain numerical stability under a large number of sparse experts. The exact formula of this function is not provided.

    Load balancing is the central challenge for such models. A common approach adds an auxiliary term to the loss function to penalize uneven loads. However, an auxiliary loss alters the optimization direction of the main task. The report instead uses a Quantile Balancing mechanism: it tracks the distribution of tokens received by each expert and adjusts routing biases according to a set of rules, without introducing an auxiliary loss.

    When implementing bias adjustment, the following points deserve attention:

    1. Statistics window: whether to accumulate per step or use a sliding window. A window that is too short causes routing jitter.
    2. Update step size: if the bias changes too much per step, token assignments to experts switch frequently, hurting convergence.
    3. Observed metrics: record each expert’s token share and the ratio of maximum load to average load. If a capacity limit is set, also record the fraction of overflow tokens.

    The report provides neither hyperparameters nor comparative experiments, so claims of “faster convergence” and “higher accuracy” cannot be verified from the report alone.

    4. Bounded Log-Decay Parameterization: Keeping Chunked Computation on Matrix Multiplication

    Problem Background

    During training with linear attention, the sequence is usually split into chunks. Within each chunk, a decay factor Γ must be accumulated, and its reciprocal is used for scaling. As Γ accumulates over positions it becomes very small, so its reciprocal grows large, which can easily overflow or lose precision in low-precision formats such as BF16. To work around this, implementations often handle diagonal blocks separately, which breaks the continuous invocation of general matrix multiplication (GEMM) on Tensor Cores.

    The Proposed Solution

    The report changes the parameterization of log-decay from Negative-Softplus (unbounded) to a scaled sigmoid with a lower bound:

    1. # Pseudocode from the original report; g_min is a fixed lower bound
    2. g_th = g_min * Sigmoid(e_A * h_t) # g_min = -5

    Here, h_t is the hidden state of the t-th token, and e_A is a learnable parameter in the report. The report does not specify the shape or initialization of e_A, nor the dimensionality of h_t. According to the pseudocode, the log-decay falls within (-5, 0), so the per-token decay factor falls within (e^-5, 1).

    Numerical Check

    With the settings in the report—a chunk size of 16 tokens and a log-decay lower bound of -5—the accumulated log-decay within a chunk is at most -80, corresponding to a maximum reciprocal scaling factor of about exp(80) ≈ 5.5×10^34. BF16 has the same number of exponent bits as FP32, with a maximum finite value of about 3.4×10^38, so this upper bound is within the representable range. This result is derived from the parameters given in the report, not a conclusion stated in it. Note that BF16 has only 7 mantissa bits, so precision loss still exists and must be validated with real training data.

    Claimed Benefits

    The report claims that this parameterization allows all causal blocks to call dense GEMM directly, significantly improves training throughput, and supports 1M-context extrapolation without positional encoding. All three claims require supporting data: hardware model, baseline implementation, sequence length, batch size, and how throughput is measured (e.g., tokens/s or MFU). Since the report provides none of this information, these points can only be treated as design goals pending verification.

    5. Cross-Layer Attention Residuals

    In deep Transformers, representations gradually converge, and the efficiency of inter-layer information transfer declines. The report introduces block-level attention residuals (AttnRes), which share part of the attention computation results between adjacent layers.

    The potential benefits fall into two categories: reduced redundant computation and reduced cross-node communication. The report claims this is advantageous for training on clusters of tens of thousands of GPUs, but provides no figures on communication volume or memory savings. When reading the report, two questions are worth examining: which intermediate results are shared, and whether sharing changes gradient paths in ways that affect training stability.

    6. Multi-Stage Context Extension and Agentic RL Post-Training

    Context Extension

    The report uses four pre-training stages at 8K, 64K, 256K, and 1M context lengths, progressively extending from short to long. The common rationale for this approach is that short-sequence training is cheaper: foundational capabilities are built on short text first, and the model then gradually adapts to long sequences. In practice, the following three points need to be made explicit:

    • The data mixture at each stage—for example, the proportions of long documents, code, and multimodal data.