Key Technical Architectures Behind Trillion-Parameter MoE Multimodal Models

    This article analyzes the core engineering techniques behind a recent open-source trillion-parameter sparse MoE multimodal model, covering three reusable innovations: a lower-bounded decay parameterization that stabilizes sparse attention in low precision, an auxiliary-loss-free load balancing method for ultra-large-scale MoE training, and chunked attention residuals combined with multi-stage long-context training. These techniques offer practical value for any team working on LLM training and inference optimization.

    1. Background: Why Trillion-Parameter Sparse Multimodal Models?

    Recent progress in large language models follows three clear trajectories: ever-growing parameter counts, context windows extending toward the million-token scale, and capability evolving from pure text toward multimodality and agentic behavior. Where these three lines converge, a core conflict emerges: the compute cost of dense architectures grows linearly or even superlinearly with parameter count, making it impossible to simultaneously satisfy the triple requirements of large capacity, long context, and low inference cost.

    Mixture-of-Experts (MoE) architectures were born to resolve this conflict. Take a recently published open-source technical report as an example: the model has 2.8 trillion total parameters, yet only about 104 billion parameters are activated per token—an activation ratio below 4%. This “large capacity, sparse activation” design allows the model to store massive knowledge while keeping per-token compute within acceptable bounds. The model also natively supports visual multimodal input and offers a 1M-token context window, providing foundational capability for long-document analysis, repository-level code understanding, and long-horizon agent tasks.

    This article examines three core technical points from that report. Each has direct reuse value for engineering teams working on LLM training and inference optimization.

    2. Core Technique 1: Lower-Bounded Decay Parameterization for Sparse Attention

    2.1 The Problem

    Linear attention and its variants (such as Delta-style operators), with their constant-size state, are leading candidates for long-context modeling. However, there is a long-overlooked engineering bottleneck: the numerical stability of the decay factor.

    Specifically, linear attention is typically trained with a chunked parallel strategy that splits the sequence into tiles of 16 or 32 tokens. When computing the intra-tile diagonal block, the reciprocal of the cumulative decay factor Γ must be taken. When the decay parameter is generated through an unbounded parameterization (such as negative Softplus), this reciprocal easily overflows or underflows in low-precision formats like BF16. As a result, the diagonal block cannot follow the standard matrix-multiplication path and degrades to position-by-position computation, losing access to dense Tensor Core GEMM acceleration.

    2.2 The Solution

    The report’s solution is remarkably simple: replace the unbounded log-decay parameterization with a lower-bounded scaled Sigmoid form:

    1. g_th = g_min * Sigmoid(A_h · z_th) # fixed g_min = -5

    The elegance of this change lies in three aspects:

    • Locked lower bound: Since the Sigmoid output always lies in (0, 1), the decay value is constrained to [-5, 0). The reciprocal scaling factor within any tile naturally stays in a safe numerical range, with no overflow or underflow under BF16 precision.
    • Unified compute path: The diagonal block no longer needs a special position-pair computation branch. All causal tiles (diagonal and off-diagonal) can be scheduled through dense Tensor Core GEMM uniformly, significantly improving operator throughput.
    • Support for NoPE: Because the decay mechanism implicitly carries temporal information, the model can adopt a NoPE (no positional encoding) scheme and extrapolate directly to million-scale contexts, simplifying long-context extension.

    2.3 Engineering Takeaway

    For teams implementing their own linear attention operators, the lesson is that numerical stability optimization does not necessarily require complex algorithmic changes. Sometimes adjusting only the mathematical form of a parameterization yields order-of-magnitude performance gains at the CUDA/CUTLASS level. This is a textbook example of algorithm-kernel co-design.

    3. Core Technique 2: Auxiliary-Loss-Free Load Balancing for Ultra-Large-Scale MoE

    3.1 The Dilemma of Traditional Approaches

    The classic challenge in MoE training is load balancing: if the router concentrates traffic on a few experts, the remaining experts are undertrained and model capacity is wasted. The common industry practice is to introduce an auxiliary loss to enforce balance, but the auxiliary loss creates gradient conflicts with the main task loss—the stronger the balancing, the greater the degradation of main-task performance. It is a compromise of trading performance for balance.

    When the number of experts scales to 896, the problem intensifies: the more experts there are, the fewer samples each expert receives, the more stochastic the routing becomes, and the more pronounced the side effects of the auxiliary loss.

    3.2 The Quantile Balancing Mechanism

    The quantile balancing strategy adopted in the report moves “balancing” from the loss-function level to the routing-decision level:

    1. Compute quantile statistics of each token’s routing scores across all experts, dynamically determining the routing threshold for the current batch.
    2. Experts exceeding the threshold are dynamically adjusted through quantile calibration, ensuring each expert’s load stays within a workable range.
    3. The entire process requires no additional loss term, so the main-task gradients remain uncontaminated.

    Combined with a SiTU-GLU-style improved activation function and gated multi-head latent attention for KV cache compression, this scheme maintains training stability under extreme sparsity, achieving both balance and performance.

    3.3 General Value

    Even for teams operating at a scale far below 896 experts, the auxiliary-loss-free balancing idea is worth borrowing: any approach that replaces regularization terms with data-driven dynamic thresholds can reduce the conflict cost of multi-objective optimization.

    4. Core Technique 3: Chunked Attention Residuals and Training Strategy

    4.1 Cross-Layer Information Sharing

    Deep Transformers suffer from a representational bottleneck: information must propagate layer by layer, and gradient decay in deep networks limits effective depth. The chunked attention residual mechanism establishes direct cross-layer information channels between attention modules, allowing shallow representations to bypass intermediate layers and reach deeper layers directly. The chunked design also reduces memory footprint and cross-device communication overhead. This is especially important for distributed training at the 10,000-GPU scale—every reduction in communication volume translates into higher cluster utilization.

    4.2 Multi-Stage Context Extension and Post-Training

    Pretraining follows a multi-stage schedule of 8K → 64K → 256K → 1M: the model first converges quickly on short contexts, then the window is expanded step by step, avoiding the memory and optimization pressure of long sequences from the start. More importantly, million-token-scale agentic reinforcement learning post-training is layered on top of pretraining, enabling the model to learn multi-step planning and tool use in real long-horizon tasks—precisely the key source of current agent capabilities.

    5. Conclusion and Reflections

    The value of this technical report lies not only in demonstrating a trillion-parameter model itself, but in providing a reusable engineering methodology:

    Technique Problem Solved Reuse Cost
    Lower-bounded decay parameterization Numerical overflow in sparse attention, operator efficiency Low, minimal changes
    Quantile load balancing Training instability in ultra-large-scale MoE Medium
    Chunked attention residuals Representational bottleneck and communication overhead in deep networks Medium
    Multi-stage long-context training Convergence difficulty at million-scale context High

    For teams planning to build LLM capabilities—whether through in-house development or by building applications on cloud-based LLM services (such as general-purpose model inference and fine-tuning platforms)—understanding these underlying technical principles supports better architecture selection and cost estimation. Sparsification, long context, and agentic capability will remain the three dominant themes of LLM evolution for the foreseeable future.