Core Techniques and Engineering Practices for Trillion-Parameter Multimodal Large Models

Background: Large Model Competition Enters a New Stage of Ultra-Large Scale and Full Multimodality

As large model technology evolves, parameter counts are rapidly moving from the hundreds of billions toward the trillions, modality support is expanding from text-only input to images, audio, and video, and context windows are jumping from thousands of tokens to millions. These shifts impose three fundamental challenges on underlying architecture design.

First, how can computational efficiency and information retention be balanced in extremely long-context scenarios? Second, how can training stability and load balancing be maintained in Mixture-of-Experts architectures with hundreds or even thousands of experts? Third, how can models acquire deep reasoning and long-horizon planning capabilities for agentic tasks?

A recently released technical report on a MoE multimodal large model systematically addresses these problems. Its technical approach offers highly transferable lessons for teams building large-scale foundation models.

Architecture Overview: Scaling Up Both Capacity and Design

The model adopts a classic decoder-only architecture. Its core characteristics can be summarized as follows:

• Scale parameters: The total parameter count reaches 2.8 trillion, but the activated parameters are only 104 billion, yielding a sparsity ratio of approximately 27:1. This means that only about 3.7% of parameters are used during a single forward pass, significantly reducing per-step inference cost.
• Modality support: The model natively supports both vision and text inputs instead of relying on post-hoc adapters. This makes cross-modal understanding, such as explaining code logic from a diagram, a built-in capability rather than a patched-on feature.
• Context length: It supports a context window of up to 1 million tokens, equivalent to processing the full text of a lengthy novel trilogy in one pass. This provides a foundation for long-document parsing, complex codebase understanding, and multi-turn agent interaction.

Notably, the model does not rely solely on grouped-query attention or multi-query attention. Instead, it combines two techniques: a linear attention variant with bounded decay control, referred to in the report as KDA, and a gated multi-head latent attention mechanism, referred to as Gated MLA. The goal is to combine the high throughput of linear attention with the ability of latent attention to focus on critical information.

Innovation 1: Bounded Decay Parameterization for Unlocking Tensor Core Potential in Linear Attention

The Problem: Precision Overflow Reduces Hardware Utilization

Linear attention mechanisms replace Softmax with kernel functions, reducing computational complexity from quadratic to linear. However, on real hardware, these operators face a hidden bottleneck.

During chunked parallel computation, implementations must maintain a cumulative decay factor. To compute within a chunk in parallel, the reciprocal of this decay factor is required. Under limited-precision formats such as BF16, when the decay coefficient is very small, meaning long-distance dependencies are strongly retained, the reciprocal can expand rapidly and cause underflow or overflow.

As a result, many implementations are forced to add a special processing path for diagonal blocks that include the current chunk boundary. These blocks fall back to position-by-position computation and cannot use Tensor Cores for dense matrix multiplication. This prevents linear attention from reaching its theoretical throughput on GPUs.

The Solution: Replacing an Unbounded Decay Function with a Bounded Sigmoid Variant

The proposed solution is simple and engineering-friendly. The unbounded negative Softplus function in the decay coefficient log space is replaced with a scaled Sigmoid function that has a lower bound.

Specifically, the decay is computed as g_th = g_min * Sigmoid(e_ahz_th), where g_min is fixed at -5.

The core value of this parameterization is that within any 16-token data chunk, the reciprocal scaling factor is strictly limited to a safe numerical range. Because Sigmoid output is always between 0 and 1, multiplying by -5 ensures that the decay logarithm never falls below -5. This eliminates the risk of numerical overflow.

Engineering Benefits: A Unified GEMM Path and Higher Throughput

Since the special diagonal-block branch is eliminated, all causal blocks, both diagonal and non-diagonal, can directly use dense Tensor Core matrix multiplication instructions.

Experimental results show that this change nearly doubles GPU utilization for the attention computation unit. More importantly, the design naturally supports NoPE architecture, meaning the 1-million-token context can be directly extrapolated without requiring additional length extrapolation techniques or positional interpolation.

Innovation 2: Quantile Balancing for Expert Load Balancing Without Auxiliary Loss

The Challenge: Limitations of Traditional Load Balancing Methods

In a Mixture-of-Experts architecture with 896 routed experts, load balancing is essential for training stability. Traditional methods rely on auxiliary loss to constrain routing distributions. However, auxiliary loss introduces additional hyperparameters and competes with the main task loss, which can lead to suboptimal performance or training instability.

The Solution: Dynamically Adjusting Routing Thresholds Based on Quantiles

Quantile balancing offers a different approach. It completely removes auxiliary loss and adopts a deterministic balancing strategy based on statistics.

At each training step, the system computes quantile values from the routing scores of all tokens in the current batch, such as the K-th quantile, and uses these as dynamic thresholds.

• If an expert load exceeds the quantile threshold, its selection probability is suppressed.
• If an expert load falls below the threshold, its routing score is enhanced.

This mechanism can be understood as an adaptive regulator that ensures the number of activated experts remains within a healthy range at all times. Since computing quantiles only requires sorting routing scores, the additional overhead is negligible. Yet it can maintain training stability on models with hundreds of billions of parameters.

Innovation 3: Block Attention Residuals for Reshaping Deep Information Flow

The Problem: Representation Degradation in Deep Transformers

When a Transformer exceeds a certain number of layers, such as several hundred, the model becomes prone to representation collapse or optimization difficulty. Standard residual connections ensure gradient flow, but they do not guarantee that lower-layer features continue to contribute effectively to higher-layer representations.

The Solution: Cross-Layer Bypass for Attention States

The block attention residual mechanism treats the attention matrix itself, rather than just the attention output, as the information carrier across layers.

In practice, a bypass connection is added across multiple Transformer blocks, allowing attention computation in later layers to directly reference attention patterns from earlier layers. This is conceptually similar to feature pyramid ideas in computer vision, but it is much rarer in language models.

This design offers two intuitive advantages:

1. Deeper representation breakthrough: Information can skip multiple layers and fuse directly, avoiding loss during layer-by-layer transmission.
2. Memory and communication optimization: Because attention states are reused, some intermediate activations can be released, reducing peak memory in long-context scenarios. During cross-node training, this also means lower communication volume.

Training Strategy: Multi-Stage Context Extension and Agentic Reinforcement Learning

The training path is also notable. The model is not trained from scratch on long contexts in a single stage. Instead, it follows a progressive strategy.

• Pretraining stage: The context length advances in steps of 8K, then 64K, then 256K, and finally 1M tokens. This curriculum-style approach allows the model to focus on different levels of semantic relationships at different stages. It first builds local dependency modeling capability, then gradually extends to cross-paragraph and cross-document global dependencies.
• Post-training stage: Agentic reinforcement learning is introduced at the million-token level. Training tasks are designed to require multi-step reasoning, tool calling, and information retrieval across extremely long contexts. This directly aligns the model with real deployment scenarios where it acts as an agent hub for complex tasks rather than simple question answering.

Summary and Reusable Recommendations

For developers and algorithm engineers, the three key innovations in this technical report can be abstracted into the following practical principles:

Technical Innovation	Core Idea	Transferable Scenarios
Bounded decay parameterization	Use mathematical transformation to control numerical range and eliminate hardware special-case branches	Efficient GPU operator implementation for any linear attention variant
Quantile balancing	Replace auxiliary loss with statistical sorting to achieve dynamic balancing at near-zero cost	Load balancing in large-scale MoE and mixed-precision training
Block attention residuals	Pass intermediate states across layers to break the inter-layer information bottleneck	Extremely deep network architecture design and long-sequence model training

These methods do not depend on a specific data distribution or model size. They are broadly applicable. When developers face the engineering challenge of larger models and longer contexts, these approaches can serve as a reliable set of technical alternatives.

From an industry perspective, the release of technical reports like this is accelerating the transformation of trillion-parameter large models from exclusive experiments at a few institutions into reproducible and customizable general infrastructure for more teams.