How Large-Scale MoE Multimodal LLMs Are Evolving: Architecture, Training, and Engineering Practices

1. Background: Why “Trillion Parameters + Sparse Activation” Became the Main Path

As large models enter the trillion-parameter era, a core tension has surfaced: model capacity demands ever more parameters, while inference cost demands controllable compute per token. Mixture-of-Experts (MoE) was designed precisely to decouple this tension—carrying knowledge capacity with trillion-level total parameters while activating only up to tens of billions of parameters per token, trading a fixed inference compute budget for a higher intelligence ceiling.

Meanwhile, three business requirement lines are pushing architectural complexity further:

1. Native multimodal fusion: vision and other modalities are moving from “bolt-on encoders” toward unified modeling with language, and attention mechanisms must simultaneously handle cross-modal alignment and long-sequence modeling.
2. Ultra-long context: context windows are moving from tens of thousands of tokens toward the million-token scale, and the quadratic complexity of traditional softmax attention has become a hard bottleneck, forcing near-linear-complexity attention variants into the picture.
3. Agentization: models must complete multi-step tool calls and long-horizon reasoning over trajectories spanning millions of tokens, which imposes new requirements on both the training paradigm (reinforcement learning) and context memory mechanisms.

Several recently published technical reports on trillion-parameter MoE multimodal models point to highly similar solution combinations. Rather than focusing on a single model, this article takes a general technical perspective to break down the three key innovations most worth reusing by engineers, along with the accompanying training strategy and a practical deployment checklist.

2. Overall Architecture: A Systems Engineering Trade-off Across Capacity, Cost, and Stability

The architecture of these models typically consists of three cooperating layers:

(1) Attention layer: a hybrid of gated latent attention and incremental linear attention. Latent attention projects KVs into a low-dimensional latent space, substantially compressing the KV cache size during inference—a prerequisite for long-context deployment. Incremental (Delta-rule-style) linear attention replaces the ever-growing explicit KV cache with a fixed-size state matrix, reducing computational complexity to near-linear. By mixing the two at the layer level, the model can dynamically trade off between “precise retrieval” and “efficient compression.”

(2) Expert layer: hundreds of routed experts plus GLU-style activation variants. The feed-forward layer is replaced with a large sparse expert pool, paired with gated linear unit-style activations to increase representational density at the same activated parameter count. The real difficulty is not capacity but training stability under extreme sparsity—this is the subject of Section 4.

(3) Cross-layer information sharing: attention residuals. Shallow-layer attention outputs are injected into deeper layers as residuals, mitigating representational degradation in deep networks while amortizing GPU memory and cross-device communication overhead. See Section 5 for details.

3. Key Technique 1: Bounded Parameterization of Linear Attention Decay

The pain point. Delta-rule-style linear attention commonly uses chunk-wise parallelism: within a chunk, it must compute a “diagonal part” (interaction between the current chunk and historical state) and a “non-diagonal part” (token-to-token interaction within the chunk). The problem lies in the decay factors—to preserve numerical equivalence, implementations often need the reciprocal of the cumulative decay. The traditional approach parameterizes decay as an unbounded negative-softplus form; when sequences are long and decay is deep, the cumulative decay approaches zero and its reciprocal overflows directly under BF16 precision. In practice, engineers can only let the diagonal chunk degrade into a special per-position-pair computation path that bypasses matrix-multiply acceleration units, causing operator throughput to plummet.

The solution. Rather than doing numerical clipping at runtime, build the “bounded” constraint into the parameterization itself: change the log decay from an unbounded form to a scaled Sigmoid with a fixed lower bound:

# Traditional: unbounded decay, cumulative reciprocal prone to underflow
log_decay = -F.softplus(a)               # range (-inf, 0)
# Bounded parameterization: scaled Sigmoid, decay locked to a lower bound
g_min = -5.0                              # fixed lower bound
log_decay = g_min * torch.sigmoid(a)      # range (g_min, 0)

Take a 16-token compute chunk as an example: each token’s decay factor is no lower than e^(-5), so the cumulative decay over 16 tokens has a lower bound of e^(-80), whose reciprocal upper bound is roughly 5.5×10^34—still within BF16’s representable range (upper limit about 3.4×10^38). As a result, the diagonal and non-diagonal chunks can uniformly use dense GEMM, completely removing the special branch.

Three benefits:

• Throughput: the full path is accelerated by hardware matrix multiplication, greatly improving the engineering usability of the linear attention operator.
• Stability: training no longer produces NaNs from decay overflow, significantly reducing debugging cost.
• Extrapolation: combined with a no-positional-encoding strategy, direct extrapolation to long contexts becomes smoother.

The reuse value is high: the change is concentrated in a single line of decay parameterization code, yet it solves both the numerical and engineering bottlenecks at once. Any team building its own linear attention operators should borrow this approach.

4. Key Technique 2: Auxiliary-Loss-Free MoE Load Balancing

The pain point. Traditional MoE relies on auxiliary losses to penalize uneven expert load, but this has three persistent problems:

1. Auxiliary loss gradients interfere with the main loss, directly harming final accuracy.
2. Weights are extremely hard to tune: too loose a constraint skews load, too tight a constraint makes experts converge toward similarity.
3. When the number of experts reaches the hundreds and each token activates only a very small number of experts, balancing converges very slowly.

The solution: Quantile Balancing. The key shift in thinking is that load balancing is a control problem, not an optimization objective. Instead of modifying the loss function, add a dynamic bias that does not participate in gradient updates on top of the routing scores:

with torch.no_grad():
    load = expert_token_count / total_tokens        # normalized load per expert
    pivot = torch.quantile(load, 0.5)               # load quantile
    bias -= lr_b * torch.sign(load - pivot)         # overloaded experts lower bias, idle experts raise bias
logits = router(x) + bias                           # bias only affects routing selection

Its essence is a control loop: any deviation of load from the quantile triggers a correction, and convergence speed is governed by the bias update step size. It is fully decoupled from the main loss and does not pollute the gradient direction.

Benefits: expert-parallel all-to-all communication is no longer dragged down by a few hot experts, significantly improving training throughput. At the same time, because the accuracy penalty term is removed, expert differentiation becomes more pronounced, preserving the original intent of MoE—“each expert learns its own specialty.”

5. Key Technique 3: Block-Level Attention Residuals

Deep Transformers follow an empirical rule: beyond a certain depth, attention outputs tend to homogenize, and the marginal benefit of stacking more layers declines rapidly. A low-cost solution is cross-layer information sharing: group several layers into a block, project the shallow-layer attention outputs within the block, and add them to deeper layers as residuals.

This brings two benefits:

1. Representation level: deep computation always retains an anchor to the “original query perspective,” alleviating representational bottlenecks in very deep networks.
2. Cost level: intermediate activations can be reused, reducing GPU memory usage and easing the communication pressure of pipeline parallelism.

The engineering cleverness lies in the “block-level“ design: organizing residuals by block rather than fully connecting layer by layer avoids O(L²)-complexity cross-layer connections, keeping modification cost manageable.

6. Training Paradigm: Multi-Stage Context Extension and Long-Horizon Agentic RL

(1) Multi-stage pre-training: progressive progression from 8K → 64K → 256K → 1M. Training with ultra-long sequences all at once is both wasteful in compute and unstable. The logic of staged extension is: in early stages, learn language and world knowledge with short sequences and high data density; in middle stages, gradually inject long-horizon data such as long documents, code repositories, and multi-turn conversations; at each stage, fine-tune only position-related components (such as the frequency configuration of rotary positional encoding or decay parameters), preserving previously acquired short-range capabilities to the greatest extent. This “progressive gear-shifting” outperforms “one-step到位” in both compute efficiency and capability retention.

(2) Agentic RL post-training: million-token-scale long-horizon reinforcement learning. Pre-training grants knowledge; reinforcement learning shapes behavior. In verifiable environments such as tool calling, multi-step retrieval, and code execution, task success or failure serves as the reward signal to train the model to maintain goal consistency over ultra-long trajectories. This kind of training tests infrastructure in three main areas: storage and replay efficiency for ultra-long-context rollouts, automatic verifiability of rewards (preferring verifiable rewards such as test pass rates), and training stability (KL constraints plus gradient clipping as a double safeguard).

7. Engineering in Practice: A Reusable Checklist

Training side:

• Perform numerical boundary analysis on all custom operators first: map the overflow/underflow paths of every intermediate quantity under BF16. If a bounded parameterization can solve it, do not rely on runtime clipping as a fallback.
• Connect MoE load monitoring to training dashboards early: expert hit rates, overflow rates, bias magnitudes, and load variance should all be visualized with alerts.
• For long context, adopt staged extension paired with progressive data-mix switching to avoid abrupt capability regressions.

Inference and deployment side:

• The combination of linear attention and latent attention can reduce KV cache pressure by an order of magnitude, but be sure to test state-refresh boundaries and first-token latency.
• For large-scale expert parallelism, prioritize managing all-to-all communication hotspots; the balance of hot-expert distribution directly determines the throughput ceiling.
• For ultra-long-context scenarios, first measure the effective retrieval length in real business cases, avoiding overpaying for “million-level” metrics.

Platform side: Training such models places far higher demands on compute scheduling, checkpoint recovery, experiment tracking, and monitoring/alerting than traditional tasks. With general infrastructure capabilities such as GPU compute, container platforms, logging, and monitoring provided by Baidu AI Cloud, teams can focus their energy on algorithms and data rather than rebuilding the underlying engineering chain from scratch.

8. Conclusion

How should we view ultra-large-scale MoE multimodal models? It can be condensed into three statements:

1. Architecturally, it is a three-way trade-off across “capacity—cost—stability”: sparse activation solves cost, bounded parameterization and auxiliary-loss-free balancing solve stability, and block-level residuals solve the depth bottleneck.
2. In training, it is a combination punch of “multi-stage pre-training + long-horizon reinforcement learning”—short sequences learn knowledge, long sequences learn behavior.
3. In engineering, the real moat is not in the paper formulas but in details like numerical boundaries, communication hotspots, and load balancing—and these details are exactly what developers most deserve to accumulate and reuse.