Ultra-Scale MoE Multimodal LLMs: A Deep Dive into Sparse Training, Long Context, and Agentic RL
A technical walkthrough of how today's ultra-scale Mixture-of-Experts multimodal models are engineered. It covers why extreme sparsity plus million-token context creates new problems, and then breaks down five proven responses: lower-bounded decay parameterization for numerically stable linear attention, hybrid attention with gated MLA and block attention residuals, auxiliary-loss-free expert balancing via Quantile Balancing, progressive long-context pretraining, and Agentic RL for long-horizon reasoning. It closes with the three highest-leverage techniques to reuse and the infrastructure requirements for training and serving models at this scale.
Why “ultra-scale + extreme sparsity + native multimodality” is the new paradigm
Over the past two years, the center of gravity in LLM design has shifted from “dense and single-modality” to “ultra-scale, extremely sparse, natively multimodal.” One route that has become typical: adopt a Mixture-of-Experts (MoE) architecture, push total parameters into the trillions, and keep the parameters actually activated per token under a hundred billion. That balances model capacity against per-forward-pass compute. On top of it, vision is fused natively and context is extended to the million-token range.
The payoff is straightforward — capacity can scale almost without limit, while inference cost depends only on the activated parameters. But the same design also amplifies three engineering problems that dense, smaller models had hidden:
- Long-context compute efficiency: attention grows quadratically with sequence length, which makes dense attention close to infeasible at a million tokens.
- Training stability with hundreds of experts: routing across hundreds or nearly a thousand experts is prone to severe load skew, and sparse gradients make training signals fragile.
- Long-horizon agentic reasoning: an agent has to stay logically consistent across very long trajectories, which raises new demands on post-training.
The rest of this article breaks down the countermeasures that have already been validated in practice.
1. Numerical stability in linear attention: lower-bounded decay parameterization
Standard Softmax attention has compute and memory complexity that grows quadratically with sequence length, and it is the first bottleneck for long context. Linear attention — along with incremental architectures such as the DeltaNet family — reduces that complexity to linear. The core mechanism is a state matrix that can be updated recursively, paired with chunk-wise parallel algorithms to actually extract GPU throughput.
In practice, though, these methods have long been held back by a subtle numerical problem: the reciprocal of the within-chunk accumulated decay overflows or underflows at finite precision.
Take BF16 as an example. The within-chunk accumulated decay Γ is built by summing a series of log-decay terms. If the log decay uses an unbounded negative softplus form, it can approach negative infinity at extreme values, which leads to:
- The reciprocal of the decay factor,
1/Γ, overflowing to Inf or underflowing to 0, corrupting the entire block’s output; - Diagonal tiles being forced down an explicit “position-pair” computation path to work around this, which rules out dense matrix multiplication (GEMM) on Tensor Cores and wastes compute.
Lower-bounded decay parameterization takes a remarkably light approach: replace the unbounded log decay with a bounded function that has a lower bound, using a scaled Sigmoid to pin it inside a safe interval.
# Unbounded baseline: log decay can approach -inf, so the reciprocal can overflow# log_decay = -softplus(theta)# Lower-bounded decay parameterization: clamped to [g_min, 0), reciprocal range stays controlledg_min = -5.0log_decay = g_min * torch.sigmoid(theta) # always lower-bounded by g_mindecay = torch.exp(log_decay) # reciprocal is safe within any 16-token tile
Because the decay is restricted to [g_min, 0) (with g_min = -5 as a common engineering choice), the reciprocal scaling factors inside any 16-token block fall within the numerically safe range. Two consequences matter:
- No special-case branch: diagonal and non-diagonal tiles no longer need separate handling. Every causal tile can go through dense Tensor Core GEMM, which raises throughput noticeably;
- Extrapolation comes for free: combined with the recursive nature of linear attention, and without any explicit positional encoding (NoPE), the model can extrapolate directly to million-token lengths.
The reuse value of this change is very high: the engineering effort is tiny (a few lines of parameterization), yet it directly improves how the core operator is implemented on CUDA/CUTLASS. When building your own long-context model, this is the highest-ROI change to make first.
2. Hybrid attention: gated MLA plus block attention residuals
Linear attention alone still loses accuracy on long-range exact retrieval, so the mainstream approach is a hybrid architecture: cheap linear attention handles long-range information compression, while exact full attention handles fine-grained modeling of critical tokens, with the two alternating by layer.
Two further designs are worth attention on top of this.
Gated multi-head latent attention (Gated MLA). The core idea of latent attention (MLA) is to compress Key/Value into a low-dimensional latent space, cutting the KV cache substantially. This is decisive at million-token context, because the KV cache is often the number one consumer of memory. Adding gating lets the model modulate attention outputs dynamically, increasing selectivity in how representations are expressed.
Block attention residuals. Deep Transformers tend to hit a representation bottleneck where each additional layer learns less new information. Block attention residuals extend the granularity of residual connections from a single layer to a block, creating a shared information path across layers. This breaks the depth-representation ceiling while the block-level design keeps memory footprint and cross-device communication lower — and in ultra-scale distributed training, communication is frequently the hidden bottleneck.
3. Ultra-scale sparse MoE: stable training and auxiliary-loss-free balancing
Once routing spans hundreds or nearly a thousand experts at a very low activation ratio, load balancing becomes the make-or-break factor for convergence. If routing keeps favoring a small subset of experts, you get “hot experts overloaded, cold experts starved” — capacity is wasted, and sparse gradients oscillate violently.
The conventional solution relies on an auxiliary load-balancing loss: on top of the language modeling objective, an extra term penalizes load imbalance. It works, but it carries two side effects — the additional gradient interferes with optimization of the main task, and its weight is a sensitive hyperparameter that can degrade model quality if tuned badly.
Quantile Balancing (QB) offers an auxiliary-loss-free alternative: instead of injecting a penalty into the loss function, it monitors the load distribution across experts online and adjusts routing biases dynamically by quantile, pushing down the probability of overloaded experts and raising that of underloaded ones until the load converges toward balance.
Its value: it avoids gradient interference from the auxiliary loss, so convergence is faster; it is more stable at scale; and it needs almost no dedicated tuning for load balancing. For teams training extremely sparse MoE models, this kind of external, decoupled balancing mechanism is more controllable than a built-in loss term.
4. Progressive long-context pretraining: curriculum-style scaling instead of one big jump
Training directly at a million tokens is expensive and easy to destabilize. The proven approach is multi-stage progressive scaling — for example, advancing through 8K → 64K → 256K → 1M in steps:
- Curriculum effect: the model first masters foundational language ability on short sequences, then adapts to longer dependencies stage by stage, with each stage reusing the weights and knowledge from the previous one;
- Controlled cost: the large majority of training tokens are naturally short, so the long-context stages need comparatively little training volume, concentrating expensive ultra-long-sequence training in the final stage;
- Smooth positional extrapolation: pairing this with RoPE base-frequency adjustment or a NoPE design keeps length scaling from causing a violent shift in the attention distribution.
This “short to long” progression has become the general best practice for long-context models.
5. Agentic RL post-training: cracking long-horizon reasoning
Pretraining decides what the model can do; post-training decides whether it can finish the job. Typical agentic tasks involve multi-step tool calls, task planning, and long-horizon reasoning, with trajectories running from tens of thousands to a million tokens. Using reinforcement learning (RL) for post-training means designing rewards around “was the task completed,” and solving two hard problems:
- Sparse rewards: a clear success or failure signal only exists at the end of the trajectory, leaving intermediate steps without feedback;
- Long-horizon credit assignment: determining which step did something right or wrong.
The countermeasures combine trajectory-level outcome rewards with step-level process rewards, helping the model stay logically consistent and aligned with the goal across very long trajectories. Million-token-scale Agentic RL training exists precisely so the model can “remember, reason clearly, and act correctly” on real long-horizon tasks.
6. Engineering takeaways and deployment considerations
Overall, three designs have the highest reuse value:
- Lower-bounded decay parameterization: a tiny change that unlocks an efficient Tensor Core implementation of the linear attention operator;
- Quantile Balancing (QB): auxiliary-loss-free dynamic balancing for ultra-scale MoE — decoupled and cheap to tune;
- Block attention residuals: breaking the depth-representation bottleneck while lowering both memory and communication cost.
On the deployment side, training and serving models at this scale puts heavy demands on infrastructure: elastic GPU compute to support distributed training; distributed training frameworks for data, tensor, and pipeline parallelism; object storage to hold massive training datasets and checkpoints; message queues to connect the data pipeline; and logging plus monitoring and alerting to keep training observable and speed up fault localization. Baidu AI Cloud, for instance, offers cloud product categories such as general-purpose compute, storage, containers, and monitoring that can serve as infrastructure options for this kind of large-model training and inference work, letting teams compose and scale resources as needed. Confirm current availability in your target region against the official documentation before relying on any specific option.
Closing
The competitive core of ultra-scale MoE multimodal models is not just “more parameters.” It is the set of carefully designed mechanisms that tame several mutually conflicting demands — extreme sparsity, ultra-long context, multimodality, and strong agentic behavior — at the same time. For developers, understanding the five main threads — numerical stability, hybrid attention, auxiliary-loss-free balancing, progressive context scaling, and Agentic RL — is worth more than memorizing any one model’s name, because these designs are general engineering capabilities that transfer directly into a pipeline you build yourself.
