Technical Evolution of Ultra-Large-Scale MoE Models: Key Engineering Challenges and Solutions

I. Introduction: Beyond Parameter Scale, Three Crucial Questions

In recent years, the competition among open-source large language models has shifted from “who has more parameters” to “who can solve harder problems under equal compute constraints.” Recent practices from new-generation open-source models reveal common characteristics: total parameters reaching the trillion scale (e.g., 2.8 trillion), activated parameters controlled at the hundred-billion scale (around 100 billion), routing expert counts approaching 900, native support for multimodal inputs like vision, and context windows expanded to the million-token level.

However, from an engineering perspective, parameter scale itself is not the evaluation criterion. What truly determines the usability of such models is whether three underlying problems are systematically solved:

1. Computational efficiency of ultra-long contexts: The overhead of standard self-attention grows quadratically with sequence length, making million-token scenarios impossible to either train or infer affordably.
2. Training stability for massive expert MoE: When the number of experts reaches hundreds and each token activates only a tiny fraction, routing can easily become imbalanced, causing training to collapse at any time.
3. Long-horizon reasoning in agentic scenarios: Multi-step tool calling, error recovery, and cross-step state memory require the model to maintain consistent decision quality over extremely long trajectories.

The following sections break down the most representative technical solutions currently used in the industry and summarize reusable engineering experiences around these three main themes.

II. Attention Architecture: Maximizing Matrix Compute for Linear Attention

2.1 Hybrid Attention Becomes the New Consensus

The compute and memory overhead of standard multi-head attention grows quadratically with sequence length, and the KV cache expands linearly in long-context scenarios. Therefore, new-generation models widely adopt a “hybrid attention” strategy:

• A few layers retain global attention, responsible for precise long-range information retrieval.
• A majority of layers use incremental decay-based linear attention operators, replacing the full KV cache with a fixed-size recurrent state to reduce complexity to a linear level.
• On top of this, gated latent attention designs are applied: low-rank compression is performed on key-value states with gated filtering introduced to further reduce cache and memory access overhead.

2.2 Numerical Underflow in Diagonal Blocks: The Overlooked Throughput Bottleneck

For linear attention to be efficient, chunked parallelism is necessary. The inter-chunk part is naturally a matrix multiplication, which fully utilizes the GPU’s matrix acceleration units. However, the intra-chunk (diagonal block) part needs to process the reciprocal of the cumulative decay factor, which easily overflows or underflows under limited precision like BF16. The traditional engineering compromise is to degrade the diagonal block into a position-by-position scalar path, completely bypassing matrix multiplication acceleration—this often becomes a significant source of latency for the entire operator.

2.3 Solution: Adding a “Lower Bound” to the Decay Rate

New-generation solutions parameterize the log decay rate from unbounded Negative-Softplus to a bounded scaled Sigmoid (pseudocode):

# Old: Unbounded parameterization, prone to extreme decay, 
# its reciprocal overflows/underflows under limited precision
log_decay = -softplus(e * z_t)
# New: Bounded parameterization, decay is locked in a safe range [g_min, 0)
g_min = -5.0
log_decay = g_min * sigmoid(e * z_t)

With a fixed lower bound (e.g., g_min = -5), the dynamic range of the reciprocal scaling factor within any 16-token tile is locked in a numerically safe range, allowing the diagonal block computation to run entirely through the matrix multiplication path. The value of this modification lies in the fact that minimal code changes and nearly identical semantics can bring an operator-level throughput leap, making it a classic “low-cost, high-reward” optimization.

2.4 Cross-Layer Information Sharing: Block-Level Attention Residual

Another notable direction is block-level attention residual: sharing the attention output of adjacent layers across layers in a residual format. This yields two benefits:

• It mitigates the representation homogenization problem prevalent in deep Transformers, allowing deep attention layers to receive differentiated fine-grained signals from shallow layers, breaking the representation bottleneck caused by depth.
• Since key-value computation and caching are shared across layers, memory footprint and cross-card communication volume decrease simultaneously, which is particularly friendly for multi-card parallelism in ultra-deep, ultra-wide models.

In practice, sharing is typically organized by “blocks” (e.g., grouping several layers together) rather than globally, to avoid gradient instability caused by excessive coupling.

III. MoE Side: Auxiliary-Loss-Free Dynamic Load Balancing

3.1 New Challenges with Hundreds of Experts

When the number of routing experts approaches 900 and each token activates only a tiny fraction, the flaws of the traditional “auxiliary loss balancing method” are amplified: auxiliary gradients interfere with the main language modeling gradients, and the balancing strength coefficient is extremely difficult to tune—either insufficient balancing leads to expert collapse, or excessive balancing damages model quality. Under extreme sparsity conditions, the tuning cost and training risk of this method are unacceptable.

3.2 Quantile Dynamic Balancing: Turning Balancing from a “Loss Term” into a “Control Loop”

New-generation solutions generally shift towards an auxiliary-loss-free approach: instead of adding a balancing penalty term to the training objective, a real-time control loop is attached to the router (pseudocode):

# Dynamically adjust routing bias each step based on 
# quantile feedback of the expert load distribution
load = expert_load_histogram()              # Track current load per expert
q_low, q_high = quantile(load, [0.1, 0.9])  # Get load quantiles
for e in experts:
    if load[e] > q_high: bias[e] -= alpha   # Decrease weight for overheated experts
    if load[e] < q_low:  bias[e] += alpha   # Increase weight for underutilized experts
router_logits = raw_logits + bias           # Does not pollute main gradient

This dynamic bias adjustment based on quantile statistics has three advantages: the main loss gradient remains pure, free from interference by balancing terms; the balancing response is fast, taking effect intra-step; and it remains stable under extreme sparsity (many experts, very low activation ratio). Combined with shared experts handling general knowledge to reduce routing pressure, and a rational expert parallel placement strategy, the scaled training of massive-expert MoE becomes genuinely feasible in engineering.

3.3 Numerical and Activation Design Details

Ultra-large-scale sparse training is highly sensitive to numerical precision. Besides balancing strategies, attention must be paid to: selecting GLU variants with gating for activation functions paired with proper initialization to suppress activation outliers; introducing temperature and normalization for routing scores to prevent logit drift; and maintaining fine-grained monitoring throughout training to promptly catch expert load skew and loss spikes.

IV. Ultra-Long Context: Curriculum-Based Four-Stage Pre-training

Million-token context is not achieved overnight. The mature industry practice is a “curriculum-based” multi-stage extension, typically following the rhythm of 8K → 64K → 256K → 1M:

1. 8K Stage: Focuses on general corpora to build foundational language capabilities, consuming the vast majority of training compute.
2. 64K/256K Stage: Gradually introduces long documents, code repositories, and multi-document concatenation data, synchronously adjusting positional encoding extrapolation parameters so the model learns cross-window retrieval and aggregation.
3. 1M Stage: Accompanied by a large number of synthetic long-range dependency tasks (multi-hop QA, cross-chapter information flow analysis, repository-level code understanding), turning “can fit” into “can reason”.

Two engineering points are worth emphasizing:

• Architecture First: Only when the linear complexity solution on the attention side is in place can the compute cost of the long-context phase be afforded—this is why attention architecture innovations often precede window expansion.
• Anti-Forgetting: Each stage requires replaying a certain proportion of short-sequence general data, otherwise short-context capabilities will noticeably degrade.

For evaluation, beyond standard “needle in a haystack” retrieval tests, more focus should be placed on real long-range tasks: repository-level code repair, multi-document cross-checking, and long-range state tracking, to avoid the illusion of “adequate retrieval but inadequate reasoning.”

V. Agentic RL: From Conversational Model to Execution System

5.1 Why Agents Must Use Reinforcement Learning

The essence of agentic scenarios is long-horizon decision-making: tool calling, environment observation, failure retries, and plan revisions. Supervised fine-tuning can only imitate “good trajectories” and cannot cover failure recovery and strategy exploration—behaviors “not present in demonstration data.” Reinforcement learning centered on verifiable rewards (such as code execution results or whether tool calls achieve the goal) has become the mainstream path: letting models learn to align with “outcomes” rather than “forms” through real environment feedback.

5.2 Infrastructure Pressure from Million-Token Trajectories

Agent trajectories easily reach hundreds of thousands or even a million tokens (environment states, history, and tool outputs continuously accumulate), imposing hard requirements on RL training infrastructure:

• Inference-Side Throughput: The rollout phase requires high-concurrency sampling services, otherwise training-side GPUs will sit idle.
• Reward Computation Pipeline: Sandbox execution and result verification must be asynchronous and horizontally scalable.
• Curriculum Design: Gradually transition from short tasks and simple tools to long-horizon multi-stage tasks to avoid early training divergence.
• Data Freshness: The generation-consumption latency of on-policy data must be minimized, otherwise the policy will disconnect from the data distribution.

Notably, million-token context capability and Agentic RL mutually reinforce each other: the former allows the model to “fully see” the path it has taken, while the latter teaches the model “how to proceed next.”

VI. Top Three Reusable Technologies and Implementation Recommendations

From a pure engineering reuse perspective, the following three technologies offer the highest “return on investment”:

1. Bounded Decay Parameterization: Any team using decay-based linear attention should check the numerical range of the diagonal block path. The change requires only a few lines of code but yields operator-level throughput improvements. During implementation, re-evaluate the lower bound value based on sequence length and precision schemes.
2. Auxiliary-Loss-Free Quantile Dynamic Balancing: Applicable to all large-scale MoE training. It turns “balancing” from a hyperparameter-sensitive loss term into controllable feedback, making training more stable with less tuning. It is recommended to launch this alongside real-time monitoring and alerting of expert load histograms.
3. Block-Level Attention Residual: Suitable for deep, multi-card parallel models, providing immediate memory and communication benefits. Sharing scope should be organized by blocks, and gradient flows must be thoroughly validated.

For teams planning to conduct such training in the cloud, infrastructure should be evaluated against large-scale cluster standards: heterogeneous compute clusters with high-bandwidth interconnects, high-throughput object storage supporting TB-level checkpoint reads/writes, containerized scheduling platforms oriented towards fault self-healing, and a logging, monitoring, and alerting system covering loss curves, expert loads, and hardware metrics. Cloud platforms such as Baidu AI Cloud have already consolidated these capabilities into general product categories, allowing teams to focus their efforts on algorithms and data. On the inference application side, a hybrid strategy of “retrieval augmentation + long context” is recommended: not all content is worth stuffing into the window; pairing this with KV cache reuse and segmented summarization achieves a balance between effectiveness and cost.

VII. Conclusion

Evaluating new-generation ultra-large-scale MoE multimodal models should not stop at parameter counts and benchmark scores. Instead, it should examine whether they systematically address three questions: how attention maintains linear efficiency under ultra-long sequences, how MoE maintains training stability under extreme sparsity, and how the model achieves reliable long-horizon reasoning in agentic scenarios. Bounded decay parameterization, auxiliary-loss-free quantile balancing, and block-level attention residuals—these “small-incision” technologies correspond to the most painful engineering bottlenecks and are the parts most worth being directly reused by various training teams. Understanding these design motivations and boundary conditions holds more long-term value than remembering the name of any specific model.