Large-Scale MoE Models on Ascend: From Adaptation Challenges to Efficient Training

1. Background: System-Level Challenges of 3-Trillion-Parameter Sparse Models

As large model parameter counts climb into the trillions, sparse expert architectures such as Mixture-of-Experts (MoE) have become the mainstream choice for pushing past compute boundaries. Unlike dense models, these architectures activate only a subset of parameters during inference and training, keeping model capacity high while controlling computational cost. However, deploying ultra-large MoE models is far from simple parameter stacking—the compute, communication, and memory pressures impose systemic requirements on the underlying infrastructure.

From an architectural standpoint, these models typically adopt a hybrid design of “a few shared attention layers plus many parallel expert layers.” One flagship open-source model, for example, has 2.8 trillion total parameters, a stable latent sparse expert architecture with 896 routed experts, and activates only 16 of them per token. This design requires frequent dynamic token-to-expert routing at runtime and introduces two thorny problems: imbalanced expert load can leave some compute units idle, and small per-expert batch sizes can severely underutilize GPU capacity.

The hybrid attention mechanism further increases engineering complexity. Some attention blocks use linear attention to reduce the computational complexity of long sequences, while others retain traditional sparse attention to capture locally critical information. The two mechanisms have vastly different computational characteristics—the former relies on state recurrence and chunked processing, the latter on sparse indexing and non-contiguous memory access—which directly constrains the design space for operator fusion strategies and execution scheduling.

2. Compute Side: Adapting Hybrid Attention Architectures and Optimizing Operators

With KDA and DSA attention interleaved, the most immediate difficulty lies in operator fusion and execution scheduling. KDA linear attention requires recurrent computation of sequence states, a pattern naturally suited to chunked scanning; DSA sparse attention requires non-contiguous memory access based on token sparse indices. The mix of these two patterns defeats conventional operator fusion strategies—no single kernel can efficiently handle both memory access patterns at once.

A practical approach is to split the attention layer into two stages: state update and sparse aggregation. In the state update stage, the high bandwidth of super-node architectures is used to map KDA recurrence onto multiple compute units in parallel, with pipelining to hide data dependency latency. In the sparse aggregation stage, pre-sorting and reordering techniques improve the locality of non-contiguous access, converting random access into near-sequential access and thereby raising cache hit rates.

For expert layers, operator optimization must also account for dynamic batch size variation. Since the number of tokens each expert receives differs at every training step, static operator configurations are inadequate. One viable approach is multi-batch bucketing—grouping experts with similar token counts and applying a uniform batch size for matrix multiplication within each group, effectively improving compute utilization.

3. Communication Side: Load Balancing and Routing Optimization Under Expert Parallelism

Compared with dense models, MoE models depend exponentially more on the communication network. After each expert layer, tokens must be dispatched to different compute nodes according to routing results, generating substantial All-to-All communication. With 896 experts, communication overhead can even exceed computation itself.

More problematic is the hot-expert problem introduced by sparse routing. During large-scale training, data distribution characteristics cause some general-purpose experts to receive far more tokens than average, while some specialized experts may remain underloaded for long periods. This imbalance not only reduces overall throughput but also causes memory fragmentation and overloads specific nodes.

Engineering practice typically combines three countermeasures:

• Routing-aware token dispatch: introducing auxiliary load-balancing loss during routing computation, constraining the entropy of the routing distribution to avoid extreme imbalance.
• Dynamic expert replication: automatically creating multiple replicas of persistently hot experts across nodes, while merging parameters of long-idle experts to reduce memory footprint.
• Communication-compute overlap: using separate communication and compute streams to prefetch routing information for the next batch of tokens during expert computation, and leveraging the low-latency interconnect of super-node architectures to hide communication time within computation.

Notably, the choice of expert parallelism degree significantly affects communication efficiency. A very high expert parallelism degree (e.g., EP64) reduces per-card memory pressure but increases the number of destination nodes per token and raises communication frequency. Joint tuning across network topology, intra-node bandwidth, and model scale is needed to find the balance between computation and communication.

4. Memory Side: Distributed Deployment Strategies for 2.8T Parameters

Memory management is another key bottleneck in ultra-large model training. At BF16 precision alone, 2.8 trillion parameters occupy roughly 5.6 TB of memory. Even on 8-GPU nodes, a single node cannot hold all parameters, let alone reserve space for optimizer states, gradients, and activations.

The mainstream approach combines tensor parallelism with expert parallelism. Tensor parallelism splits single-layer parameters across GPUs within a node along matrix dimensions, using NVLink-level bandwidth for fast synchronization; expert parallelism places different experts entirely on different nodes, using the cluster network for cross-node communication. The difficulty of this hybrid parallelism lies in coordinating the two levels: tensor parallelism degree should match the number of GPUs per node, while expert parallelism degree must balance communication efficiency against memory redundancy.

In practice, the combination of FSDP2 and EP has proven effective. FSDP2 (Fully Sharded Data Parallel) fully shards parameters and gradients, restoring them via AllGather only when needed; expert parallelism provides cross-node distribution and aggregation of expert parameters. Together, each GPU holds only a shard of model parameters, and FSDP’s gradient sharding reduces redundant storage of optimizer states.

In LoRA (low-rank adaptation) fine-tuning scenarios, memory pressure is further relieved because most weights are frozen and only the newly added low-rank matrices are trained. This makes it possible to fine-tune a 2.8T model on 128 GPUs without additional gradient checkpointing or offloading strategies.

5. Automated Adaptation: Engineering Practice Toward Day-Scale Onboarding

Given rapidly evolving model architectures, manually writing adaptation code can no longer keep pace with iteration. The industry is exploring agent-driven automated development workflows that delegate customization steps—operator mapping, parallelism strategy selection, communication pattern recognition—to automated tools.

Specifically, by parsing the model’s computation graph, subgraphs matching the expert-routing plus MoE pattern can be automatically identified, and corresponding communication primitives and scheduling logic generated. Historical training performance data can also train machine learning models to predict throughput under different parallel configurations, enabling better parallel plans at initial deployment. This automated pipeline shortens onboarding time for new models from weeks to days, significantly improving infrastructure adaptability.

6. Practical Path from Adaptation to Optimization

Overall, engineering ultra-large MoE models is not a matter of stacking a single technology stack but a global trade-off among computation, communication, and memory. For teams planning to deploy such models, the following path is worth considering:

1. Establish the communication topology first: when designing cluster networks, focus on the ratio of intra-node bandwidth to spine-layer bandwidth to avoid idle compute caused by communication bottlenecks.
2. Adopt hierarchical parallelism: use tensor parallelism within nodes, expert parallelism across nodes, and pipeline parallelism to handle inter-layer dependencies, achieving three-dimensional parallel coordination.
3. Build dynamic tuning mechanisms: continuously monitor per-expert token distribution, communication time, and memory usage during training, and dynamically adjust routing strategies or parallelism configurations.
4. Establish an automated adaptation pipeline: invest in model parsing and automatic parallel planning tools to reduce onboarding time for new models.

Today, LoRA fine-tuning of trillion-parameter models at FP8 precision on 128-GPU clusters has been validated, but higher-order full pretraining still requires cluster scheduling at the thousand-GPU or even ten-thousand-GPU scale. As super-node architectures continue to evolve and agent-driven compilation optimization matures, the engineering barrier for large models will continue to fall, providing a solid foundation for frontier applications such as multimodal understanding and complex agent tasks.