Adapting Ultra-Large-Scale Sparse MoE Models on Domestic Supernodes

As open-weight models enter the trillion-parameter era, the demands of model training and fine-tuning have shifted from single-GPU performance to system-level synergy. Recently, a next-generation native multimodal model with 2.8 trillion total parameters was open-sourced. A domestic supernode platform promptly completed its adaptation, validating the feasibility of LoRA fine-tuning on a 1,000-GPU scale cluster. Starting from the model’s architectural characteristics, this article dissects the engineering challenges of ultra-large-scale sparse MoE models across three dimensions—computation, communication, and memory—and outlines the corresponding adaptation strategies to provide a reusable reference for developers building large model infrastructure.

1. Model Architecture: Sparse Experts and Hybrid Attention

This flagship model utilizes the Stable LatentMoE sparse expert architecture, featuring a total parameter scale of 2.8 trillion. It contains 896 routed experts, with each token activating only 16. This design maintains model capacity while keeping per-token computational overhead manageable, but it imposes stricter requirements on the distributed deployment and routing of expert parameters.

For attention, the model adopts a hybrid architecture: approximately three-quarters of the attention blocks use linear attention combined with a Top-2048 sparse attention mechanism. This supports a context length of up to 1 million tokens and provides native image understanding capabilities. The core of linear attention relies on state recursion and chunked computation—the former accumulates hidden states progressively along the sequence dimension, while the latter splits long sequences into chunks to enhance parallelism. Sparse attention, on the other hand, depends on index filtering and non-contiguous memory access. The interleaving of these two mechanisms significantly increases the complexity of operator fusion and execution scheduling.

In terms of capabilities, driven by large-scale joint reinforcement learning and system-level optimization, this category of model demonstrates significant improvements in scenarios like long-range autonomous programming, complex agent tasks, 3D content creation, financial research, and deep knowledge work. These scenarios share common traits: long sequences, numerous interaction rounds, and high demands for contextual consistency, which align perfectly with the design intent of the aforementioned attention architecture.

2. Triple Challenges: Coupled Constraints in Computation, Communication, and Memory

2.1 Computation: Increased Complexity in Operator Fusion and Scheduling

The state recursion in linear attention inherently creates sequence dependencies. While chunked computation mitigates the lack of parallelism, it introduces overhead for passing states between chunks. Index filtering in sparse attention leads to non-contiguous memory access patterns, making it difficult to optimize memory access efficiency through simple contiguous prefetching. When these two types of operators are stacked within the same network layer, executing them in separated phases generates massive read/write overhead for intermediate results and kernel launch overhead.

The engineering solution is to implement large forward and large backward fused operators: integrating multiple computation phases into a single kernel to reduce the writing and repeated loading of intermediate tensors. For example, merging the state update, chunk normalization, and output projection of linear attention allows a single kernel launch to accomplish work that previously required multiple scheduling rounds. The trade-off of fusion is increased register pressure and implementation complexity, requiring meticulous tuning tailored to the specific hardware’s data paths.

2.2 Communication: Load Imbalance in Sparse Routing

Sparse routing across 896 experts is the primary source of communication bottlenecks. Because routing results depend on the semantic features of input tokens, expert token distribution can easily become uneven during runtime: some experts handle far more tokens than the average, resulting in small and highly volatile per-expert computation batches. This leads to two consequences: devices hosting “hot” experts become pipeline bottlenecks, while devices hosting “cold” experts sit idle. Simultaneously, the data volume in All-to-All communication fluctuates dynamically with routing distribution, making static communication planning ineffective.

Common industry practices include introducing capacity factors with token dropping/rearrangement strategies to smooth the distribution, employing hierarchical All-to-All for expert parallelism to reduce single communication volumes, and implementing lightweight load prediction at the routing layer to adjust groupings in advance. These methods must be co-designed with the parallel architecture rather than optimized in isolation.

2.3 Memory: The Hard Constraint of Weight Volume

Storing 2.8 trillion parameters in BF16 precision alone requires approximately 5.6 TB. This volume far exceeds the memory capacity of a single machine, necessitating distributed partitioning. Furthermore, while LoRA fine-tuning only updates low-rank adapters, the forward pass still requires the full model weights. Therefore, the focus of memory optimization lies in the sharded placement and on-demand aggregation of weights, rather than reducing the total weight size.

3. Adaptation Strategy: Hybrid Parallelism and Full-Link Optimization

3.1 FSDP2 + EP Hybrid Parallel Architecture

To address the constraints above, the adaptation strategy employs a hybrid parallel architecture combining FSDP2 and Expert Parallelism (EP). FSDP2 handles the sharding and gradient synchronization of non-expert parameters (such as attention and embedding layers), reducing per-device memory usage through parameter sharding. EP handles the distributed deployment of the 896 experts across devices. Each device holds a subset of experts; tokens are routed via All-to-All to the target expert’s device for computation and then scattered back to the original device.

The key to this combination is ensuring the split ratio between the two parallel dimensions matches the cluster scale. On a 128-GPU supernode, the validated configuration overlays FSDP2 with EP64—meaning the expert dimension is split 64 ways, and the remaining dimensions are handled by FSDP2. In practice, the partitioning strategy should be determined jointly by the number of experts, per-GPU memory, and interconnect bandwidth: too few EP partitions results in too many experts per GPU and memory strain; too many increases All-to-All communication hops and latency.

3.2 Memory Management: Sharded Placement and Efficient Aggregation

Under the FSDP2+EP architecture, memory management must solve two problems: the sharding granularity of expert parameters and the on-demand aggregation of parameters during the forward pass. After expert parameters are partitioned along the EP dimension, a single device retains only a subset of expert weights. Aggregation occurs when a token’s routing hits an expert, completed via communication rather than full weight loading. For non-expert parameters, FSDP2 performs an All-Gather before computation and a Reduce-Scatter after, forming a shard-aggregate-reshard cycle. In engineering implementation, aggregation operations should overlap with preceding computations as much as possible to hide latency through communication.

3.3 Communication Optimization: Tuning for Sparse Routing

The core of communication optimization is reducing ineffective traffic in All-to-All operations. Actionable measures include: bucket sorting routing results so tokens destined for the same device are arranged contiguously, improving transmission efficiency; setting capacity limits for hot experts, rerouting excess tokens to sub-optimal experts to avoid single-point congestion; and leveraging the unified memory addressing capabilities within the supernode to reduce protocol overhead for cross-node communication. The supernode architecture itself, characterized by high bandwidth, low latency, and unified memory addressing, provides the physical foundation for these optimizations.

3.4 Operator Acceleration: Fused Implementation for Linear Attention

To address the state recursion and chunked computation of linear attention, the adaptation strategy designed fused forward and backward operators. The pseudo-code is as follows:

# Fused operator illustration: state update + chunk normalization + output projection
def fused_linear_attn_forward(x, state, block_size):
    outputs = []
    for blk in split_blocks(x, block_size):
        state = update_state(state, blk)      # state recursion
        normed = block_normalize(blk, state)  # chunk normalization
        outputs.append(project(normed))       # output projection
    return concat(outputs), state

After fusion, the original three kernel launches are reduced to one, and intermediate tensors are no longer written back to global memory. Backward propagation is handled similarly, merging gradient computation with state backpropagation to reduce redundant memory access. It is worth noting that the block size of the fused operator must match the hardware cache capacity: blocks that are too large cause register overflow, while blocks that are too small lack sufficient parallelism. The optimal value typically requires empirical testing.

4. Engineering Workflow: Agent-Driven Day-Level Integration

Model adaptation is not just a technical issue of operators and parallelism; it is also a matter of engineering efficiency. Traditionally, integrating a new model often requires manual analysis of computation graphs, layer-by-layer operator semantic comparison, and manual parallel configuration adjustments, taking weeks. The adaptation strategy introduces an agent-driven automated development workflow. Tasks such as model structure parsing, operator mapping, and parallel strategy searching are handled by automated tools, compressing the integration cycle for new models to days and supporting rapid iteration.

The general workflow operates as follows: parse the model’s computation graph and weight structure, identify unsupported operators and match them with existing implementations or generate fusion plans, automatically search for parallel partitioning strategies based on cluster scale, and finally perform correctness validation on a small-scale cluster before scaling up to the full GPU count. For developers, this shifts the focus of adaptation work from “manual per-operator debugging” to “strategy configuration and result verification.”

5. Validation and Future Outlook

Currently, the adaptation strategy has completed LoRA fine-tuning validation using FSDP2+EP64 on a 128-GPU supernode, covering the full link of forward computation, backward propagation, and parameter updates. The validation focused on: the numerical correctness of fused operators, the distribution balance of sparse routing under real workloads, and whether memory usage and communication overhead under hybrid parallelism fall within expected ranges.

Looking ahead, the adaptation of trillion-scale sparse MoE models will trend in two directions. First, more aggressive operator-level fusion, merging portions of attention, routing, and feed-forward network stages to further compress kernel scheduling overhead. Second, the automation of parallel strategies, dynamically generating partitioning plans based on model structure and cluster topology to reduce manual trial and error. For infrastructure developers, understanding the correspondence between model architecture and hardware characteristics is more critical than memorizing a specific configuration—architectures change, but constraints remain: computation requires fusion, communication requires balance, and memory requires sharding. These three principles are the constant propositions for landing ultra-large-scale models.