Scaling a Trillion-Parameter Open-Source Model: SuperNode Adaptation and Inference Optimization

The Systemic Challenges of Trillion-Parameter Models

When the parameter count of an open-source large language model jumps to 2.8 trillion, the technical complexity of deployment goes far beyond simple software installation. This model employs an ultra-large-scale sparse Mixture-of-Experts (MoE) architecture, innovatively integrating KDA linear attention with DSA sparse attention, while also supporting Multi-Token Prediction (MTP). Such a highly complex architecture significantly raises the model’s capability ceiling, but it also imposes three severe demands on the underlying compute infrastructure.

The first challenge is the non-linear expansion of memory capacity. Even when stored in the industry-standard BF16 precision, the static weights of 2.8 trillion parameters exceed 5.6 TB. This means a single accelerator card, or even a single server, cannot hold a complete copy of the weights. It necessitates multi-card parallelism and memory pooling technologies to shard the model weights across the entire compute cluster. The second challenge is the bandwidth bottleneck of high-speed Scale-Up interconnects. In an MoE architecture, each token must be dynamically routed to multiple expert networks, a process accompanied by frequent All-to-All communication. If the intra-node interconnect bandwidth is insufficient, communication latency will directly negate compute gains, leading to a situation where “the faster you compute, the longer you wait.” The third requirement is the triple demand of production inference: high throughput, low latency, and high concurrency. Production environments require processing as many requests per unit time as possible to amortize costs, while maintaining low Time-to-First-Token and average token latency, and also handling sudden traffic spikes.

These challenges cannot be solved by hardware upgrades alone. They require a top-down, system-level view and a co-design of hardware and software. This is the core value of the “SuperNode” concept: by packaging multiple accelerator chips, large-capacity high-bandwidth memory, and ultra-high-speed interconnects into a single logical compute unit, it provides a physical “compute chassis” for trillion-parameter models.

Architectural Advantages and Adaptation Focus of the SuperNode Compute Chassis

To address the model characteristics mentioned above, a common industry solution is to build high-cardinality SuperNode clusters. These clusters typically use high-bandwidth, low-latency Scale-Up networks to tightly interconnect multiple accelerator cards, creating a form akin to a “super-GPU.” In the practice discussed here, the SuperNode offers two core advantages. First, its massive aggregate memory can hold a complete copy of the 2.8-trillion-parameter weights, avoiding the extra communication overhead of cross-node access. Second, its high-throughput Scale-Up interconnect can handle the instantaneous traffic bursts generated by MoE models during token dispatch and expert parallelism.

However, the hardware foundation is only a prerequisite. The real challenge lies in “Day-0” adaptation of the software stack. Day-0 adaptation means that at the moment (or even before) a model is open-sourced, the inference engine, scheduling framework, and underlying operators must be jointly integrated and optimized, ensuring that users can achieve “plug-and-play” high-performance deployment on the SuperNode once they obtain the model weights. This process is far more complex than editing a config file; it involves meticulous memory planning, communication-computation overlap strategy design, and kernel-level manual optimization.

Specifically, the core of the adaptation work is to eliminate three types of loss. The first is memory fragmentation loss. This requires optimizing memory pooling and weight sharding algorithms to precisely control the peak memory usage of static weights, KV Cache, and activations within the physical memory limit. The second is communication wait time. This entails using computation-communication overlap techniques so that accelerators can continue executing ready attention or feed-forward network computations while waiting for All-to-All data to return. The third is kernel invocation overhead. Core operators frequently called in the MoE architecture must be deeply customized against the underlying instruction set to avoid the performance penalty of overly generalized library functions.

Core Kernel Optimization: The Key Battle for Performance

In the inference path of a trillion-parameter MoE model, the execution efficiency of several kernels directly determines the end-to-end performance ceiling. Instruction-level tuning of these specific kernels was the top priority in this adaptation effort.

The first key kernel is FlashMLA (Multi-head Linear Attention). This kernel serves the KDA linear attention mechanism, handling global information aggregation over ultra-long sequences. Unlike traditional softmax attention, linear attention uses kernel tricks to reduce computational complexity from O(N²) to O(N). However, on actual silicon, this mathematical simplification must be translated into highly efficient memory access patterns. The optimization focus was on blocking reads of the Q, K, and V matrices and reusing on-chip cache to avoid repeated accesses to global memory. Concurrently, for the cumulative nature of linear attention, efficient vectorized accumulation instruction sequences were designed to reduce instruction issue overhead. After tuning, both memory bandwidth utilization and compute unit utilization were significantly improved in long-sequence scenarios.

The second core kernel is the DeepGEMM MOE operator. In MoE layers, tokens are routed to different expert networks for matrix multiplication. Because the number of tokens each expert receives is imbalanced (following a long-tail distribution), traditional GEMM implementations result in numerous inefficient small matrix multiplications. The optimization strategy involved adopting Grouped GEMM technology, which merges matrix multiplications with similar dimensions into a single group and uses fine-grained task scheduling for load balancing. Furthermore, by modifying the data blocking strategy of matrix multiplication (e.g., aligning from the M dimension to the K dimension), the L2 cache miss rate was reduced, allowing SIMD units to operate more fully.

The third critical component involves adapting MegaMoE and MoonEP. These two components handle large-scale expert parallelism strategies and efficient token dispatch, respectively. The core challenge for MegaMoE lies in the dynamic loading and unloading of expert containers. Due to limited memory, not all experts can reside in memory permanently. Based on routing statistics, experts must be intelligently swapped between disk and memory. MoonEP focuses on optimizing the implementation of All-to-All communication. By analyzing the compute-communication topology, multiple small data transfers are merged into a few large block transfers. This, coupled with RDMA technology that bypasses kernel protocol stack overhead, significantly reduces communication latency. Additionally, given the potential for uneven routing to create a hot expert, a load-balancing strategy at the communication granularity was designed to prevent individual compute nodes from becoming bottlenecks.

Service-Oriented Deployment: The Last Mile from Weights to a Production-Ready API

Deploying the model on a SuperNode is only the first step. Transforming it into a stable, high-availability external API service requires solving engineering challenges. In this practice, the model has been simultaneously listed on a mainstream AI development platform, providing standardized API services. This relies on multi-layered optimizations within the serving framework.

The foremost priority is batch strategy optimization. To meet high concurrency requirements, the inference service must adopt a Continuous Batching mechanism. Unlike traditional static batching, continuous batching allows requests to dynamically join or leave the current iteration batch. Since GPU utilization is directly related to batch size, this mechanism ensures the accelerator cards remain under consistently high load. Secondly, to handle contexts as long as 1 million tokens, a Prefix Caching mechanism is necessary. When multiple requests share the same system prompt or user question, the system automatically reuses the previously computed KV Cache for that portion, avoiding redundant computation. Thirdly, elastic scaling strategies are crucial. By monitoring the number of queued requests and the average token generation speed, the system can automatically scale compute resources up or down within predefined thresholds.

For enterprise users, leveraging bare-metal servers or managed inference engine services provided by cloud vendors can further simplify this deployment process. Through pre-configured container images and orchestration scripts, users can quickly obtain a performance-tuned inference environment without worrying about the underlying hardware topology or communication library configurations. This allows them to focus on higher-level business logic and model application innovation.

Conclusion and Outlook

This Day-0 adaptation effort for a trillion-parameter open-source model fully demonstrates that in the era of ultra-large-scale models, hardware-software co-design and deep kernel optimization are indispensable moats. Simply “running” the model is no longer difficult. However, achieving high throughput and low latency in a high-concurrency production environment requires a trinity of deep integration and tuning across the model architecture, chip characteristics, and inference framework. As the parameter scale of open-source models continues to rise, this pattern of joint adaptation starting from the source will gradually become the new normal for large-model infrastructure delivery.