Supernode Architecture and Inference Optimization: Deploying Trillion-Parameter Open-Source Models

From Exploding Model Scale to Reinvented Infrastructure

Recent years have shown two clear trends in large language model development. First, total parameter counts have grown far faster than Moore’s Law, climbing from hundreds of billions to trillions and heading toward tens of trillions. Second, model architectures have evolved from early dense Transformers to very large sparse Mixture-of-Experts (MoE) designs that keep inference quality high while controlling compute cost. Once total parameters reach the 2.8-trillion scale, a single multi-GPU server or a traditional data center network can no longer meet basic training and inference requirements — and that is precisely what gave rise to the “supernode,” a new form of compute infrastructure.

A supernode is a compute unit that physically aggregates a large number of AI accelerators (GPUs or dedicated AI chips) and links them through an ultra-high-bandwidth, ultra-low-latency Scale-Up network, so that the entire system behaves logically like one giant computer. Unlike the Scale-Out networks of conventional data centers, which are typically built on 10GbE or slower Ethernet, Scale-Up interconnects inside a supernode usually deliver hundreds of GB/s or even TB/s of bandwidth, with latency in the microsecond or sub-microsecond range — enough to support communication-sensitive parallel strategies such as tensor parallelism and expert parallelism.

Models like Kimi K3.1 Pro are distinctive in several ways: 2.8 trillion parameters, an MoE architecture, a hybrid attention design that combines KDA linear attention with DSA sparse attention, and support for Multi-Token Prediction (MTP). During inference, the model must not only load all expert weights — since any token may be routed to any expert — but also keep the latency between attention computation and expert-routing communication extremely low. As for memory, even at BF16 precision, 2.8 trillion parameters require roughly 5.6 TB of accelerator memory. That far exceeds the ~640 GB capacity of a single 8-GPU server (assuming 80 GB per card) and also surpasses the aggregated memory of small clusters such as 2-node/8-GPU or 4-node/32-GPU setups. A supernode must therefore provide at least tens of terabytes of aggregated memory, with high-speed interconnects allowing every accelerator to access remote memory as efficiently as local memory.

The Three-Layer Technology Stack for Supernode Adaptation

Day 0 adaptation of trillion-parameter MoE models typically involves three layers of technology working together. The first is the hardware layer: accelerator compute and memory specifications, the topology and bandwidth of the Scale-Up interconnect, and rack-level power and cooling design. The second is the system software layer: deep learning compilers, operator libraries, communication libraries, and the scheduling engine for parallel strategies. The third is the inference framework layer: how mainstream serving frameworks such as vLLM and SGLang implement the model graph, KV Cache management, continuous batching, and mechanisms like PagedAttention.

At the hardware layer, the central challenge is balancing memory capacity against aggregated bandwidth. For a 2.8-trillion-parameter MoE model, the ideal is to keep all parameters resident in accelerator memory to avoid the latency of loading weights from external storage. Because MoE models activate only a subset of experts in each forward pass (for example, Top-2 routing), expert load across accelerators is dynamically imbalanced, so the volume of intermediate activations exchanged across nodes is enormous. The FlashMLA operator, for instance, optimizes multi-head attention on H-series architectures and relies on the supernode’s high-bandwidth interconnect to reduce KV Cache transfer overhead. The DeepGEMM MOE operator focuses on efficient grouped matrix multiplication for experts: by partitioning, quantizing, and fusing expert weight matrices, it significantly reduces memory accesses and increases compute density.

At the system software layer, deep learning compilers — such as the general-purpose compiler capabilities represented by the T-Head SAIL software stack — act as the bridge. The compiler must map high-level model definitions onto low-level hardware instruction sets and automatically generate optimal communication primitive schedules based on the hardware topology. The key here is automatic search over parallel strategies: for a 2.8-trillion-parameter model, combinations of tensor parallelism (TP), expert parallelism (EP), pipeline parallelism (PP), and data parallelism (DP) must be evaluated jointly. Consider EP alone: if all experts are distributed across 64 accelerators and each token routes to 2 experts, this produces 64-way All-to-All communication. If the Scale-Up topology is fully connected or near-fully connected, latency remains manageable; but with a ring or tree topology, communication costs rise sharply, forcing the compiler to redesign communication scheduling to avoid hotspots.

Adaptation at the inference framework layer is even more granular. In vLLM and SGLang, the widely used continuous batching and PagedAttention mechanisms require redesigned KV Cache allocation and reclamation strategies in multi-accelerator supernode environments. With trillion-parameter models, a 1-million-token context window means a single request’s KV Cache can reach tens of GB or more, which depends on sharded KV Cache management under tensor parallelism. Meanwhile, SGLang’s RadixAttention (prefix cache reuse) must be extended in a distributed manner for supernode scenarios, keeping cache indices consistent across accelerators with no redundant storage.

From Day 0 to High Concurrency: The Full-Chain Validation Path

The core goal of Day 0 adaptation is to complete full deployment and validation of the inference service on the supernode as soon as new model weights are released. Validation typically covers three areas: functional correctness (token-by-token output comparison against a reference implementation), performance targets (throughput, time to first token, and inter-token latency), and stability (72+ hours of continuous operation, watching for memory leaks, communication timeouts, and similar anomalies).

In practice, fine-grained operator-level tuning is the foundation of hitting performance targets. Take MegaMoE and MoonEP as examples (generic MoE inference communication approaches). Their core optimizations include: first, precomputing and caching expert routing tables to avoid the overhead of repeatedly computing routing probabilities online; second, topology-aware scheduling of All-to-All communication, dynamically adjusting communication buffer sizes and send/receive order based on how tokens are distributed between local and remote experts; and third, micro-batching strategies that split large requests into multiple micro-batches according to expert load-balancing principles, dispatching them to different accelerators to raise overall throughput.

Beyond optimizing the inference path itself, the choice of deployment model matters. The mainstream options today are direct deployment on bare-metal servers and containerized deployment through managed intelligent engine services. Bare-metal deployment suits scenarios with extreme performance requirements that call for deep kernel customization (NUMA binding, huge pages, RoCE congestion control parameters). Containerized deployment emphasizes elasticity and operational automation: an intelligent scheduler places different inference instances (for example, replicas of different models) onto different resource slices of the supernode, enabling multi-tenant isolation and resource reuse.

It is also worth noting that newly launched supernode regions expand large-scale compute supply. Regions with abundant power and cool climates can operate very large data centers at a lower total cost of ownership (TCO). For enterprise users, deploying inference services in a supernode region provides access to memory pools of hundreds of terabytes, and makes it easier to form a data-locality loop with other AI services in the same region — such as data preprocessing and model fine-tuning — reducing the cost and latency of cross-region data transfer.

Performance Bottlenecks and Future Directions

Even after Day 0 adaptation is complete, trillion-parameter models face multiple constraints in high-concurrency production environments. The first is the memory wall: keeping all 2.8 trillion parameters resident in accelerator memory is feasible, but it leaves limited headroom for the KV Cache. As concurrent requests grow, KV Cache usage spikes and may trigger swapping to CPU memory or NVMe SSDs, causing latency to soar. The common industry response combines FP8/INT8 quantization to compress expert weights (cutting memory usage by more than 50%), reserving dedicated high-bandwidth memory partitions for the KV Cache, and introducing streaming loading — temporarily migrating the weights of inactive experts to CPU memory and prefetching them when routing predicts they are about to be activated.

Another bottleneck is congestion control on the Scale-Up network. In MoE models, the All-to-All communication pattern readily causes many-to-many network congestion, especially when expert load is highly imbalanced — for example, when a few popular experts receive the vast majority of routed tokens. Current mainstream countermeasures include load-aware expert placement (replicating hot experts across multiple accelerators), communication-compute overlap (running local attention computation while waiting for remote expert results), and priority scheduling (assigning higher communication priority to latency-sensitive early tokens).

Looking ahead, as model scale moves toward tens of trillions of parameters, the supernode may evolve from a single physical rack into cross-rack clusters connected by optical interconnects. As optical circuit switching (OCS) matures, the interconnect topology among thousands of accelerator cards could be dynamically reconfigured, flexibly adjusting bandwidth allocation to match different model workload characteristics. Innovation in model architecture itself will also shape infrastructure design: for instance, if linear attention fully replaces sparse attention in the future, KV Cache capacity requirements will drop substantially, but All-to-All communication patterns will change, and supernodes may need to focus more on activation compression than on cache management.

For developers and enterprise users entering the trillion-parameter era, building engineering capabilities around supernode infrastructure is essential. It is not simply a matter of “renting more GPUs” — it requires understanding how model parallel strategies map onto hardware topology, mastering operator library tuning beneath the inference framework, and using the intelligent engine services offered by cloud providers for elastic scaling and fault self-healing. As the Day 0 adaptation of Kimi K3.1 Pro illustrates, a mature technology stack requires coordinated layers: hardware (sufficient memory and interconnect bandwidth), system software (compilers and communication libraries), frameworks (inference engines), and platform (resource scheduling, monitoring, and alerting). Only with this alignment can trillion-parameter open-source models deliver high-concurrency, low-latency AI services quickly, stably, and cost-effectively — on-premises or in the cloud.