Deploying Trillion-Parameter Sparse LLMs on Ultra Nodes: Inference Best Practices

1. Background: Deployment Challenges of Massive Parameters and Extended Contexts

In recent years, Large Language Model (LLM) development has followed two parallel trajectories. First, parameter scales have continuously expanded from hundreds of billions to the trillion-parameter level. Second, context windows have grown from a few thousand tokens to the million-token range. The intersection of these trends significantly increases the engineering complexity of inference deployment.

Regarding parameter scale, when a model reaches the trillion-parameter level, the number of actually activated parameters remains substantial even when utilizing a Sparse Mixture-of-Experts (MoE) architecture. Compared to dense models, sparse models only activate a subset of experts during a single forward pass, theoretically reducing computational load. However, this comes at the cost of a much larger total weight footprint, placing new demands on GPU memory capacity, weight loading speed, and inter-device communication bandwidth.

For context length, a million-token window means the sequence length for attention calculations multiplies exponentially. The computational complexity of traditional attention mechanisms grows quadratically with sequence length. Without alternative solutions like linear or sparse attention, the time and memory overhead for long-sequence inference becomes unacceptable. Therefore, algorithmic and operator optimizations for long contexts must progress in tandem.

Additionally, new features like multi-token prediction further alter the computational graph structure of the inference process. Traditional autoregressive generation predicts one token at a time, whereas multi-token prediction simultaneously outputs multiple candidate tokens in a single forward pass. While this improves decoding throughput, it introduces stricter requirements for cache management, sampling logic, and batch scheduling.

These compounding factors mean that simply adding hardware resources does not directly yield optimal inference performance. The key to unlocking hardware capabilities lies in designing rational parallel strategies, selecting compatible inference frameworks, tuning critical operators, and streamlining the end-to-end engineering pipeline from model weights to inference services.

2. Ultra Nodes as the Hardware Foundation for Trillion-Parameter Inference

An Ultra Node can be understood as a tightly coupled collection of compute nodes connected via high-speed interconnects. Compared to standard distributed clusters, an Ultra Node provides significantly higher Scale-Up interconnect bandwidth and lower communication latency within the node. This hardware architecture is crucial for trillion-parameter sparse models, as Sparse MoE models frequently exchange hidden states and expert outputs across different devices during inference.

In traditional Scale-Out architectures, cross-node communication typically relies on networking with higher costs and greater latency. Ultra Nodes, however, aggregate multiple accelerator cards into a larger logical unit via high-speed interconnects, effectively controlling the communication overhead of model and tensor parallelism. For trillion-parameter models, each weight shard post-splitting may be accessed by multiple compute steps. The high intra-node bandwidth ensures that weight reading and gradient synchronization (if in the training phase) do not become primary bottlenecks.

The value of Ultra Nodes in inference deployment is also evident in memory aggregation. The GPU memory capacity of a single card is limited, making it incapable of holding the entire weights of a trillion-parameter model even with quantization. By pooling the memory of multiple cards within an Ultra Node, model weights can be distributed and stored across several accelerators, reading on demand during computation. This approach requires interconnect links with sufficient bandwidth and low enough latency to make remote weight reads perform nearly identically to local accesses.

In practice, Ultra Nodes usually feature a unified software stack and resource scheduling capabilities. For developers, this means the entire Ultra Node can be requested and utilized as a single large compute unit, eliminating the need to manually manage the topology and communication details of each accelerator card. This abstraction lowers the engineering barrier for large-scale model deployment, allowing teams to focus more on model tuning.

3. Parallel Strategy Design for Trillion-Parameter Sparse Models

When deploying trillion-parameter sparse models, the choice of parallel strategy directly impacts end-to-end performance. Common parallelism methods include data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism (EP). Each has its applicable scenarios, and real-world deployments often require combining them.

Data parallelism is the most basic form, where each card holds a complete model replica and only the input data is partitioned. For trillion-parameter models, a single card cannot hold the entire model weights, so data parallelism can only serve as an outer-layer scaling mechanism and cannot be used independently.

Tensor parallelism splits the weight matrices within a single layer across multiple cards. After each card computes its partial results, they are merged via communication. This method involves heavy communication traffic, making it suitable for the high-bandwidth interconnect environment within an Ultra Node. For matrix multiplications in attention and feed-forward layers, tensor parallelism effectively reduces single-card memory usage but is sensitive to communication topology. Restricting tensor parallelism to within the Ultra Node minimizes latency impacts.

Pipeline parallelism slices the model by layers across different devices, with each device handling a subset of layers. This approach has relatively low communication volume but can easily lead to device idle time. Micro-batch scheduling can improve utilization to some extent. For large models with numerous layers, pipeline parallelism is an effective means of distributing weight pressure.

Expert parallelism is unique to sparse MoE models. Because different experts are distributed across various devices, each token is dispatched to the corresponding expert device based on routing results. When the number of experts far exceeds what a single card can hold, expert parallelism is inevitable. The crux lies in expert load balancing and communication scheduling: if certain experts are frequently hit while others remain idle, hotspots form, dragging down the entire batch. Therefore, statistical analysis of the routing distribution is necessary before deployment, and expert placement strategies should be adjusted where possible.

In actual deployment, the standard practice is to use tensor parallelism and expert parallelism within the Ultra Node, pipeline parallelism for cross-node scaling, and data parallelism for outer-layer replica expansion. The combination of parallel strategies must be comprehensively evaluated based on model structure, hardware topology, and inference load characteristics. There is no universally optimal configuration.

4. Long-Context Attention Mechanisms and Memory Management

The direct challenge of million-token contexts stems from attention computation. Standard self-attention calculates the correlation between every pair of tokens in a sequence, and its memory footprint grows quadratically with sequence length. For million-token scales, even using half-precision, the memory required just for the attention matrix far exceeds single-card capacity. Therefore, more efficient attention mechanisms must be introduced.

One category of solutions is linear attention, which reformulates attention computation into a kernel function form, reducing computational complexity from quadratic to linear. Its core idea leverages the associativity of matrix multiplication to change the computation order, avoiding the explicit construction of a full attention score matrix. Linear attention significantly reduces computational and memory overhead in long-sequence scenarios, but its impact on task accuracy requires careful evaluation.

Another category is sparse attention, which does not compute attention scores for all token pairs, but only selects a subset of important ones. Selection strategies can be based on local windows, strided sampling, or dynamic routing. Sparse attention can serve as a complement to linear attention, striking a balance between efficiency and accuracy. The basic idea of hybrid attention design is to combine linear and sparse attention across different layers or heads, accommodating both global perception and local precision.

Memory management is equally critical in long-context inference. The KV cache stores the key-value vectors of historical tokens, continuously expanding as the sequence grows. To address this, paged cache management can be employed, partitioning the KV cache into fixed-size blocks allocated and released on demand. This approach reduces memory fragmentation, improves utilization, and supports a larger number of concurrent requests. Quantizing the KV cache to lower precision is another effective measure, drastically reducing memory requirements with controllable accuracy loss.

The multi-token prediction mechanism places additional demands on cache management. A single forward pass may generate multiple candidate tokens, which may be accepted or discarded in subsequent steps. The cache system must support efficient branching and rollback operations to avoid retaining excessive invalid cache for unaccepted prediction paths.

5. Inference Framework Adaptation and Critical Operator Tuning

For trillion-parameter sparse models, the adaptation capability of the inference framework determines whether the underlying hardware can be fully utilized. Current mainstream open-source inference frameworks are relatively mature in supporting large models, but they still require targeted extensions when facing new features like ultra-long contexts, sparse expert parallelism, and multi-token prediction.

Core issues the framework layer must resolve include distributed loading and switching of model weights, propagation and execution of parallel strategies, scheduler behavior under long contexts, and the sampling and verification process for multi-token prediction. Taking batch scheduling as an example: long-context requests often have vastly different sequence lengths. A simple First-Come-First-Serve (FCFS) policy can cause short requests to be blocked behind long ones for extended periods. A more reasonable approach is continuous batching, dynamically inserting newly arrived requests into the current batch, and using preemption or chunking strategies to prevent long requests from monopolizing resources.

Operator-level tuning significantly impacts actual performance. In sparse MoE models, the computation of expert feed-forward networks involves batched matrix multiplication. Because each token activates different experts, these computations can be scattered and irregularly shaped. Directly using general matrix multiplication (GEMM) APIs often fails to achieve ideal efficiency. Therefore, specialized optimization for grouped matrix multiplication is required, including adjusting data layouts based on expert groupings, reducing the overhead of small matrix multiplication calls, and utilizing specialized kernels to increase computational intensity.

Optimization of attention-related operators is equally crucial. For hybrid structures of linear and sparse attention, differentiated kernels must be implemented for different computational patterns. For instance, the linear attention path can focus on optimizing the order of matrix chain multiplication and memory access, while the sparse attention path requires optimizing the construction and memory access of sparse indices. The introduction of FlashAttention-like concepts—avoiding the materialization of the full attention matrix through tiled computation—remains effective in long-context scenarios but requires adjustments for new attention structures.

Furthermore, optimizing communication operators is vital in expert and tensor parallelism. Although Ultra Nodes possess high internal bandwidth, the frequency, granularity, and timing of communication calls must be carefully orchestrated. Merging small communications into larger operations, overlapping computation with communication, and selecting optimal communication paths based on topology information are standard optimization directions.

6. Servitization and Engineering Implementation of Inference

Deploying the optimized model as a usable inference service requires solving several servitization-level issues.

First is the choice of service architecture. The bare-metal approach suits teams with high requirements for performance and control, allowing developers to manage the OS, drivers, and inference frameworks themselves for maximum tuning space. Managed services provide more comprehensive infrastructure capabilities, including auto-scaling, health checks, and monitoring alerts, ideal for teams looking to reduce operational overhead. These two approaches are not mutually exclusive. In practice, one can use managed services to quickly validate the pipeline during the verification phase and switch to a bare-metal environment for stress testing and extreme tuning.

Second is model format and loading efficiency. The weight files of trillion-parameter models are massive. If loaded sequentially during cold starts, the service readiness time can reach tens of minutes or longer. Therefore, strategies like sharded concurrent loading, memory mapping, and incremental loading must be adopted to shorten startup times. For multi-replica deployments, pulling weight shards on demand from shared storage is preferable to downloading the entire model every time.

Third is API design and concurrency control. For long-context scenarios, request timeout settings, queue lengths, and concurrency limits must be re-evaluated. Clients submitting million-token requests may face long queueing times and Time-To-First-Token (TTFT) latency. The server needs to provide reasonable queue feedback mechanisms and proactively throttle when the load approaches its limit, ensuring the stability of accepted requests.

Finally, there is observability. After deploying large-scale inference services, continuous attention must be paid to metrics like device utilization, communication bandwidth, request queue time, TTFT, and tokens per second. Breaking down these metrics by model layer, device, and request type helps quickly identify performance bottlenecks and resource waste. Integrating log systems with monitoring alerts enables teams to rapidly locate issues when failures occur.

7. Best Practices and Considerations

When deploying trillion-parameter sparse models on Ultra Nodes, the following recommendations are worth noting:

1. Prioritize verifying the correctness and performance of a single forward pass before end-to-end servitization. Before large-scale deployment, run the complete computational graph on a small number of devices to check the sanity of output values and the time distribution of each step.
2. Design parallel strategies based on the actual hardware topology. Do not blindly pursue extreme configurations for any single parallelism method. Start with a small-scale setup, gradually increase parallelism, and observe if the performance curve scales linearly. If communication overhead rises rapidly, it indicates the current strategy has hit the interconnect bandwidth limit, requiring an adjustment in partitioning or communication scheduling.
3. Actively adopt attention optimization and cache compression technologies where the model structure allows. Long contexts are the primary source of memory consumption. The choice of attention mechanism and KV cache strategy directly determines the maximum sequence length and concurrency a single card can handle.
4. Establish clear lifecycle rules for multi-token prediction branch management. When accepting or rejecting candidate tokens, promptly release the cache and intermediate states of invalid paths to prevent memory from being occupied by discarded branches.
5. Focus on security and compliance. Even if the model is deployed locally, access control, log desensitization, and audit capabilities for the inference service must be part of the design. For multi-tenant service scenarios, adequate consideration should be given to network isolation, quota management, and data encryption.
6. Finally, make stress testing a prerequisite for going live. Beyond standard throughput and latency metrics, test performance under ultra-long sequence requests, burst concurrency, prolonged steady-state operation, and device failure scenarios to ensure the inference service is sufficiently robust in production.