Local Deployment Architecture and Cost Estimation for Trillion-Parameter MoE Models

    This article examines the local deployment of a 2.8-trillion-parameter MoE model, covering memory footprint from sparse activation and MXFP4/MXFP8 quantization, pure-GPU supernode communication constraints, CPU offloading for cost reduction, and enterprise deployment recommendations.

    As large language models move toward trillion-parameter scale, Mixture-of-Experts (MoE) architectures use sparse activation to control per-inference computation while maintaining model capability. However, local deployment of very large MoE models remains a challenging engineering problem. Using a recent 2.8-trillion-parameter MoE model as an example, this article breaks down hardware cost estimation and architecture selection practices for local deployment.

    1. Model Architecture and Memory Footprint Analysis

    To estimate deployment costs accurately, it is necessary to start with the underlying architecture and data precision and analyze the main sources of memory usage.

    1.1 MoE Architecture and Sparse Activation

    The model has 2.8 trillion total parameters and uses a typical MoE architecture with 896 expert networks. During inference, each token activates only 16 experts, an expert activation ratio of about 1.79%. It is important to note that although the compute-level activation ratio is very low, this does not equal a strict parameter activation ratio. For deployment, the memory footprint of the MoE portion must be calculated based on total parameters; the activation ratio only affects dynamic compute consumption. This means the inference system must first load all 896 experts into storage, then bring the corresponding 16 experts into computation at runtime according to the routing strategy.

    1.2 Quantization-Aware Training and Precision Trade-offs

    To compress storage under a very large parameter count, the model introduces quantization-aware training from the SFT (supervised fine-tuning) stage and ultimately uses MXFP4 weight quantization with MXFP8 activation quantization.

    Converting 2.8 trillion parameters to 4-bit format gives a theoretical lower bound of about 1400 GB for raw data. However, according to the OCP specification, the MXFP4 format also requires an 8-bit scale for every 32 4-bit elements. After adding the scale, the total weight element size is approximately 2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB. This is only the lower bound for weight data and does not include non-4-bit tensors, routing metadata, memory alignment overhead, runtime workspace, or the increasingly large KV cache.

    For comparison, a mainstream 671B-parameter model requires about 685 GB of memory for main parameters plus MTP parameters in FP8 format. Even with AWQ 4-bit quantization, it still requires about 335 GB of weights plus 37 GB of activations, or about 400 GB in total, to barely run on eight high-end GPUs with 80 GB of memory each. The 2.8-trillion-parameter model discussed here is more than four times larger, so the lower bound for full deployment memory is directly raised above 1.5 TB.

    2. Pure-GPU Supernode Cluster Deployment and Communication Bottlenecks

    With memory requirements above 1.5 TB, configurations with eight or even sixteen GPUs per server can no longer support full deployment. A supernode cluster becomes a necessary choice.

    2.1 Extreme Sensitivity of All-to-All Communication

    Large-scale MoE inference is extremely sensitive to network communication bandwidth, especially All-to-All communication. When 896 experts are distributed across multiple accelerators, each token must move its activation values across cards to the node hosting the target expert after routing, then move the results back to the original node. This high-frequency cross-card data transfer introduces significant communication latency. If not handled properly, communication overhead can directly consume the compute benefit brought by sparse activation.

    2.2 Building a High-Speed Interconnect Domain

    To alleviate communication bottlenecks, official guidance and industry practice commonly recommend using 64 or more accelerators to form a supernode architecture. Through NVLink or an equivalent high-speed interconnect protocol, the communication domain can be enlarged so that expert activation transfers stay within a single physical communication domain as much as possible. This requires an environment with not only massive compute but also extremely high bandwidth and low-latency direct inter-node networking. In cloud architectures, high-performance computing clusters and high-speed network interconnectivity are designed for such large-scale parallel inference scenarios and can effectively reduce cross-node communication loss.

    3. Heterogeneous Inference: CPU Offloading as a Cost-Reduction Approach

    Compared with expensive pure-GPU cluster solutions, another direct and highly cost-effective approach is heterogeneous GPU-CPU inference.

    3.1 Decoupling Compute and Memory Access

    In the forward computation of an MoE model, each layer contains not only sparse expert networks but also a shared Attention part. In this model, the main architecture uses Delta Attention plus Attention Residuals and integrates a Gated MLA component. This part is compute-intensive and memory-access-heavy, making it suitable for remaining on GPUs and using high-bandwidth memory for fast computation.

    3.2 Offloading Sparse Experts to Memory

    In contrast, the 896 sparse experts account for most of the model’s parameters and storage space, but only a very small number are activated in each inference. Therefore, these experts can all be offloaded to standard CPU memory. Large-memory servers, such as single nodes with multi-terabyte memory, can hold the massive weights, and the CPU’s multi-core compute can be used during inference to calculate the experts selected by routing.

    This approach greatly reduces the rigid dependency on GPU memory capacity and makes it possible to deploy trillion-parameter models with limited GPU resources. Although overall inference speed may be lower than a pure-GPU cluster, heterogeneous inference offers high engineering value in workloads where throughput requirements are not extreme or cost control is the core objective.

    4. Enterprise Deployment Selection Recommendations

    For enterprises, local deployment of such a large model is a systems engineering task involving hardware procurement, network architecture, inference engine optimization, and other dimensions.

    1. Pursuing ultra-low latency: For high-concurrency real-time chat scenarios, a pure-GPU supernode solution is recommended, using high-speed interconnect technology to build a large-scale cluster and combining it with an efficient expert parallelism strategy.

    2. Balancing cost and performance: For batch processing, offline analytics, or cost-sensitive workloads, a CPU-GPU heterogeneous architecture is a better choice. Large-memory compute instances and object storage services from cloud providers can help control initial investment while providing elastic scaling.

    3. Implementing quantization strategy: Before deployment, verify the inference engine’s support for low-bit quantization such as MXFP4 and for scale tensors to avoid runtime fallback to high-precision computation and resulting OOM (out-of-memory on GPU).

    In summary, deploying a trillion-parameter MoE model is not simply a matter of stacking compute. It is a careful trade-off among the memory wall, the communication wall, and compute efficiency. With appropriate architecture selection and quantization strategy, enterprises can operate this frontier AI workload at controllable cost.