A Practical Guide to Cost and Performance Estimation for Local Deployment of Very Large MoE Models

A Practical Guide to Cost and Performance Estimation for Local Deployment of Very Large MoE Models

As large model parameter counts move toward the trillion scale, the Mixture-of-Experts (MoE) architecture has become mainstream because it combines a large total parameter count with a relatively small number of activated parameters per token. However, deploying such models in a private, on-premises environment presents major memory and compute challenges. This article uses a publicly discussed trillion-parameter-class MoE model with roughly 2.8 trillion total parameters as an example and analyzes the technical requirements and cost estimates for local deployment across three performance tiers.

1. Core Architecture and Memory Footprint Baseline

Before discussing deployment options, it is necessary to clarify how memory usage is calculated for MoE architectures. Assume the target model has 2.8 trillion total parameters, uses 896 expert networks, and activates only 16 experts per token, resulting in an expert activation ratio of about 1.79%. Its backbone uses Delta Attention with Attention Residuals, combined with a Gated MLA component.

1.1 Decoupling storage from computation

In MoE architectures, memory usage is determined by the total parameter count, while compute cost is determined by the activation ratio. This means that even if less than 2% of experts are activated during inference, the full weights of all 896 experts must still be loaded into addressable storage. This is the fundamental constraint for any deployment lower bound.

1.2 Weight size estimation after quantization

This model was trained with quantization-aware training during the SFT stage, using MXFP4 weights and MXFP8 activations. Converting 2.8 trillion parameters to 4-bit format gives a raw data lower bound of approximately 1,400 GB. However, according to the OCP specification, MXFP4 requires one 8-bit scale factor for every 32 4-bit elements. The actual space occupied by weight elements plus scale factors is approximately:

2.8T × 4.25 bits ÷ 8 ≈ 1,487.5 GB

This does not yet include non-4-bit tensors, metadata, memory alignment overhead, runtime workspace, or the KV cache. Therefore, the practical weight storage lower bound for full deployment is around 1.5 TB.

2. Deployment Requirements and Cost Breakdown Across Three Tiers

Based on the baseline above, deployment requirements can be divided into three progressive tiers, each with its own technical approach and hardware cost profile.

Tier 1: Basic Operation Without Strict Speed Requirements

Goal: Fit the nearly 1,500 GB of weights into storage so that the model can generate tokens, even if decoding speed is only in the single digits of tokens per second.

Technical approach: CPU-GPU heterogeneous inference

For extreme cost-control scenarios where speed is not a priority, the most direct approach is heterogeneous CPU-GPU inference. Each MoE layer contains a compute-intensive shared attention component and a storage-intensive sparse expert component. The attention portion can be kept in high-bandwidth GPU memory for computation, while all 896 experts, which dominate storage but are sparsely activated, can be offloaded to CPU memory on a large-memory server for feed-forward computation.

Hardware configuration and cost:

This approach requires high-end consumer or professional GPUs with around 24 GB of VRAM for attention computation, plus a dual-socket high-end CPU server with at least 1.5 TB to 2 TB of DDR5 memory. Total hardware cost can be kept in the range of tens of thousands to over one hundred thousand RMB. The trade-off is very slow decoding speed, with memory bandwidth as the absolute bottleneck.

Tier 2: Single-User Serving at 25 TPS

Goal: Achieve smooth interaction for a single user at a decoding speed of 25 tokens per second.

Technical approach: Full weight residency in GPU memory and single-node multi-GPU deployment

To reach 25 TPS decoding speed, relying entirely on CPU memory offloading is no longer practical. The primary weights must be moved into GPU memory. For reference, a commonly discussed 671B-parameter MoE model requires approximately 685 GB in FP8 format. Even with AWQ 4-bit quantization, it needs roughly 335 GB for weights plus 37 GB for activations, totaling nearly 400 GB to run on eight H100 GPUs with 640 GB of combined memory.

The target model in this article is more than four times larger than that reference model. Full deployment would require close to 1,500 GB of GPU memory, which means a single 8-GPU server, even with the latest large-memory accelerators, cannot hold all weights.

Hardware configuration and cost:

To achieve 25 TPS for a single user, at least 16 to 24 enterprise-grade 80 GB GPUs such as H100s or equivalent accelerators are required, forming a cluster with 1,280 GB to 1,920 GB of total GPU memory. High-speed NVLink interconnect is also necessary. A single server of this class typically costs more than 2 to 3 million RMB. Under this configuration, compute capacity is abundant, but memory utilization operates under tight constraints.

Tier 3: 10 Concurrent Users at 25 TPS with Quantization No Lower Than FP4

Goal: Support 10 concurrent requests, with each output stream maintaining at least 25 TPS, for a total throughput of 250 TPS.

Technical approach: Superpod architecture and high-speed interconnect fabric

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple GPUs, every token must have its activation values routed to the GPUs hosting the selected experts, computed, and then routed back. Communication latency directly reduces the benefits of compute capacity. If cross-node communication latency is too high, the throughput required for 10 concurrent requests cannot be sustained.

For this reason, the reference deployment recommendation is to use a superpod consisting of 64 or more accelerators. Superpods use NVLink or equivalent high-speed interconnect technology to create a large communication domain, keeping the frequent expert activation transfers within a single domain and minimizing communication latency.

Hardware configuration and cost:

This tier represents enterprise-grade full deployment. It requires 64 or more top-tier accelerators, typically distributed across 4 to 8 high-density compute nodes, along with petabyte-class high-speed distributed storage and dedicated InfiniBand or RoCE v2 high-speed network switches.

Hardware procurement alone can exceed 10 million RMB. In addition, high facility power requirements, with single-node power consumption often above 10 kW, cooling costs, and ongoing operational expenses must be considered. In the cloud, similar bare-metal server instances of equivalent scale can be rented to spread the initial capital expenditure.

3. Best Practices for Deployment Architecture

1. Run benchmarks first: Once the weights are available, standard datasets should be used to benchmark single-node memory bandwidth and cross-node interconnect bandwidth. Pay particular attention to expert routing hit rates and actual activation ratios so that batch size and concurrency strategy can be fine-tuned.

2. Optimize the KV cache: In 10-concurrency scenarios, the KV cache grows rapidly as context length increases. Techniques such as Paged Attention are recommended, along with asynchronous swapping of long-context states using object storage and message queues, to preserve GPU memory for active computation.

3. Integrate with cloud-native infrastructure: For small and medium-sized enterprises, directly purchasing a superpod may not be cost-effective. Instead, container platforms combined with elastic GPU resource pools from cloud providers can be used to dynamically scale multi-node multi-GPU clusters during peak periods. Logging services and monitoring and alerting systems can track per-GPU memory utilization and communication latency in real time for refined cost control.

4. Summary

Deploying a trillion-parameter MoE model is a combined challenge involving storage, compute, and communication bandwidth. From heterogeneous CPU-GPU deployments in the tens of thousands of RMB that simply run the model, to single-user multi-GPU setups costing millions for smooth interaction, to superpod clusters costing over ten million RMB for 10 concurrent users, cost increases exponentially across tiers. Organizations planning private deployments should carefully evaluate their actual concurrency requirements and latency tolerance, consider validating early with elastic cloud compute services, and then gradually migrate to on-premises superpod architectures as needs mature.