Local Deployment of Ultra-Large MoE Models: A VRAM and Compute Estimation Guide

As open-source large models push toward trillion-scale parameter counts, Mixture-of-Experts (MoE) architectures have become the mainstream approach for balancing compute cost against model capability. However, when enterprises or developers attempt to deploy models with trillions of parameters locally, they face substantial hardware adaptation challenges. This article uses a typical industry 2.8-trillion-parameter MoE model as a case study to dissect the underlying deployment logic and estimate the required hardware configurations and costs across three tiers: “just get it running,” “single-user 25 TPS,” and “10 concurrent users at 25 TPS each.”

1. Core Architecture Characteristics and VRAM Baseline Analysis

Before estimating deployment costs, it is essential to understand the target model’s technical architecture. The model has 2.8 trillion total parameters, uses an MoE architecture with 896 expert networks, and activates only 16 experts per token. Although the expert activation ratio is only about 1.79%, this does not mean VRAM requirements can be reduced proportionally.

1.1 The Storage-Compute Separation of MoE Architecture

In MoE inference, the activation ratio affects only the compute volume of a single forward pass, but VRAM occupation must be calculated based on total parameters. The routing mechanism dynamically selects experts at runtime based on input, meaning all 896 experts’ weights must reside permanently in addressable high-speed storage, ready at all times.

1.2 Quantization-Aware Training and the VRAM Floor

The model introduced quantization-aware training during the SFT stage, with native support for MXFP4 weights and MXFP8 activations. Under the MXFP4 specification, every 32 4-bit elements require an accompanying 8-bit scaling factor. Therefore, converting 2.8 trillion parameters to 4-bit yields a raw data floor of approximately 1400 GB. Accounting for scaling factors (parameter count × 4.25 bit ÷ 8), the net weight VRAM is approximately 1487.5 GB.

This does not yet include non-4-bit tensors, metadata, memory alignment overhead, runtime workspace, or dynamically growing KV Cache. For comparison, a common 671B-parameter model requires approximately 685 GB of VRAM in FP8 format, or about 335 GB for weights plus 37 GB for activations under AWQ 4-bit quantization, barely fitting into 8 high-end accelerator cards with 80 GB each. The model discussed here is more than 4× that size, which directly determines the hardware floor for full deployment.

2. Deployment Tier 1: Heterogeneous Inference for “Just Get It Running”

Goal: The model loads and produces tokens, without regard to speed, running on a single machine or small cluster.
Technical approach: GPU + CPU heterogeneous offloading

Faced with a VRAM barrier of nearly 1500 GB, pure-GPU deployment is not only prohibitively expensive but also suffers from severe resource waste. The most direct breakthrough is heterogeneous inference—separating compute-intensive tasks from storage-intensive tasks.

In this scheme, the shared attention portion of the model backbone (typically using innovative attention residual and gated multi-head latent attention components) has high compute density and stringent bandwidth requirements, so it remains on the GPU to leverage high bandwidth. The 896 sparse experts, which account for the overwhelming majority of storage, can all be offloaded to CPU memory (DRAM) since only a tiny fraction are activated at any time.

Hardware configuration estimates:

• Memory system: A server with approximately 2 TB to 4 TB of DDR5 memory to hold full weights and runtime overhead.
• GPU accelerator cards: One or a small number of compute cards with high VRAM bandwidth, dedicated to Attention computation.
• Cost profile: Hardware investment is compressed to the hundreds-of-thousands range, but the tradeoff is extremely low decoding speed (typically single-digit tokens per second).

3. Deployment Tier 2: Single-User Output at 25 TPS

Goal: Smooth single-user interaction with output speed reaching 25 tokens/s.
Technical approach: High-speed interconnect supernodes and aggressive VRAM utilization

When 25 TPS is required, the memory access latency of CPU offloading becomes a fatal bottleneck. All weights must be moved back into GPU VRAM, which places extremely high demands on communication interconnects.

3.1 The Necessity of Supernode Interconnects

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth between cards. With 896 experts distributed across dozens of accelerator cards, each token during forward propagation must—after routing—transmit activation values across cards to the card hosting the target expert, then transmit results back after computation. If cross-node communication relies on traditional networks (such as Ethernet or standard InfiniBand), communication latency will directly consume the compute gains.

Therefore, it is necessary to build a supernode cluster based on high-speed direct interconnects, expanding the communication domain to the entire cluster and ensuring that this high-frequency cross-expert data movement stays within a single supernode’s high-speed communication domain.

3.2 Hardware Configuration Estimates

• At least 20 or more high-end accelerator cards with 80 GB VRAM each (such as industry-standard H-series compute cards), forming a supernode with total VRAM exceeding 1600 GB.
• Must rely on an all-to-all network topology, ensuring bidirectional bandwidth between any two cards reaches hundreds of GB/s.
• Cost profile: A single supernode cluster’s hardware cost jumps to the millions of RMB range, suitable for individual researchers or small-scale high-value inference scenarios with strict latency requirements.

4. Deployment Tier 3: High-Throughput Scenario with 10 Concurrent Users at 25 TPS Each

Goal: Support 10 concurrent requests with each request maintaining single-user output speed above 25 TPS, at a quantization level no lower than 4-bit.
Technical approach: Continuous batching and distributed routing optimization

In the 10-concurrent scenario, the system must not only fit the model but also complete routing distribution and expert computation for 10 requests within an extremely short time. This requires the system to effectively merge memory access requests when processing multiple requests in parallel.

4.1 KV Cache and Runtime VRAM Surge

In concurrent scenarios, KV Cache VRAM occupation grows linearly with concurrency count and context length. Assuming a single request context of 8K, the KV Cache required for 10 concurrent requests will easily occupy tens of GB or even hundreds of GB of VRAM. Therefore, total VRAM reserves must include at least a 20% dynamic buffer pool on top of the 1500 GB weight baseline.

4.2 Expert Routing Load Balancing

10 concurrent requests means up to 160 expert instances (10 requests × 16 experts) may be activated simultaneously. If tokens from multiple requests hit the same expert, queuing will occur. Fine-grained continuous batching technology must be introduced, along with dynamic mapping of expert loads at the distributed scheduling layer, and multi-replica strategies when necessary to alleviate congestion on hot experts.

4.3 Hardware Configuration Estimates

• A large supernode cluster composed of 64 or more accelerator cards is recommended.
• The system requires a professional distributed inference framework supporting dynamic routing scheduling and VRAM pooling.
• Cost profile: Hardware investment in the tens of millions. In cloud-native scenarios, it is generally recommended to use high-performance compute instances from major cloud providers for on-demand provisioning to reduce fixed capital expenditure.

5. Summary and Architecture Selection Recommendations

Deploying ultra-large MoE models is essentially a game around “VRAM capacity” and “communication bandwidth”:

1. Minimal validation stage: Use CPU-offload heterogeneous schemes, trading time for space to validate model logic and business integration.
2. Single-user efficiency stage: A high-speed interconnect supernode architecture is mandatory; loading all weights into VRAM is the only path to 25 TPS.
3. High-concurrency production stage: Requires distributed expert replicas, continuous batching, and large-capacity interconnect clusters, with sufficient KV Cache VRAM headroom at the architecture level.

For most enterprises, building physical clusters of dozens or even hundreds of compute cards carries not only heavy capital pressure but also extremely high operational complexity. A more reasonable engineering practice is to leverage mature cloud-native compute platforms, rent compute instance groups equipped with high-speed direct interconnects on demand, and use cloud object storage for rapid model weight loading—thereby flexibly handling inference demands of different scales while controlling costs. Before deployment, pay close attention to the AI compute cluster solutions and distributed inference acceleration capabilities offered by major cloud providers, and use these as the foundation for building highly available inference services.