Local Deployment of Ultra-Large MoE Models: A Hardware and Cost Estimation Guide

Public reports indicate that an ultra-large mixture-of-experts (MoE) model with 2.8 trillion parameters is about to be released. Deploying models of this scale on local hardware places extreme demands on the hardware architecture. Using this class of model as a reference baseline, this guide breaks down the underlying memory calculation logic, heterogeneous inference architectures, and concurrency requirements, and works out the hardware configurations and cost estimates for different deployment goals.

1. Memory and Communication Bottlenecks of Ultra-Large MoE Models

1.1 Architecture Characteristics and Memory Footprint

Take a 2.8-trillion-parameter MoE architecture with 896 sparse experts, where each token activates only 16 of them (an expert activation ratio of roughly 1.79%). The critical point: the memory footprint of an MoE deployment is calculated from the total parameter count, while the activation ratio only affects compute consumption. In other words, the weights of all 896 experts must be loaded into storage in their entirety, and dynamic routing pulls the relevant experts in at inference time.

On the backbone side, models of this type typically use fused attention mechanisms (combining components such as Delta Attention and residual connections). Quantization-aware training (QAT) is applied starting from the training stage, with weights stored in a 4-bit floating-point format (such as MXFP4) and activations in an 8-bit floating-point format (MXFP8).

1.2 Raw Data Size After Quantization and the Practical VRAM Floor

Converting 2.8 trillion parameters to 4-bit gives a raw data floor of roughly 1400GB. Under the actual MXFP4 specification, however, every 32 4-bit elements also carry an 8-bit scale factor. Calculating on that basis, the combined data volume of weight elements plus scales is 2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB.

That figure covers only the weights themselves. It does not yet account for non-4-bit tensors, metadata, memory alignment padding, runtime workspaces, or dynamically allocated KV cache. For comparison, a typical 671B-class model needs about 685GB of VRAM in FP8, and even with INT4 quantization it still needs roughly 400GB to run smoothly on 8 accelerator cards. Against that baseline, the minimum footprint of a fully enabled 2.8T-scale deployment is several times higher.

1.3 Supernode Interconnect Challenges

Official guidance recommends deploying such a model on a supernode built from 64 or more accelerators. Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth across the entire network. Because the 896 experts are distributed across multiple physical cards, the model must - for every single token it processes - move the activations to the card hosting the targeted expert once routing completes, run the computation there, and then move the results back.

If the communication domain is too small or the bandwidth insufficient, this cross-card transfer latency directly eats into the accelerator’s compute capacity. Expanding the communication domain with high-speed interconnect technology, and keeping data movement inside a single supernode, is therefore essential for inference efficiency.

2. Three Deployment Tiers and Hardware Cost Estimates

Enterprises and developers have very different business needs, so local deployment targets can be divided into three tiers, each with a corresponding hardware plan and cost assessment.

Tier 1: Just Get It Running, Regardless of Speed

Goal: Address and load nearly 1500GB of weights and get the model to produce tokens, even if output runs at only single-digit tokens per second.

Technical approach:
The most direct route is a GPU + CPU heterogeneous inference architecture. Each MoE layer contains a shared attention component, which is compute-intensive - keep it on the GPU to take advantage of its high bandwidth. The 896 sparse experts account for the bulk of the storage footprint, yet only a tiny fraction of them is activated per inference pass, so they can be offloaded entirely into CPU memory, with a large-memory server providing the capacity.

Hardware estimate:

• CPU memory: At least 1.5TB to 2TB of DDR5 memory to hold the complete expert weights plus system scheduling overhead.
• GPU: A single mid-to-upper-spec consumer or professional accelerator card with 80GB of VRAM is sufficient, mainly to host attention computation and the KV cache.
• Cost rating: Very low - essentially the cost of one large-memory server plus a single card. Suitable for research institutions analyzing model structure or running minimal inference validation.

Tier 2: Single-User Output at 25 TPS

Goal: Sustain a decode speed of 25 tokens per second (TPS) for a single-user request, matching a comfortable human reading pace.

Technical approach:
Under the heterogeneous offloading scheme, the primary bottleneck is PCIe bandwidth from CPU memory to the GPU. Reaching 25 TPS requires optimizing the data prefetch pipeline: while the next layer’s experts are computing, use DMA to prefetch the expert weights required by subsequent token routes into GPU VRAM ahead of time. The KV cache and the 16 currently active experts must also stay resident in VRAM, with the remaining experts left in system memory.

Hardware estimate:

• GPU: Multiple 80GB-class accelerator cards linked by high-speed interconnects to form an inference pool - not only to hold high-frequency experts and the attention components, but also to provide ample VRAM for the KV cache.
• CPU and memory: A server board with high-clock, multi-channel memory support, ensuring the memory-to-GPU PCIe Gen5 bandwidth runs fully saturated.
• Cost rating: Medium - roughly several high-end GPUs plus the accompanying enterprise server. Suitable for small and mid-sized teams running an internal single-user research assistant or code-completion tool locally.

Tier 3: 10 Concurrent Users at 25 TPS Each, No Quantization Below FP8

Goal: Team-level concurrency - serve 10 concurrent requests while keeping per-user output above 25 TPS, with quantization kept at FP8 or higher.

Technical approach:
At this level, weights can no longer be fully offloaded to slower CPU memory, and interconnect bandwidth and multi-card scheduling now impose enterprise-grade requirements. Ten-way concurrency dramatically inflates the batch size, causing expert routing to generate massive volumes of all-to-all traffic across cards. A supernode architecture is mandatory: ultra-high-bandwidth interconnects such as NVLink must weave 64 or more accelerators into a single communication domain to eliminate cross-node network latency. At the same time, the higher quantization level (FP8) means giving up 4-bit compression, so VRAM demand climbs sharply.

Hardware estimate:

• Supernode cluster: A supernode cluster built from at least 64 enterprise-grade compute accelerators (for example, advanced computing instances equipped with high-speed interconnect networks from cloud providers such as Baidu AI Cloud).
• Memory pool and network: More than 5TB of unified addressable VRAM within the cluster, plus a microsecond-latency RoCE v2 or InfiniBand lossless network fabric.
• Cost rating: Very high - an eight-figure hardware investment. Suited to large internet companies or MaaS (Model-as-a-Service) providers serving the public; purchasing cloud supernode instances directly is usually the better option, avoiding massive capital expenditure and the ongoing operations and maintenance costs of owned hardware.

3. Summary and Deployment Recommendations

Local deployment of an ultra-large MoE model is not simply a matter of piling up compute - it is a three-way trade-off among memory capacity, communication bandwidth, and routing/scheduling algorithms. Until the model weights are fully released, every configuration above is an estimate based on architectural principles. For real-world deployments, we recommend:

1. Validate in small steps first: Use a heterogeneous offloading setup (Tier 1) to verify that the model weights are complete and that business logic behaves coherently.
2. Calibrate with benchmarks: Once the weights are released, use standard datasets to precisely measure peak VRAM usage at different batch sizes and all-to-all communication latency.
3. Scale elastically in the cloud: For multi-concurrency production scenarios, favor advanced computing cluster services built on cloud-native infrastructure, and take advantage of the high-speed interconnect networks of cloud supernodes to avoid the hardware depreciation and interconnect bottlenecks of a self-built datacenter.