Hardware Cost and Tiered Planning for Local Deployment of Ultra-Large MoE Models

1. Why Total Parameters Dictate Deployment Feasibility Over Activated Parameters

A common pitfall in large-scale sparse MoE models is estimating inference costs based solely on activated parameters. When deploying locally, the primary challenge is “where to store the weights.”

Consider a trillion-scale MoE architecture with 2.8 trillion total parameters and nearly 900 experts, where each token activates only a dozen or so experts. While the computational consumption per forward pass is significantly lower than a dense model, this does not mean you only need to provision memory based on the activation ratio. During inference, the router must have access to all expert weights to select the correct expert path based on token characteristics. In other words:

• The activation ratio determines “how much to compute”;
• The total parameter count determines “how much to store”;
• The routing and communication design determines “whether stored data can be efficiently accessed.”

Therefore, in the hardware planning phase, the first step is to calculate the lower limit of weight storage based on total parameters, and then scale up layer by layer according to quantization format, grouping granularity, runtime overhead, and concurrency requirements.

2. How to Estimate the Minimum VRAM After Weight Quantization

2.1 From Bits to Bytes

Assuming a model with total parameters P using b-bit quantization for storage, the raw weight volume can be approximated as:

Raw weight volume ≈ P × b / 8 bytes

For a 2.8 trillion parameter model with 4-bit quantization:

2.8T × 4 / 8 = 1400GB

This is only a theoretical lower limit. Actual deployment typically requires considering the following factors:

1. Quantization scales and zero-points: Group or block quantization appends scale parameters to each block. For example, if every 32 elements share an 8-bit scale, the effective bit-width might increase from 4.0 to 4.25 or higher.
2. Non-quantized tensors: Certain normalization layers, embedding layers, routers, and bias terms may be kept in FP16, BF16, or FP8, requiring separate volume calculations.
3. Alignment and fragmentation: Memory allocation is usually page or block-aligned, so actual usage is higher than the arithmetic value.
4. KV cache: For long sequences or high concurrency, the KV cache can become a significant additional overhead.
5. Runtime workspace: Includes communication buffers, activation staging, and kernel launch workspaces.

Calculating the 2.8T parameter model with 4-bit quantization and an 8-bit scale per 32 elements, the effective volume of the weights is approximately:

2.8T × 4.25 / 8 ≈ 1487.5GB

Adding non-quantized tensors, KV cache reserves, and runtime buffers, the VRAM requirement for a full-precision single-replica inference easily falls into the 1.6TB to 2TB+ range.

2.2 Why Looking Only at Weight Volume Isn’t Enough

Many teams attempting local deployment for the first time only calculate the size of the weight files. However, an inference service is a continuously running system, not a one-time load. What truly determines stable service is peak VRAM usage, which includes:

• Resident weights;
• Activation values varying with batch size and sequence length;
• KV cache growing with concurrent requests;
• Temporary buffers from communication libraries and compute kernels;
• Redundancy for multi-replica setups or rolling updates.

Therefore, for capacity planning, it is recommended to reserve at least a 20% to 30% VRAM margin. For trillion-scale models, this means preparing hundreds of gigabytes of additional space.

3. Hardware Planning for Three Deployment Tiers

Users typically focus on three tiers: functional validation, smooth single-user interaction, and multi-concurrency production.

3.1 Tier 1: Functional Validation (Regardless of Speed)

The goal of this tier is to verify whether the model can be loaded and complete inference. It is suitable for weight structure validation, single-request functional testing, or academic research.

The minimum requirement is that total VRAM exceeds the sum of weight volume and minimal runtime overhead. If weights consume approximately 1.5TB after 4-bit quantization, an accelerator cluster with a total VRAM exceeding 1.6TB is required at minimum.

To achieve this VRAM scale, two common approaches exist:

1. Multi-GPU stacking: For example, with 80GB per GPU, more than 20 cards are needed. With 141GB or 192GB per GPU, about 8 to 12 cards are required.
2. Large-memory inference cards: Using inference devices with 256GB or 512GB per card might only require 4 to 8 cards to fit the model.

Note that loadable does not mean efficiently runnable. If inter-GPU interconnect bandwidth is insufficient, a single forward pass might take minutes. This tier is strictly for functional validation, not external service.

3.2 Tier 2: Single-User Smooth Interaction (25 tokens/s)

When interactive experience matters, output speed becomes the key metric. Achieving 25 tokens per second for a single user means the total latency per token—from decoding to sampling to transmission—must be kept under 40ms.

Main factors affecting output speed include:

• Compute per token;
• Expert routing and inter-GPU communication latency;
• Batch size and scheduling efficiency;
• Compute utilization of quantized kernels.

In trillion-scale MoE models, although each token activates only a few experts, these experts are distributed across multiple GPUs. The all-to-all communication after routing becomes a bottleneck. If inter-GPU interconnects are slow, communication wait times alone can push single-token latency to hundreds of milliseconds, even with ample single-GPU compute.

To achieve 25 tokens/s, the following is typically required:

1. High-bandwidth inter-GPU interconnect: Use high-speed buses like NVLink, PCIe Gen5 or above, or specialized inference interconnects to ensure all-to-all communication is not the primary bottleneck.
2. Sufficient compute throughput: Under single-concurrency conditions, the compute and memory bandwidth corresponding to activated parameters must support the target output speed.
3. Low-latency quantized kernels: Use FP8 or 4-bit quantized kernels to reduce weight read times.
4. Optimized expert placement: Try to place frequently co-activated experts within the same communication domain or node.

Hardware at this tier is no longer just about VRAM; it is a combined issue of “VRAM + Compute + Interconnect.” In practice, an 8-GPU high-speed interconnected node is usually required, with per-GPU VRAM ideally at 141GB or 192GB for stability.

3.3 Tier 3: Multi-Concurrency (10 Concurrent Users at 25 tokens/s)

The core of concurrent deployment is balancing throughput and latency. Ten concurrent requests demanding 25 tokens/s means the system’s total output rate must reach 250 tokens/s. If input lengths are long, the prefill phase introduces additional fluctuations.

Key shifts in concurrent deployment include:

• Larger batch sizes linearly increasing compute demand;
• Significantly larger total KV cache;
• Scheduling strategies must accommodate latency-sensitive requests;
• A single node might not satisfy both VRAM and compute needs, necessitating cross-node deployment.

At this tier, simply adding more VRAM is insufficient. Total throughput demands must be converted into compute requirements:

Total throughput ≈ Concurrency × Output speed

Ten concurrent users × 25 tokens/s equals 250 tokens/s. The compute consumed per output token depends on activated parameters, sequence length, and batch efficiency. As batch size increases, memory bandwidth utilization may improve, but communication conflicts also rise.

Therefore, deploying for 10 concurrent users at 25 tokens/s typically requires a multi-node solution:

• 8 accelerators per node;
• High-speed intra-node interconnect and high-bandwidth, low-latency inter-node networks;
• Weights sharded by expert dimension and distributed across multiple nodes;
• MoE-specific inference engines for expert parallelism and pipeline scheduling.

This architecture approaches a small inference cluster, with hardware scaling to 16, 32, or more GPUs.

4. Why Communication Topology Outranks Single-GPU Compute

MoE inference has a distinct characteristic: every token undergoes routing at each layer, sending intermediate representations to the devices hosting the corresponding experts. For a model with 900 experts and a dozen activated per token, this all-to-all communication is extremely frequent.

If 900 experts are distributed across 64 GPUs, each token at every layer must communicate with the GPUs hosting its 16 active experts. A complete forward pass spans dozens of layers, multiplying communication events. Communication latency directly impacts end-to-end latency, and bandwidth limits dictate system throughput ceilings.

Therefore, when selecting hardware, prioritize:

1. Maximizing communication domains: Concentrate expert distribution within high-speed interconnect domains.
2. MoE-friendly topologies: Fat-tree, fully interconnected, or high-radix switch fabrics are better suited for all-to-all patterns.
3. Avoiding frequent cross-chassis communication: Placing co-activated expert combinations within the same node significantly reduces cross-node traffic.

This is why ultra-large MoE inference is rarely about “plugging a few GPUs into a host” but requires careful design of super-nodes or inference clusters.

5. A Basic Framework for Cost Estimation

Without referencing specific vendor pricing, a cost model can be built based on hardware categories.

5.1 VRAM Dominates the Cost

For trillion-scale models, VRAM capacity is a rigid constraint that dictates baseline hardware. Using the 4-bit quantized 1.5TB weight benchmark:

• If effective VRAM per GPU is 80GB, at least 20 GPUs are needed to hold weights, scaling to 24 with overhead.
• If effective VRAM per GPU is 141GB, about 11 to 12 GPUs are needed.
• If effective VRAM per GPU is 192GB, about 8 to 9 GPUs are needed.

Pricing varies significantly across different VRAM tiers, meaning total cost does not scale linearly with GPU count.

5.2 Interconnect and System Costs are Often Underestimated

Multi-GPU training or inference usually requires a complete system or super-node solution rather than standalone GPUs. High-speed interconnect backplanes, switches, optical modules, power supply, and cooling all add costs. For systems with 8 or more GPUs, interconnects and rack infrastructure can account for 20% to 40% of the total investment.

5.3 Cost Spans Across Tiers 1, 2, and 3

• Functional validation: Minimum requirement is meeting VRAM needs. Might require a few to a dozen large-memory GPUs, but offers no speed guarantees.
• Smooth single-user: Requires high-speed interconnects and sufficient compute. Costs are typically 1.5 to 3 times higher than functional validation.
• Multi-concurrency production: Requires multi-node setups, high-performance networking, and inference optimization. Costs may double again and incur ongoing operational expenses.

Thus, before weights are officially released, a pragmatic approach is to define target concurrency and latency first, reverse-engineer compute, VRAM, and interconnect needs, and finally evaluate whether to purchase hardware outright or rent cloud inference resources.

6. Is Cloud Deployment More Economical?

For most developers or SMEs, locally deploying trillion-scale MoE models means bearing hardware procurement costs alongside power, cooling, rack space, maintenance, and disaster recovery. Unless there is a long-term, stable high-concurrency demand, the upfront investment is difficult to amortize.

Cloud inference typically provides:

• Hourly or per-token billing options;
• Instance combinations with various VRAM and compute specs;
• Pre-optimized inference images and scheduling capabilities;
• Elastic scaling, avoiding idle hardware provisioned for peak loads.

For occasional validation or intermittent usage, cloud pay-as-you-go pricing is often more economical than buying local hardware. For sustained high-concurrency scenarios, long-term discounts or reserved instances can lower unit costs. Choosing between local and cloud fundamentally depends on workload stability and utilization rates.

7. Preparation Steps Before Weight Release

Even before full weights are released, teams can accomplish significant planning work in advance.

7.1 Build Memory and Compute Estimation Tables

Based on officially announced total parameters, expert count, activated expert count, quantization formats, and max sequence length, create a table to calculate:

• Weight volume across different quantization bit-widths;
• KV cache across different batch sizes and sequence lengths;
• Peak VRAM requirements;
• Theoretical compute requirements corresponding to target throughput.

7.2 Validate Inference Framework Support for Large-Scale MoE

Ahead of time, validate the selected inference framework’s expert parallelism, routing communication, and quantization kernel stability using mid-sized MoE models. Focus on:

• all-to-all communication efficiency;
• KV cache management under long sequences;
• Whether accuracy degradation from 4-bit quantization is within acceptable bounds.

7.3 Small-Scale Stress Testing and Extrapolation

If testing trillion-scale models directly is impossible, use MoE models with tens of billions of parameters for benchmarking. Record latency and throughput variations across single, 4, and 8 GPU setups to extrapolate larger-scale resource requirements. While communication patterns might shift during extrapolation, this establishes a baseline trend analysis.

8. Conclusion

Local deployment of trillion-scale MoE models follows a clearly tiered path:

• Functional validation only solves the VRAM capacity problem;
• Smooth single-user demands concurrent达标 in VRAM, compute, and interconnect;
• Multi-concurrency production enters the realm of distributed inference cluster design.

Before full weight release, all numbers are estimates rather than procurement guidelines. A safer approach is to build a capacity model based on total parameters and quantization formats, derive compute and communication needs from target throughput, and finally calibrate assumptions via small-to-medium scale benchmarks. For teams with fluctuating utilization or limited operational capabilities, prioritizing elastic cloud inference resources is generally more cost-effective than large upfront local hardware purchases.