Deploying Ultra-Large MoE Models Locally: VRAM Estimation and Tiered Performance Cost Analysis

Deploying Ultra-Large MoE Models Locally: VRAM Estimation and Tiered Performance Cost Analysis

As large language models approach the trillion-parameter scale, the Mixture of Experts (MoE) architecture has become the mainstream solution for balancing computational overhead and model capabilities. However, local deployment of ultra-large MoE models faces immense VRAM pressure and communication overheads. This article explores how to locally deploy ultra-large MoE models and estimate the associated costs, providing an in-depth analysis of hardware configurations across three distinct performance requirement tiers.

1. The Fundamental Impact of Core Architecture on VRAM and Communication

Before evaluating deployment costs, it is crucial to clarify the architectural characteristics of ultra-large MoE models and their underlying hardware requirements.

1.1 VRAM Asymmetry Between Total and Activated Parameters
Taking a typical MoE model with 2.8 trillion total parameters as an example, it may contain nearly a thousand expert networks, but only a very small fraction (e.g., an activation ratio of about 1.79%) is activated per token during inference. It is important to note that the expert activation ratio only affects the floating-point operations (FLOPs) during inference; VRAM usage is strictly calculated based on the total parameter count. This means that even if a single inference pass utilizes only a few experts, the deployment environment must load the weights of all experts entirely into VRAM or local memory to allow the routing mechanism to call upon them at any time.

1.2 Storage Dividends from Low-Bit Quantization-Aware Training
Modern ultra-large models typically introduce Quantization-Aware Training (QAT) during the Supervised Fine-Tuning (SFT) stage, natively supporting low-precision formats (such as 4-bit weights paired with 8-bit activations). Under 4-bit quantization, the raw data lower limit for 2.8 trillion parameters is approximately 1400GB. However, according to Open Compute Project (OCP) specifications, every thirty-two 4-bit elements require an additional 8-bit scaling factor. Factoring this in, the weights and scaling factors combined consume about 1487.5GB of VRAM. This figure represents pure weight data alone and does not include non-4-bit tensors, routing metadata, VRAM alignment overhead, runtime workspaces, or the dynamically growing KV Cache.

1.3 All-to-All Communication and Supernode Interconnect Dependency
Large-scale MoE inference is extremely sensitive to cross-device all-to-all communication bandwidth. Because hundreds of experts are distributed across multiple accelerator cards, after routing computation for each token, the activations must be transferred to the cards hosting the target experts for computation, and the results must then be transferred back. This high-frequency cross-card data movement can easily introduce communication latency, thereby eating into computational dividends. Therefore, the industry generally recommends adopting a “supernode” architecture composed of dozens or even hundreds of accelerators. By utilizing NVLink or equivalent high-speed interconnect protocols, the communication domain can be expanded as much as possible, ensuring that core data movement operations do not cross independent network communication domains.

2. Tiered Deployment Requirements and Hardware Configuration Estimation

Depending on business tolerance and concurrency requirements, hardware configurations and costs for local deployment scale exponentially. The following analysis estimates these costs across three tiers.

Tier 1: Execution Only, Speed Disregarded

Requirement Characteristics: The model only needs to load successfully and output results. There are no requirements for time-to-first-token or generation speed. This is suitable for extremely low-frequency verification testing or offline, latency-insensitive tasks.

Deployment Strategy and Cost:

1. Storage Media Degradation: Since VRAM cannot hold the nearly 1500GB of quantized weights, a “CPU memory + SSD swapping” strategy can be adopted. All weights are stored on high-capacity NVMe solid-state drives, loaded into system memory on demand in blocks, and then transferred to VRAM.
2. Hardware Configuration Estimation: Requires 8 general-purpose accelerator cards with 80GB of VRAM each (providing 640GB of VRAM for hot weights and KV Cache), paired with over 1TB of system memory and several terabytes of enterprise-grade NVMe storage.
3. Performance Bottleneck: Limited by PCIe bandwidth and SSD IOPS, the token generation speed per inference may be less than 1 token per second (tps). The overall hardware investment primarily goes toward base compute servers and high-capacity storage. The initial fixed cost is relatively controllable, but computational efficiency is extremely low.

Tier 2: Single-User Usage, Output Speed Reaching 25 tps

Requirement Characteristics: Meets the fluent conversational experience for a single user, requiring an output speed of over 25 tokens per second, with time-to-first-token kept within a reasonable range.

Deployment Strategy and Cost:

1. Full VRAM Residency: To achieve a 25 tps throughput, core weights must reside entirely in GPU VRAM, completely eliminating the I/O bottleneck between SSDs and system memory. Calculating based on 1500GB of weights, at least 20 80GB accelerator cards are required. Factoring in runtime overhead, this may need to be expanded to 24 cards.
2. Communication Optimization: At this scale, the interconnect topology between accelerator cards is critical. Cluster nodes supporting large-scale cache-coherent interconnects must be configured to ensure that expert routing communication latency remains at the microsecond level.
3. Performance Bottleneck: Under single-user, single-concurrency conditions, compute utilization cannot reach its peak. The bottleneck typically lies in the intra-node interconnect bandwidth and the overhead of the routing scheduling algorithm. Hardware costs will scale significantly to a mid-to-large cluster level.

Tier 3: 10 Concurrency, 25 tps per User, and Quantization No Lower Than FP8

Requirement Characteristics: Supports 10 concurrent inference requests simultaneously, while maintaining a per-user speed of over 25 tps. Furthermore, the quantization level cannot be lower than FP8 (8-bit floating-point) to ensure numerical stability and model accuracy under high concurrency.

Deployment Strategy and Cost:

1. VRAM Surge from Precision Enhancement: If the quantization level is increased from 4-bit to FP8, the pure weight footprint for 2.8 trillion parameters will double, approaching 2800GB to 3000GB. This means at least 40 80GB accelerator cards are needed just to barely fit the weights.
2. Compute and Memory Bandwidth Overlap: 10 concurrency with a 25 tps requirement means the system must handle the generation of 250 tokens per second. Although MoE architectures have few activated parameters, multiple tokens under high concurrency may hit different experts, causing actual activated compute requirements and VRAM read/write bandwidth demands to multiply. The system requires at least a supernode cluster configured with 64 high-end accelerator cards (e.g., models with single-card memory bandwidth exceeding 3TB/s).
3. Dynamic Batching and KV Cache Management: 10 concurrent requests will generate a massive KV Cache, necessitating the introduction of VRAM paging management technologies like PagedAttention, combined with dynamic batching to optimize scheduling. At this point, high-speed cluster interconnects (such as 400G/800G RoCEv2 networks) and the scheduling capabilities of cloud-native infrastructure like container platforms and monitoring alerts become standard.
4. Cost Evaluation: Deployment at this level is equivalent to a supercomputing center scale. Not only is hardware procurement costly, but power supply per rack (tens of kilowatts) and liquid cooling retrofitting are mandatory engineering challenges.

3. Summary and Recommendations

Local deployment of ultra-large MoE models is not merely about stacking hardware; it is a systems engineering challenge centered around VRAM capacity, interconnect bandwidth, and scheduling efficiency. For teams only requiring verification, costs can be reduced through low-speed storage. For production-grade applications pursuing high concurrency and high precision, building a high-speed interconnected supernode cluster is the only viable solution.

When planning such infrastructure, beyond procuring physical hardware, enterprises can also evaluate solutions from established cloud providers. For example, leveraging high-performance computing clusters and high-speed interconnect networks provided by cloud vendors, combined with their mature container platforms and object storage services, can effectively reduce underlying operational complexity and initial capital expenditure. This allows teams to focus more on optimizing upper-layer model routing algorithms and business logic. Once the weights and technical reports are officially open-sourced, it is recommended to verify these estimations item by item against standard benchmarks to finalize the procurement list.