Local Deployment of Trillion-Parameter MoE Models: VRAM, Speed, and Cost Calculations

1. Defining the Scope

This article discusses the local deployment of a large model with a total parameter count reaching the trillion scale, utilizing a Mixture-of-Experts (MoE) architecture. Before the official release of the model weights and technical reports, all figures regarding VRAM usage, throughput, and hardware configurations are engineering estimates based on publicly available architectural information, rather than empirical test results. Once the weights are released, these figures must be verified through benchmarking. This premise is crucial because the deployment cost of a trillion-parameter MoE model is highly sensitive to quantization formats, parallelism strategies, and interconnect bandwidth. Any change in these assumptions will significantly alter the conclusions.

To avoid theoretical abstractions, the analysis is broken down into three requirement tiers:

1. Just able to run: Speed is not a concern.
2. Single-user usage: Output speed reaches 25 tokens/s.
3. 10 concurrent users: Output speed remains at 25 tokens/s or higher per user, with a quantization level of at least 4-bit.

These three tiers correspond to entirely different hardware budgets, with potential cost differences spanning several multiples.

2. What Trillion Parameters Mean for VRAM

2.1 MoE VRAM is Calculated Based on Total Parameters

Consider a 2.8-trillion parameter MoE model with 896 experts, where each token activates 16 experts, resulting in an expert activation ratio of about 1.79%. It is essential to distinguish between two concepts:

• Expert activation ratio: The proportion of experts selected by the router out of the total number of experts.
• Parameter activation ratio: The proportion of parameters actually involved in computation out of the total parameters.

These two are not equivalent. The most critical engineering rule here is: MoE deployment VRAM usage is calculated based on total parameters; the activation ratio only affects compute consumption, not storage requirements. During inference, all experts must reside in addressable storage and are called upon when routed to. In other words, you cannot just load 1.79% of the weights—all 896 experts must be present.

2.2 The True Lower Bound of 4-bit Quantization

Assuming this model introduces quantization-aware training from the SFT (Supervised Fine-Tuning) stage, with weights in a 4-bit floating-point format and activations in an 8-bit floating-point format. A rough calculation: 2.8 trillion parameters converted to 4-bit yields about 1400GB of raw data. However, a 4-bit floating-point format is not purely 4 bits—according to the OCP specification, every 32 4-bit elements are accompanied by an 8-bit scale factor, resulting in an effective bit-width of:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This is solely for the weights themselves. Actual deployment must also account for:

• Non-4-bit tensors (embedding layers, normalization layers, certain attention components).
• Tensor metadata and alignment padding.
• Runtime workspace (activation buffers, temporary tensors).
• KV cache (which grows linearly with concurrency and context length).

Therefore, the actual VRAM requirement for a full 4-bit deployment will likely exceed 1.6TB, not 1400GB. For reference, a 671B-scale MoE model under FP8 has main parameters plus multi-token prediction parameters totaling about 685GB. After 4-bit quantization, the weights are about 335GB and activations about 37GB, totaling around 400GB to run on 8 80GB accelerator cards (640GB). A trillion-scale model is more than four times this size, which sets the lower bound for full deployment VRAM.

2.3 Why Supranodes are Recommended

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. If 896 experts are distributed across multiple cards, the activations must be transferred to the target expert’s card after token routing, and then transferred back after computation. This round-trip latency directly eats into compute utilization. The value of a supranode lies in using high-speed interconnects (such as NVLink or equivalent mechanisms) to enlarge the communication domain, ensuring that expert transfers do not leave the single high-bandwidth domain, thereby reducing communication overhead to an acceptable range. This is one of the most fundamental differences between deploying trillion-scale MoE models and standard dense models.

3. Hardware and Cost Estimation Across Three Tiers

The following estimates use 80GB VRAM accelerator cards as the basic unit. Prices are indicative market ranges and will vary based on procurement channels.

Tier 1: Just Able to Run, Speed Agnostic

The goal is the lowest cost to complete a single forward pass. A viable path involves more aggressive quantization (e.g., 2-bit) combined with CPU/GPU hybrid offloading, placing some experts in host memory or even NVMe storage, and paging them in on demand.

• VRAM requirement: 2-bit quantized weights are about 700–800GB. With offloading, the resident VRAM can be reduced to 8–16 cards.
• Host memory: 1.5–2TB or more is recommended to store offloaded weights.
• Storage: NVMe array of 4TB or more, essential for weight loading and paging.
• Cost range: A complete machine (including 8 cards + large memory + high-speed storage) is roughly in the range of hundreds of thousands of currency units. Using second-hand or leasing options can lower this further.

The realistic expectation for this tier is that a single generation might take tens of seconds to minutes. Throughput is extremely low, making it suitable only for validation, demonstrations, or offline batch processing, lacking interactive value.

Tier 2: Single-User at 25 tokens/s

25 tokens/s is the threshold for a smooth reading experience and a reasonable target for single-user interaction. This tier requires all model weights to be resident in VRAM to avoid jitter caused by paging.

• VRAM requirement: About 1.6–1.8TB under 4-bit quantization (including workspace and KV cache), equating to 20–24 80GB accelerator cards.
• Interconnect requirement: High-speed inter-card interconnect is necessary to ensure expert parallelism all-to-all communication does not become a bottleneck. Cross-node deployment requires high-bandwidth RDMA networks.
• Compute requirement: A trillion-scale MoE model activates hundreds of billions of parameters per token. A single card’s compute is sufficient to support 25 tokens/s; the bottleneck is primarily VRAM capacity and communication.
• Cost range: Estimating an 8-card machine at the million-level currency units, a 3-machine cluster reaches the multi-million level. On-demand cloud rental billed per card-hour is suitable for short-term validation.

Note that 25 tokens/s refers to the decode speed. The time-to-first-token (prefill) depends on the input length. In long-context scenarios, prefill may become the new bottleneck and requires separate evaluation.

Tier 3: 10 Concurrent Users at 25 tokens/s Each

This is a production-grade requirement and the most expensive tier. 10 concurrent users mean:

• KV cache amplification: Estimating 8K–32K context per concurrent request, the KV cache expands from tens of GB to hundreds of GB, growing linearly with context length.
• Compute amplification: Total throughput must reach 250 tokens/s. If single-card compute utilization is 30%–50%, the required number of compute cards increases significantly.
• VRAM requirement: 1.6TB for 4-bit weights + hundreds of GB for KV cache + workspace. 32–48 80GB accelerator cards are recommended, with a 20% margin reserved.
• Interconnect requirement: Supranode-level high-speed interconnect is mandatory. A hybrid split of expert parallelism and tensor parallelism is needed, keeping communication domains converged within a single node as much as possible.
• Cost range: A complete cluster is at the tens-of-millions currency level. For on-demand cloud instances, hourly costs are proportional to the number of cards, requiring capacity planning and elastic scaling for long-term operation.

A practical compromise is to implement request queuing and dynamic batching for concurrency, flattening the peak of 10 concurrent users. This allows fewer cards to handle the load at the expense of longer wait times per request, trading off user experience for cost reduction—a decision requiring product-side evaluation.

4. Three Overlooked Costs Beyond Hardware

First, power and cooling. Dozens of high-power accelerator cards running continuously can easily push cabinet power density beyond air-cooling limits, necessitating liquid cooling or higher-tier data centers. These infrastructure costs are often underestimated.

Second, operational complexity. Tuning expert parallelism splitting strategies, routing load balancing, faulty card removal, and weight version management require specialized inference framework optimization skills. Labor costs often exceed hardware depreciation.

Third, quantization precision loss. While 4-bit quantization performs acceptably on most tasks, it may cause quality degradation in numerically sensitive tasks (such as long-chain reasoning in code generation). Task-level evaluation must be conducted before deployment; relying solely on perplexity is insufficient.

5. Deployment Recommendations

1. Rent before buying: Use on-demand instances before weights are released to benchmark real VRAM usage and throughput curves, then decide on procurement scale.
2. Select tier based on stage: Use Tier 1 for validation, Tier 2 for interactive demos, and Tier 3 for production services. Do not go all-in immediately.
3. Prioritize interconnects: The bottleneck for trillion-scale MoE is often communication, not compute. Budgets should lean towards high-speed interconnects.
4. Reserve KV cache space: Model KV cache separately during VRAM planning, calculating required cards backward from target concurrency and context length.
5. Establish benchmark baselines: Run actual tests across the three tiers immediately after weight release, replacing these estimates with real data.

Local deployment of a trillion-scale MoE model is not simply a matter of “buying a few cards.” It is a systems engineering trade-off between VRAM capacity, interconnect bandwidth, and quantization precision. Calculating these three accounts clearly ensures the budget remains under control.