Deploying Trillion-Parameter MoE Models Locally: A Guide to VRAM Estimation, Hardware Selection, and Tiered Costs

1. Defining the Scope: What This Guide Can and Cannot Answer

Before diving into specific numbers, we must establish clear boundaries. As the complete weights and technical reports for trillion-scale MoE models are not yet fully public, any figures regarding throughput, VRAM consumption, and hardware configurations are engineering estimations based on publicly available architectural information, rather than empirical test results.

The methodology of this guide is structured as follows:

• Determinable aspects: Theoretical VRAM lower bounds determined by parameter counts and architectural ratios, bit-width conversions for quantization formats, and communication patterns in MoE inference.
• Estimated aspects: Runtime workspaces, KV Cache, framework overhead, and communication buffers.
• Requires empirical validation: Actual throughput (tokens/s), concurrent scaling efficiency, and precision loss post-quantization.

Once the official weights are released, readers should calibrate these estimations item by item using benchmark tests rather than blindly trusting any single theoretical number.

2. Understanding Capacity: What Actually Consumes VRAM in MoE Architectures

2.1 Total Parameters Dictate Storage, Activation Ratio Dictates Compute

Taking a MoE model with approximately 2.8 trillion total parameters as an example, its typical structure might contain hundreds of experts per layer (e.g., 896). After routing, each token activates only a small fraction of these experts (e.g., 16), resulting in an expert activation ratio of roughly 1.79%. Here lies a common point of confusion:

• Expert activation ratio ≠ Parameter activation ratio. Shared experts, the attention backbone, and embedding layers always participate in computation for every token. Therefore, the strict parameter activation ratio is higher than the expert activation ratio.
• VRAM consumption is calculated based on total parameters. During inference, all experts must be resident in memory (or quickly swapped in). Routing only decides “which experts to compute this time” and does not reduce storage requirements.
• Compute consumption is calculated based on activated parameters. This dictates the required floating-point compute scale, not the VRAM scale.

Thus, the first principle of deployment planning is: Calculate VRAM based on total parameters first, then calculate compute based on the activation ratio. The two cannot be substituted for one another.

2.2 Quantization Bit-Width Conversion: 4-bit Is Not Simply “Parameters Divided by 2”

Many people simply estimate weight size using “parameter count × bit-width ÷ 8”. For 4-bit formats with block-level scaling factors, this algorithm significantly underestimates the actual size. Taking the industry-standard MXFP4 format as an example: every 32 4-bit elements are accompanied by an 8-bit scaling factor. The equivalent bit-width is:

Equivalent bit-width = 4 + 8/32 = 4.25 bit
Weight size ≈ 2.8e12 × 4.25 / 8 ≈ 1487.5 GB

This is the theoretical lower limit of bare weights, which does not yet account for:

• Non-4-bit quantized tensors (e.g., layer normalization and some attention components retain higher precision);
• Tensor metadata and alignment padding;
• Runtime workspace (temporary buffers, communication buffers);
• KV Cache (which scales linearly with concurrency and context length).

For context: a ~670B scale MoE model under FP8 requires about 685GB for main parameters plus the multi-token prediction head. When switching to 4-bit weight quantization, it still requires around 335GB for weights plus tens of GB of activation buffers, totaling about 400GB to run comfortably on 8 x 80GB-class accelerators (640GB total). A trillion-scale model is over four times this size. The VRAM lower limit for a full deployment can be intuitively deduced from this: a single 8-GPU node is far from sufficient.

2.3 A Practical Capacity Estimation Checklist

It is recommended to accumulate the following items in order to build your own budget table:

Component	Estimation Method	Notes
Quantized Weights	Total parameters × equivalent bit-width ÷ 8	Mind the scaling factor overhead
Non-quantized Tensors	Based on framework implementation testing	Typically a few GB to tens of GB
KV Cache	Layers × heads × head dimension × 2 × context length × concurrency × precision bytes	Concurrency targets directly amplify this
Runtime Workspace	Framework-dependent; reserve 5%–10% of weight size	Communication buffers are non-negligible in large-scale MoE
Redundancy & Fragmentation	Reserve around 10%	Prevent OOM errors caused by memory fragmentation

3. Communication: The True Bottleneck of Large-Scale MoE Inference

The communication pattern for MoE inference is all-to-all. After a token is routed, its activation values must be transferred to the device hosting the target expert for computation, and then transferred back. The more experts there are and the more dispersed they are, the more frequent this transfer becomes.

This leads to two direct consequences:

1. Communication latency eats up compute. Even if single-card compute is sufficient, if all-to-all communication crosses low-speed interconnect boundaries, end-to-end throughput will be dragged down by communication, manifesting as low GPU utilization.
2. Communication domain size dictates available topology. A common industry practice is to use high-bandwidth interconnects (like NVLink or equivalent high-speed fabrics) to build supernodes. This keeps the distribution of hundreds of experts within a single communication domain so that all-to-all transfers never leave the domain. Official deployment recommendations for trillion-scale models typically start at the supernode scale (64+ accelerators) precisely for this reason.

For self-hosted deployments, this means: simply stacking machines connected by standard Ethernet will struggle to achieve usable inference throughput. In the hardware budget, the proportion of interconnect costs will be significantly higher than when deploying small or medium-sized models.

4. Tiered Objectives: A Configuration Estimation Framework

4.1 Tier 1: Just Getting It to Run (Speed Agnostic)

Goal: Verify that weights can be loaded and inference produces results. Used for research, debugging, or demonstration.

Approach: Accept extremely low throughput, and even tolerate offloading some experts (to host memory or local high-speed storage). Use a lower quantization level (e.g., 4-bit) to compress weight size, utilizing fewer accelerators paired with large-capacity host memory to piece together the minimum configuration capable of loading the model.

Estimation Essentials:

• Weight size is around the 1.5TB mark; host memory must accommodate the non-resident portions.
• The number of accelerators should be enough to “fit a single layer or partial experts per card,” typically requiring a multi-node, multi-GPU setup.
• Lower-bandwidth interconnect solutions are acceptable because offloading is already the bottleneck, making communication a secondary issue.
• Throughput may drop to single-digit tokens/s, resulting in poor interactive UX, but sufficient for correctness verification.

Cost Profile: Focused on “piecing together capacity.” Fewer accelerators are needed, but single-node memory and storage configurations are high. Overall investment is relatively contained, but usability is limited.

4.2 Tier 2: Single User, Output at 25 tokens/s

Goal: Single-user interactive usage with an output speed reaching 25 tokens/s or higher, offering a near-usable experience.

Approach: All weights must be fully resident in VRAM. KV Cache is provisioned for a single user with a short context, and the communication domain must cover all experts.

Estimation Essentials:

• VRAM requirement ≈ 1.5TB quantized weights + workspace and KV Cache overhead. Actual configuration must be noticeably higher than the theoretical lower limit.
• 25 tokens/s is an end-to-end metric. Prefill and communication overhead must be deducted; thus, both single-card compute and interconnect bandwidth require ample headroom.
• Using 80GB-class accelerators with an effective VRAM utilization of 80%, the required card count is roughly 1.5TB / (80GB × 0.8) ≈ 24 cards. Considering headroom and topological constraints, this realistically falls into the 32-card range.
• This tier requires all experts to be within the same high-speed communication domain, making a supernode mandatory.

Cost Profile: Accelerator counts enter the dozens, and interconnect costs rise significantly. This is the first balance point between “usability” and “cost.”

4.3 Tier 3: 10 Concurrent Users, Output at 25 tokens/s Each

Goal: Under a 10-way concurrent request load, each stream’s output speed should not drop below 25 tokens/s, with a quantization level no lower than 4-bit (e.g., MXFP4 weights paired with higher-precision activations).

Approach: This is the tier closest to production readiness. The core changes involve KV Cache and scheduling.

Estimation Essentials:

• KV Cache scales linearly with concurrency. Assuming an 8K context per stream, with layer and head configurations typical for trillion-scale MoE models, the KV Cache for 10 concurrent streams can reach tens to hundreds of GB. This must be listed as a separate item in the VRAM budget.
• Batching improves compute utilization but also amplifies communication volume; all-to-all buffer requirements rise in tandem.
• To maintain 25 tokens/s per stream, the decoding latency of each request within the batch must not be dragged down by others. Scheduling strategies and continuous batching capabilities become critical.
• The accelerator count typically increases by 50%–100% over Tier 2, landing in the 48–64 card range, depending heavily on interconnect topology and framework efficiency.

Cost Profile: Entering the “small cluster” realm. Beyond accelerators, high-speed interconnect switches, power supply, and cooling retrofits must be considered. Total Cost of Ownership (TCO) should be amortized over a three-year period, rather than just looking at the purchase price.

5. The Right Way to Estimate Costs

Do not simply ask “how much will it cost?” Instead, fill in the numbers item by item using the following framework:

1. Accelerator acquisition or rental cost: Required cards × unit price. Note that high-bandwidth interconnect versions carry a premium.
2. Interconnect and switching equipment: In supernode solutions, this component cannot be ignored.
3. Host and storage: Large-capacity memory, high-speed local storage (for weight loading and offloading).
4. Datacenter facilities: Power capacity and cooling capabilities. Dozens of high-power accelerators usually require specialized retrofits.
5. Software and operations: Inference framework adaptation, quantization calibration, benchmark testing, and continuous tuning labor costs.
6. Power and depreciation: Estimated based on full-load power consumption × electricity rate × operating hours. A three-year depreciation schedule is the standard metric.

A rough rule of thumb: in deployments of dozens of cards, the accelerators themselves account for about half to two-thirds of the total investment, with interconnects, facilities, and operations taking up the rest. Looking only at GPU prices will severely underestimate actual expenditures.

6. Implementation Recommendations and Validation Checklist

Once weights are released, follow this sequence to validate and prevent “paper specification” failures:

1. Load Test: Confirm quantized weights can be fully loaded. Record actual VRAM usage vs. theoretical values.
2. Single-Stream Benchmark: Fix input and output lengths. Measure prefill and decode latency segments.
3. Communication Profiling: Capture all-to-all time share to determine if limited by interconnects.
4. Concurrency Sweep: Scale from 1 stream up to 10. Record the tokens/s degradation curve per stream.
5. Precision Spot Check: Compare output differences on representative tasks before and after quantization.
6. Stability Test: Run continuously for several hours. Observe memory fragmentation and throughput drift.

Conclusion: Locally deploying a trillion-scale MoE model is essentially a three-variable constraint problem: “VRAM capacity + interconnect bandwidth + concurrency budget.” Tier 1 relies on piecing together capacity; Tier 2 relies on supernode residency; Tier 3 relies on dual redundancy in both capacity and scheduling. Until weights and empirical data are public, all numbers provided in this article should be viewed as input assumptions for your budget framework, not definitive conclusions. What truly determines success or failure is the item-by-item benchmark testing process executed after the weights are released.