Estimating VRAM and Hardware Requirements for Deploying a Trillion-Parameter MoE Model Locally

When a trillion-parameter Mixture of Experts (MoE) model releases open-source weights, developers immediately ask two questions: can I run it on the hardware I already have, and if not, how much will it cost to acquire sufficient hardware? The answers depend less on raw compute speed and more on a fundamental constraint: VRAM capacity.

This guide does not make performance promises for any specific model version. Instead, it provides a reusable estimation framework: from parameter count to minimum VRAM, from quantization formats to practical usable capacity, from single-GPU setups to supernodes, and finally, concrete hardware ranges for three deployment goals. All figures are based on stated assumptions and must be validated against real benchmarks once the official weights and architecture details are published.

Clarifying a Common Misconception: Activation Ratio Does Not Equal VRAM Usage

The defining feature of an MoE architecture is sparse expert activation. With N experts and k activated per token, the activation ratio is approximately k/N. This ratio often sits in the single digits, making the model appear deceptively lightweight.

However, this ratio determines compute cost, not VRAM usage.

The reasoning is straightforward. During inference, the execution flow is:

1. All expert weights must reside persistently in the storage hierarchy (VRAM or high-speed host memory).
2. The router selects k experts based on the token’s hidden state.
3. The token’s activations are transferred to the devices hosting the selected experts for computation.
4. Results are sent back and merged with the residual stream.

In short, you must load all experts into memory before you can activate only a small subset. The activation ratio affects FLOPs and communication volume, but the minimum VRAM requirement is determined by total parameter count.

For example, consider a configuration with 2.8 trillion total parameters, 896 experts, and 16 activated experts per token. The expert activation ratio is about 1.79%. Yet VRAM planning must account for all 2.8 trillion parameters, not 2.8 trillion × 1.79%.

From Parameter Count to Minimum VRAM: Quantization Determines Everything

Theoretical Lower Bound for Raw Weights

Precision format directly dictates VRAM footprint. Approximate bits-per-parameter overhead for common formats:

Precision Format	Bits per Parameter	Raw Data for 2.8T Parameters
FP16/BF16	16	~5600 GB
FP8	8	~2800 GB
4-bit (INT4/AWQ-style)	4	~1400 GB
4-bit + block scaling factors	~4.25	~1487 GB

The last row requires further explanation. Current mainstream 4-bit quantization is not “pure 4-bit.” It uses block quantization: each group of 32 4-bit elements includes an 8-bit scaling factor. Based on OCP-related microscaling specifications, the average per-parameter overhead becomes:

4 bit + 8 bit / 32 = 4 + 0.25 = 4.25 bit
2.8e12 × 4.25 / 8 ≈ 1487.5 GB

This figure covers raw weight data only. Real deployments must also account for:

• Non-4-bit tensors: embedding layers, normalization layers, router, and attention projections often retain higher precision.
• Metadata and alignment: tensor alignment, index tables, expert routing tables.
• Runtime workspace: temporary buffers, operator workspaces.
• KV Cache: grows linearly with concurrency and context length; can reach hundreds of GB in long-context scenarios.
• Communication buffers: send/receive buffers required for all-to-all communication.

As a general engineering rule of thumb, actual usable VRAM requirements are typically 1.3 to 1.6 times the raw data lower bound. In other words, a 1400 GB-class 4-bit weight set may require a VRAM pool of 1800–2200 GB or more for practical deployment.

A Referential Scaling Baseline

To build intuition, consider a widely deployed medium-scale MoE model as a reference: about 670 billion total parameters, roughly 685 GB of primary parameters plus multi-token prediction (MTP) parameters in FP8 format. Under 4-bit quantization, weights drop to approximately 335 GB; adding roughly 37 GB of activations brings total to about 400 GB, which runs comfortably on 8 × 80GB accelerators (640 GB) with headroom.

Scaling this baseline to 2.8 trillion parameters:

• The scale is roughly 4× the reference model.
• Even with equally aggressive 4-bit quantization, weights land in the 1500 GB range.
• Applying a 1.4× engineering margin, the minimum VRAM for full deployment exceeds 2000 GB.

A Change Brought by Quantization-Aware Training

One point deserves special mention: if the model incorporates quantization-aware training (QAT) during supervised fine-tuning (SFT), with weights using 4-bit microscaling and activations using 8-bit microscaling, quantization loss is significantly lower than post-training quantization. This is favorable for deployers: lower precision can reduce VRAM usage while preserving acceptable output quality.

However, note two caveats:

1. QAT does not change how you calculate VRAM lower bounds—it simply makes low-precision deployment more viable.
2. 8-bit activations mean activation memory does not shrink proportionally; KV Cache and intermediate activations still require 8-bit or higher precision planning.

Hardware Estimates for Three Deployment Tiers

The three tiers below provide estimation logic and hardware ranges. All numbers are estimates based on public information and must be validated with benchmarks after weight release.

Tier 1: Just Run It, Speed Irrelevant

The goal is to load the model and complete one forward pass, with no throughput or latency requirements.

• Core constraint: VRAM + host memory total ≥ weights + runtime overhead.
• Feasible paths: 4-bit weights + CPU/GPU hybrid inference, or large host memory for weight residency with layer-by-layer swapping.
• Typical configuration: 2–4 high-VRAM accelerators in a single node + 1–2 TB host memory; or pure CPU inference (extremely slow, only for validation).
• Cost range: With 4-bit weights at ~1500 GB, full VRAM residency requires ~20 × 80GB cards. With host memory offloading, this can shrink to 2–4 cards plus TB-scale memory.

The key factor for this tier is not compute but storage capacity and memory bandwidth.

Tier 2: Single-User Use with Output ≥ 25 tokens/s

This tier enters the interactive usability zone. 25 tokens/s is roughly the minimum comfortable reading speed.

• Bottleneck analysis: Autoregressive decoding is memory-bandwidth-bound. Each generated token requires reading the activated expert weights. With ~1.79% of 2.8T parameters activated per token (~50B parameters) in 4-bit format, each token requires reading approximately 25 GB of weights.
• Bandwidth requirement: 25 tokens/s × 25 GB ≈ 625 GB/s effective weight-read bandwidth.
• Practical constraints: A single accelerator’s memory bandwidth typically falls in the 1–3 TB/s range, but this is shared with KV Cache read/write, communication, and activation transfers. Usable bandwidth is often only 50%–70% of the rated value.
• Typical configuration: 8–16 high-speed interconnect accelerators in a single node, with sharded weights, expert parallelism, and tensor parallelism.
• Cost range: This tier sits at the boundary between “individual researcher/small team” and “enterprise-grade.” Hardware investment is significantly higher than Tier 1.

It is critical to note that expert parallelism introduces all-to-all communication. Every generated token requires two rounds of activation transfer between devices. Insufficient interconnect bandwidth will allow communication latency to dominate, driving tokens/s well below theoretical values. This is why interconnect quality matters seriously at this tier.

Tier 3: 10 Concurrent Users, Each ≥ 25 tokens/s, with Quantization Floor

This is a production-grade serving requirement. Total throughput must reach 250 tokens/s without degraded per-request latency.

• Compute and bandwidth: Total weight-read demand reaches 250 × 25 GB ≈ 6.25 TB/s, which a single node cannot sustain.
• Parallel strategy: Must combine expert parallelism + tensor parallelism + data parallelism, distributing 896 experts across many cards or multiple nodes.
• Communication domain: Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. Every token must transfer activations to the target expert devices after routing, then transfer back after computation. If the communication domain spans low-speed networks, latency will dominate total time.
• Supernode approach: Official-level recommendations typically involve 64+ accelerators in a supernode, using high-bandwidth interconnect (e.g., NVLink or equivalent) to create a large communication domain so expert transfers remain within a single domain.
• Quantization floor: If the requirement mandates a minimum quantization level (e.g., no lower than 4-bit microscaling), you cannot further reduce precision when VRAM is insufficient. Hardware capacity must genuinely meet the requirement.
• Typical configuration: 64+ accelerator supernode with high-speed interconnect, high-capacity VRAM, and dedicated high-speed storage for weight loading.
• Cost range: This is the only tier that enters “data-center-grade procurement,” with budgets typically in the millions of USD range.

A Reproducible Estimation Checklist

Follow these steps for a repeatable estimate:

1. Confirm total parameter count P and quantization format; calculate raw weight data W = P × bits / 8
2. Add non-quantized tensors, metadata, alignment overhead; typically +10%–20%
3. Estimate KV Cache: concurrency × context length × layers × hidden dimension × precision × 2
4. Add runtime workspace and communication buffers; typically +10%–30%
5. Derive total VRAM requirement D
6. Based on target throughput T and per-token activation volume A, calculate bandwidth requirement B = T × A
7. Choose parallel strategy; confirm the communication domain covers expert distribution
8. Validate with real benchmarks: first-token latency, tokens/s, concurrency stability

Metrics That Must Be Verified After Weight Release

Before official weights and technical reports are published, all numbers remain estimates. After release, verify the following:

• Actual weight file size: Compare against estimates; confirm whether extra tensors are included.
• Per-token activated parameter count: Read directly from the configuration file rather than reverse-engineering from expert ratio.
• Single-card load feasibility: What is the minimum number of cards for a working configuration?
• Number of cards required for 25 tokens/s single-user: Measured, not theoretical.
• Throughput and latency curves under 10 concurrent requests: Watch for congestion collapse.
• All-to-all communication share: Use profiling tools to quantify communication time as a percentage of total time.

Summary

Local deployment of a trillion-parameter MoE model is fundamentally a VRAM capacity problem combined with an interconnect bandwidth problem. No matter how low the activation ratio, the premise remains: all experts must fit in memory. No matter how aggressive the quantization, hidden costs like microscaling factors and runtime overhead persist.

The three deployment tiers correspond to three fundamentally different hardware forms: validation-grade (capacity-first), interactive-grade (bandwidth-first, 8–16 cards), and service-grade (communication-domain-first, 64+ card supernode). When budgets are limited, clarify your goal first, then estimate line by line using the process above. This is far more reliable than copying someone else’s configuration.

Finally, remember: all estimates must be validated with benchmarks after weight release. Any change in model architecture, quantization scheme, or inference framework can shift the VRAM floor by tens of percent. Treat estimates as the starting point for procurement decisions, not the end.