Local Deployment of Trillion-Parameter MoE Models: VRAM Calculation, Tiered Strategies, and Cost Estimation

1. Getting the Math Right: What 2.8 Trillion Parameters Actually Means

Before discussing the cost of local deployment, we must answer a more fundamental question: how much storage and VRAM does a model of this scale actually require? Many estimation errors stem from confusing the concepts of “total parameters” and “active parameters” in MoE (Mixture-of-Experts) architectures.

1.1 Total Parameters Dictate Storage, Activation Ratio Dictates Compute

Consider a model with 2.8 trillion total parameters using an MoE architecture with 896 experts, where each token routes to 16 experts. Based on the number of experts, the activation ratio is roughly 16/896 ≈ 1.79%. However, it is crucial to emphasize: this is the expert activation ratio, not the strict parameter activation ratio. The shared components in the main network (attention, embeddings, normalization, etc.) are activated for every token, meaning the actual proportion of parameters involved in computation is higher than 1.79%.

This distinction leads to two key engineering conclusions:

• Storage (VRAM/RAM) is calculated based on total parameters. All 896 experts must permanently reside in addressable storage, with the router dynamically fetching them per token during inference. Experts cannot be “loaded on-demand from disk,” as the latency for a single token would be unacceptable.
• Compute consumption is calculated based on active parameters. Since each token only passes through 16 experts, the FLOPs magnitude is significantly lower than that of a dense model with the same total parameter count. This is the core economic advantage of MoE.

In other words, MoE decouples the “storage wall” from the “compute wall”: the money is primarily spent on housing all the experts, while speed depends on the activation path and communication efficiency.

1.2 Quantization Bit-width: 4-bit is Not Simply “Parameters ÷ 2”

This model introduces quantization-aware training from the SFT (Supervised Fine-Tuning) stage, utilizing MXFP4 for weights and MXFP8 for activations. Converting 2.8 trillion parameters to 4-bit yields a raw data lower bound of:

2.8e12 × 4 bit ÷ 8 = 1400 GB

However, MXFP4 is a block-scaled format: every 32 4-bit elements are accompanied by an 8-bit scale. According to OCP specifications, the equivalent bit-width per weight element is:

4 + 8/32 = 4.25 bit
2.8e12 × 4.25 ÷ 8 ≈ 1487.5 GB

This 1487.5GB is still just the theoretical value of weight elements plus scales, and does not account for:

• Non-4-bit tensors (embedding layers, normalization parameters, router weights, etc., which typically retain higher precision);
• Tensor metadata and alignment padding;
• Runtime workspace (temporary buffers, communication buffers);
• KV cache (which scales linearly with concurrency and context length).

In engineering practice, a 10% to 20% buffer is typically added on top of this as planning margin. Therefore, approximately 1.6 to 1.8TB of addressable storage is a more reliable starting point.

1.3 A Reusable Calibration Method

We can use a medium-sized MoE model as a reference to calibrate our intuition. A model with roughly 670B total parameters plus about 14B multi-task parameters requires about 685GB of weights in FP8 format. Even when compressed to 4-bit quantization, the weights take up about 335GB. Adding roughly 37GB of activation buffers, it takes a total of around 400GB to run on a single 8-card 80GB machine (640GB total VRAM).

A 2.8-trillion-parameter model is over four times this scale, with a weight lower bound approaching 1.5TB. A single 8-card machine (even with higher-capacity GPUs) cannot possibly accommodate it in terms of volume. The lower bound for a “full-capacity deployment” is thus rigidly determined: it requires a multi-node, multi-GPU setup.

1.4 Communication: The Underestimated Second Wall

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. When 896 experts are distributed across multiple accelerators, the activations must be transferred to the target expert’s GPU after routing, and then transferred back. This round-trip latency will directly erode compute utilization.

Therefore, official recommendations suggest deploying such models on supernodes comprising 64 or more accelerators. By expanding the communication domain through high-speed interconnects (such as NVLink or comparable technologies), expert transfers remain within a single communication domain. This is not a luxury option for “pursuing performance,” but a necessary condition to liberate compute from communication bottlenecks.

2. Tier 1: Making It Run — Heterogeneous Offloading

Not all scenarios require high throughput. For model evaluation, effect validation, and offline small-batch generation, the goal is singular: fit the ~1.5TB of weights into addressable storage and have the model stably output tokens, even if it’s only single digits per second.

2.1 Core Concept: Compute-Intensive Parts on GPU, Storage-Intensive Parts on CPU

Each layer of an MoE model contains a shared attention component (in this case, KDA Attention plus Attention Residuals, with Gated MLA as a component). This part is activated for every token, making it compute-intensive and ideal for retaining on the GPU for high-bandwidth execution. Conversely, the 896 sparse experts occupy the vast majority of storage and only 16 are activated at a time. These are well-suited for offloading entirely to CPU memory, supplied by large-RAM servers.

This is a classic GPU+CPU heterogeneous inference path: the GPU handles attention and routing computation, while the CPU holds the expert weights and performs forward passes, with the two exchanging activations via PCIe or higher-bandwidth interconnects.

2.2 Hardware Configuration and Cost Range

The primary cost driver for this solution is memory capacity rather than GPU compute:

Component	Recommended Config	Description
CPU Memory	2TB+ (2.5~3TB recommended)	Accommodate ~1.5TB weights + workspace + KV cache
GPU	1~2 cards, 48~80GB VRAM each	Host attention backbone and activation buffers
Storage	NVMe SSD 4TB+	Store original weights, enable fast cold starts
Interconnect	PCIe 4.0/5.0	Determines CPU-GPU activation transfer bandwidth

Rough cost estimation: A large-memory server (2TB+ RAM) typically falls in the tens of thousands to over a hundred thousand currency units range. Adding 1~2 high-end GPUs brings the overall deployment cost to roughly the 100,000 to 250,000 currency units mark (excluding power and operations). Compared to a multi-node cluster, this is an order of magnitude in savings.

2.3 Trade-offs and Tuning Focus

• Extremely low throughput: Expert computation runs on the CPU, meaning single-token latency can reach hundreds of milliseconds to seconds. This is suitable for offline batch processing, not online serving.
• Memory bandwidth is the bottleneck: CPU-side expert forward passes are limited by memory bandwidth. It is advisable to select multi-channel memory platforms and focus on actual bandwidth rather than just capacity.
• Reserve KV cache: Long contexts and high concurrency will significantly squeeze memory. Space must be specifically allocated for this during planning.
• Quantization format support: Ensure the inference framework supports the MXFP4/MXFP8 block-scaled formats. Unsupported formats will require conversion at load time, which consumes extra memory and startup time.

The positioning of this tier is clear: use minimal capital expenditure to achieve “the ability to run the model,” making it ideal for technical validation and prototyping phases.

3. Tier 2: Small-to-Medium Clusters — Pursuing Usable Throughput

When business requirements demand online services or moderate concurrency, the latency of heterogeneous offloading becomes unacceptable. The experts must be moved back into GPU VRAM, utilizing multi-node and multi-GPU setups to distribute the capacity.

3.1 Capacity Calculation

With 1.5TB of weights and 80GB VRAM per card, the weights alone require about 19 cards. Factoring in activation buffers, KV cache, and communication buffers, 24 to 32 cards are practically needed to maintain a comfortable operational margin. If GPUs with larger VRAM are used, the card count decreases, but single-card costs rise, requiring a cost-benefit trade-off.

3.2 Parallel Strategy Combinations

Trillion-scale MoE typically requires a combination of parallelism strategies:

• Expert Parallelism (EP): Splits the 896 experts across different cards; the primary driver for capacity distribution.
• Tensor Parallelism (TP): Splits matrix operations in the attention backbone, typically constrained within a single node to manage communication.
• Pipeline Parallelism (PP): Splits by layer across nodes to reduce VRAM pressure on a single machine.
• Data Parallelism (DP): Increases throughput but duplicates all weights. It is generally used cautiously or only for specific layers in ultra-large models.

A common starting configuration sets the expert parallelism degree to 8 or 16, paired with intra-node tensor parallelism, keeping all-to-all communication within the node or a single rack as much as possible.

3.3 Cost Range

Based on 24~32 high-end accelerators, hardware procurement costs are generally in the range of millions of currency units. If renting, billing is typically by card-hour, which is suitable for short-term projects. The key in this tier is not “affordability,” but whether the communication topology matches the expert parallelism degree—if interconnect bandwidth is insufficient, the throughput gains from adding more cards will rapidly diminish.

4. Tier 3: Supernodes — Full-Capacity High-Performance Deployment

For high-concurrency online services and large-scale inference, the official recommendation of a supernode setup with 64 or more accelerators serves as the baseline configuration.

4.1 Why Supernodes are Necessary

As expert parallelism increases, cross-card all-to-all traffic grows quadratically. Under standard Ethernet interconnects, communication latency consumes most of the compute power, manifesting as “the more cards you add, the lower the efficiency per card.” Supernodes utilize a high-bandwidth, low-latency unified communication domain to restrict expert transfers within the same domain, maintaining a reasonable ratio between communication and computation overhead.

4.2 Configuration and Cost

• Accelerators: 64+ cards, 80GB-class VRAM per card;
• Interconnect: NVLink-level or equivalent high-speed interconnect, fully connected within the node;
• Supporting infrastructure: High-bandwidth storage, liquid cooling or high-force air cooling, redundant power.

Hardware investment typically falls in the tens of millions of currency units range, additionally requiring considerations for data center retrofitting, power capacity expansion, and operations team costs. The decision-making focal point in this tier is cost per token rather than absolute procurement price: the economics of a supernode solution only hold true when utilization is sufficiently high.

5. Decision Framework: Deriving the Plan from Budget

Here is a comparison of the three tiers in a single table:

Tier	Typical Config	Cost Magnitude	Throughput	Use Cases
Heterogeneous Offload	Large-RAM server + 1~2 GPUs	100K~250K units	Single-digit tokens/s	Validation, offline batch
Small-Medium Cluster	24~32 GPUs, multi-node	Millions of units	Medium concurrency	Internal services, medium traffic
Supernode	64+ GPUs + High-speed interconnect	Tens of millions	High concurrency	External online services

A few general recommendations:

1. Calculate storage before compute. The first constraint in MoE deployment is capacity. Always estimate using “total parameters × equivalent bit-width × 1.15” before selecting GPU models and counts.
2. Verify quantization formats in advance. Block-scaled formats have specific framework requirements. Load-time conversions introduce extra memory and time overhead; end-to-end stress testing should be performed during the planning phase.
3. Communication topology takes precedence over card count. Stacking GPUs is a fruitless investment if the expert parallelism degree does not match the interconnect bandwidth.
4. Plan KV cache separately. Context length and concurrency are the invisible killers of VRAM. It is recommended to reverse-engineer the reserved space based on target SLAs.
5. Look beyond procurement costs. Power, data center facilities, operations, and the human resources required for model iterations often exceed the cost of the hardware itself over a three-year cycle.

Local deployment of trillion-parameter models is fundamentally a tri-constraint problem of “capacity, bandwidth, and budget.” Recognizing the division of labor between total parameters and active parameters is the only way to spend every cent on the components that truly bottleneck the system.