Deploying a Trillion-Parameter MoE Model Locally: A Complete Breakdown of VRAM and Cost

A newly open-sourced 2.8-trillion (2.8T) parameter Mixture-of-Experts (MoE) model has raised a recurring question: “Can we deploy this in our own data center, and how much will it cost?” This article breaks down the full estimation logic so you can understand the deployment requirements for this model and apply the same method to budget planning for any trillion-scale MoE model.

1. Clearing Up the Biggest Misconception: VRAM Is Determined by Total Parameters, Not the Active Ratio

People encountering the MoE architecture for the first time often assume that because each token only activates a small subset of experts, VRAM usage can be reduced by the same activation ratio. That is not the case, and this misunderstanding can derail an entire procurement budget.

Take this 2.8-trillion-parameter model as an example. It has 896 experts, and each token is routed to 16 of them, giving an expert activation ratio of about 1.79% (16 ÷ 896). However, this activation happens dynamically during inference. You cannot predict which experts will be hit by the next token, so all 896 experts must remain resident in VRAM at all times. Routing determines “which experts are called this time,” not “which experts can be loaded.”

A useful analogy: an MoE model is like a library with 896 bookshelves. A reader only borrows 16 books at a time, so the reading workload is small, but the library still has to purchase and shelve every book.

There is also a technical nuance: 1.79% is the expert activation ratio, not the strict parameter activation ratio. Dense components such as the attention backbone and embedding layers are used for every token, so the actual proportion of parameters involved per token is higher. For deployment planning, this distinction mainly affects compute estimation. VRAM usage is always based on total parameters. In short: MoE saves compute per token, but it does not save VRAM.

2. How Much VRAM After Quantization: From a 1400 GB Lower Bound to Nearly 1.5 TB

Since all weights must fit in VRAM, how far can low-precision quantization reduce the footprint? This model is aggressive in this regard: quantization-aware training (QAT) was introduced from the SFT stage, with native MXFP4 weights and MXFP8 activations. In other words, 4-bit precision is not a post-training compression step. It is the target precision the model was trained for.

Even so, the numbers are substantial:

• Step 1, the raw 4-bit lower bound: 2.8 trillion parameters × 4 bits ÷ 8 = 1400 GB.
• Step 2, add block-wise scale factor overhead. According to publicly documented MX micro-scaling formats, every 32 4-bit elements share one 8-bit scale factor. The effective per-element bit width is 4 + 8/32 = 4.25 bits. The total weight size becomes approximately 2.8 trillion × 4.25 bits ÷ 8 ≈ 1487.5 GB.
• Step 3, this is still only the weights. A real deployment must also include non-4-bit tensors such as the embedding table, routing table metadata and memory alignment overhead, runtime workspace buffers such as activation and communication buffers, and KV cache, which grows linearly with context length and concurrency.

The estimation logic can be captured in a short Python function:

def moe_weight_mem_gb(n_params, bits, group_size=32, scale_bits=8):
    """Estimate MoE weight memory footprint in GB for MX-style block scaling"""
    eff_bits = bits + scale_bits / group_size  # effective bits per element
    return n_params * eff_bits / 8 / 1e9        # params -> bytes -> GB
# 2.8 trillion parameters, MXFP4 (one 8-bit scale per 32 elements)
print(moe_weight_mem_gb(2.8e12, 4))   # ≈ 1487.5 GB
# Raw 4-bit with no scale overhead
print(2.8e12 * 4 / 8 / 1e9)           # = 1400 GB

The conclusion: the practical lower bound for weights alone is about 1.5 TB, and any realistic plan must leave additional headroom on top of that.

3. Building Intuition with a Scale Comparison: A 671B-Class Model as Reference

Abstract numbers can be hard to grasp, so it helps to use another mainstream 671B-class MoE model as a reference point. This model also includes roughly 14B multi-token prediction (MTP) parameters.

• Native FP8 deployment: 671B main parameters plus about 14B MTP, at 1 byte per parameter, totals about 685 GB. A single 8-GPU server with 80 GB cards has 640 GB total, so it does not fit.
• 4-bit quantized deployment: weights are about 335 GB, plus activations of roughly 37 GB, bringing the total to around 400 GB. That fits on eight 80 GB cards, which is why single-node 8-GPU setups are common in the community.

Dimension	671B-class model	2.8T-class model
Total parameters	671B + ~14B MTP	2.8T
FP8 weights	~685 GB	~2.8 TB (theoretical reference only)
4-bit weights	~335 GB	1400 GB raw / ~1487.5 GB with scale factors
Practical footprint	~400 GB	Starting from 1.5 TB plus overhead
Typical form factor	Single node with 8 GPUs (640 GB)	64-card or larger supernode

The 2.8T model is more than four times larger than the 671B class, which immediately rules out a single-node 8-GPU solution. This establishes the hardware lower bound for full deployment: it cannot fit on one machine.

4. Why the Official Recommendation Is a 64-Card Supernode: Communication Is the Second Barrier

If the VRAM lower bound is about 1.5 TB, then in theory 20 80 GB cards could hold the weights. So why does the official guidance recommend 64 or more cards organized as a supernode? Because large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth.

During MoE inference, each forward pass follows a repeated pattern: tokens are routed to 16 selected experts, activations are sent via all-to-all communication to the cards hosting those experts, the experts compute, and the results are sent back through another all-to-all operation. With 896 experts distributed across dozens of cards, every token in every layer must be moved, computed, and moved back. Communication latency and bandwidth can easily dominate the total time. In extreme cases, the system spends more time waiting for data than performing matrix multiplications.

This explains why deployment scale cannot be based solely on “does the VRAM fit”:

1. Physical bandwidth hierarchy: intra-node high-speed interconnects can provide hundreds of GB/s of bandwidth, while cross-node networks are typically an order of magnitude slower. A supernode keeps 64 cards logically close to single-machine communication performance.
2. Expert partitioning convenience: 896 divides evenly by 64 (896 = 64 × 14), placing exactly 14 experts per card. This avoids load imbalance and partition waste.
3. Tensor parallelism bandwidth demand: the model’s attention backbone combines incremental attention with attention residuals and uses a gated latent attention component that significantly compresses KV cache. If the attention portion uses tensor parallelism, its communication frequency is higher than expert parallelism and must be kept within a single node whenever possible.

5. Local Deployment Essentials: VRAM, Parallelism, and Framework Selection

Translating these constraints into an actionable deployment plan can be summarized as a checklist:

• VRAM budget: plan for approximately 1487.5 GB of weights plus a 20%–40% margin for activations, KV cache, and workspace buffers. Do not provision right at the raw weight limit.
• Hybrid parallelism: use tensor parallelism (TP) for the attention portion within a node, expert parallelism (EP) to scale experts across cards, and pipeline parallelism where needed for multi-node scaling.
• KV cache planning: for a conventional structure, the KV footprint per token is 2 × number of layers × KV heads × head dimension × bytes per element. Latent attention designs compress this, but you should still measure and reserve capacity based on target context length and concurrency.
• Inference engine: choose a framework with native support for MXFP4 dequantization, large-scale expert parallelism, dynamic batching, and prefill/decode disaggregation. Avoid forcing a generic single-card serving solution.
• Quantization consistency: the model has been trained with QAT from the SFT stage, so native MXFP4 plus MXFP8 deployment is the intended form. There is no need to convert it to other formats.

6. What It Costs: A Reusable Cost Model

Now back to the original question. Cost estimation can be broken into four layers, each with a clear formula.

Layer 1: Minimum card count from VRAM. With flagship 80 GB cards, weights alone require about 19 cards (1487.5 ÷ 80 ≈ 18.6). With overhead, plan for at least 32 cards. With 192 GB high-memory cards, the weight lower bound is about 8 cards, and a practical minimum is 16. Combined with the communication constraints in Section 4, the official 64-card supernode remains the prudent target.

Layer 2: Hardware procurement. Using public market prices as a rough reference, flagship accelerators typically cost around $25,000–$30,000 per card, though prices fluctuate significantly and real quotes vary. For 64 cards, the accelerator cost alone would be roughly $1.6–1.9 million. Adding high-performance RDMA networking, multiple 8-GPU servers, liquid-cooled racks, and storage, experience suggests adding 30%–60% on top of the accelerator cost, bringing the total to roughly $2–3 million.

Layer 3: Operating and maintenance costs. A flagship accelerator under full load typically draws around 700 W, so 64 cards consume about 45 kW, with the full rack at the 50 kW level and liquid cooling usually required. At typical commercial electricity rates, daily power costs are in the hundreds of dollars, or roughly $50,000–60,000 per year. Add data center hosting and dedicated inference engineering staff, and annual operating costs are in the low six figures.

Layer 4: Depreciation and utilization—the real build-versus-buy question. With 4-year straight-line depreciation, $2.5 million in hardware amortizes to about $1,700 per day. Adding electricity and operations, the true daily cost of a self-built system is around $2,000. Comparing with cloud rental rates, where accelerator-hour pricing varies widely by region and provider, a rough estimate of $2–3 per card-hour puts 64 cards at roughly $3,000–4,600 per day. The decision rule is straightforward: building your own only makes sense if you can sustain above roughly 50% utilization. If your workload has clear peaks and troughs or unstable utilization, renting or managed services are cheaper.

There is also a third option: managed inference services and APIs. For most teams without a multi-million-dollar budget or dedicated inference engineering staff, using a cloud-hosted inference service with per-token or per-call pricing is the most economical path. It eliminates hardware procurement, data center retrofitting, and framework tuning costs entirely.

7. Summary: Three Takeaways

1. VRAM is based on total parameters. The 2.8T model has a weight lower bound of about 1487.5 GB in MXFP4, and a low expert activation ratio does not reduce VRAM requirements.
2. Communication determines scale. The all-to-all routing across 896 experts naturally pushes deployment toward a supernode form factor. The 64-card recommendation reflects a balance of capacity, bandwidth, and engineering stability.
3. The cost is significant. A self-built deployment means a one-time investment in the $2–3 million range plus six-figure annual operating costs. The depreciation model shows that building makes sense only for consistently high-utilization workloads. For everyone else, cloud-hosted or per-call API services are the more rational choice in most cases.