Deploying a Trillion-Parameter MoE Model Locally: VRAM Estimation and Cost Breakdown

1. Clarifying the Core Challenge: Why a Trillion-Parameter MoE Is Hard to Deploy Locally

Before asking “how much does it cost,” we must first answer a more fundamental question: what actually drives the cost? For dense models, the answer is straightforward—parameters multiplied by bytes per parameter, plus activation memory and cache. But for ultra-large MoE (Mixture-of-Experts) architectures, the VRAM budgeting logic changes in several key ways, and these changes are exactly where many deployment plans go wrong during the estimation phase.

This article uses a 2.8-trillion-parameter MoE model as the primary analysis target. We start from VRAM estimation methods, work step-by-step to derive the hardware threshold, and finally present a cost estimation framework. The methodology described is generic—you can substitute your own parameters, quantization formats, and other variables to apply it to your specific deployment scenario.

2. What MoE Architecture Really Means for VRAM

2.1 Expert Activation Ratio ≠ Parameter Activation Ratio

This model architecture uses MoE with 896 experts, but only 16 are activated per token—an expert activation ratio of approximately 1.79%. This is a common point of confusion: the 1.79% figure is the activation ratio in terms of expert count, not a strict parameter activation ratio. The model backbone also includes attention modules, shared layers, routing networks, and other non-expert components that are used in every forward pass.

More importantly, this ratio affects compute consumption, not VRAM footprint. VRAM budgeting for MoE deployment must be calculated on total parameters:

• All 896 experts’ weights must reside entirely in storage (VRAM or high-speed accessible media) because routing is dynamic—any expert could be selected by the next token.
• During inference, activation values are routed to the devices hosting the selected experts, computation is performed, and results are routed back.
• The activation ratio only determines how many parameters are actually invoked per computation, which drives FLOPs and memory access volume.

In short: buy VRAM for the total parameter count, but compute for the activated parameter count. This distinction directly determines the scale of hardware required.

2.2 Backbone Architecture Impact on VRAM Estimation

The model’s backbone consists of KDA (Kimi Delta Attention) and Attention Residuals, with Gated MLA as a component. From a VRAM estimation perspective, the key question is whether these components introduce additional non-weight resident overhead. Attention-like structures typically affect KV cache scale—Gated MLA, being a compressed/gated attention variant, organizes its KV cache differently from standard multi-head attention, and the actual footprint must be calculated based on the specific implementation. This highlights an important point: VRAM budgets cannot only count weights—KV cache can consume a significant share in large-scale, long-context inference.

3. Quantization Formats: From “Raw Data Lower Bound” to Actual Footprint

3.1 The Theoretical Lower Bound of 4-bit Weights

This model incorporates quantization-aware training from the SFT stage, with weights in MXFP4 format and activations in MXFP8. Let us first calculate the ideal scenario:

2.8 trillion parameters × 4 bits / 8 = approximately 1400 GB

This is the frequently cited “raw data lower bound.” But it is just that—a lower bound. It is unattainable in real deployment.

3.2 Scale Overhead of Block Quantization Formats

MXFP4 is a block floating-point format: every 32 4-bit elements carry an 8-bit exponent (scale). Per the OCP specification, the effective bit width per weight element is:

4 bits + 8 bits / 32 = 4 + 0.25 = 4.25 bits

Thus, the actual weight footprint is:

2.8T × 4.25 bits / 8 ≈ 1487.5 GB

This single factor is already about 6% higher than the raw data lower bound. Many rough estimates ignore this 0.25-bit overhead, which translates into tens of gigabytes of error at the trillion-parameter scale.

3.3 Five Easily Overlooked Categories of Overhead

Beyond the 1487.5 GB above, several items must also be included:

1. Non-4-bit tensors: Embeddings, layer norms, routing networks, and some attention parameters are typically not quantized to 4-bit—they retain higher precision and larger bit widths.
2. Metadata: Tensor descriptors, shard indices, quantization parameter tables, etc.
3. Alignment overhead: Memory allocators align to block boundaries, producing fragmentation at tensor edges.
4. Runtime workspace: Temporary buffers, communication buffers, operator workspaces.
5. KV cache: Grows linearly with concurrency and context length—the most unpredictable item in long-context scenarios.

Therefore, a practical estimation formula should be:

Total VRAM ≈ Weights (with scale) + Non-quantized tensors + Metadata & alignment
             + Runtime workspace + KV cache + Safety margin

In engineering practice, an additional 10%–20% safety margin is typically reserved to handle fragmentation and peak fluctuations. Based on this reasoning, the actual VRAM requirement for deployable capacity will be notably higher than 1487.5 GB.

4. Horizontal Comparison: Deployment Thresholds Across Model Sizes

Using a publicly known MoE model roughly one-quarter the size of the target model as a reference makes the scale difference easier to grasp.

This reference model has approximately 671B primary parameters, plus about 14B MTP parameters, totaling approximately 685B:

• In FP8 format, weights occupy about 685 GB.
• With 4-bit weight quantization, weights drop to about 335 GB, plus about 37 GB for activations—totaling roughly 400 GB, sufficient to run on 8 × 80 GB accelerators (640 GB combined).

By contrast, 2.8 trillion parameters are roughly 4 times larger than this reference model. Even with the same 4-bit quantization, weight occupancy exceeds 1.4 TB. After adding all other overheads, a single-node 8-GPU setup is completely infeasible. This is why the “full-capability deployment floor” is directly dictated by model scale.

Model Scale	Quantization Format	Approx. Weight Footprint	Typical Hardware
~685B	FP8	~685 GB	Feasible on 8 × 80 GB
~685B	4-bit + activations	~400 GB	Comfortable on 8 × 80 GB
2.8T	MXFP4	≥1487.5 GB (excluding other overheads)	Requires multi-node supernode

5. Why a Supernode Is Mandatory: Communication Is the Real Bottleneck

Having enough VRAM is only the first hurdle. The official recommendation for this class of model is deployment on a supernode composed of 64 or more accelerators. The core reason is not compute—it is communication.

Large-scale MoE inference uses an all-to-all communication pattern:

1. Each token selects 16 experts via the routing network.
2. These experts are distributed across multiple GPUs, so activation values must be sent to the devices hosting the selected experts.
3. The target devices complete the expert computation, and results are sent back and aggregated.

This means every layer and every token requires a cross-device all-to-all communication step. With 896 experts distributed across dozens of GPUs, routing distribution is naturally imbalanced, and the devices hosting hot experts become communication congestion points. When communication latency cannot be hidden by compute, accelerator utilization drops significantly—you pay for GPUs, but the GPUs wait for data.

Engineering mitigation strategies include:

• Hybrid Expert Parallelism (EP) and Tensor Parallelism (TP): Controls the number of experts per GPU and balances routing load.
• High-bandwidth interconnects: In-node high-bandwidth domain interconnects and low-latency RDMA-class networks between nodes to guarantee all-to-all bandwidth and tail latency.
• Communication-computation overlap: Pipeline activation transfer with local expert computation to hide latency.
• Routing balancing strategies: Use capacity factors, expert load-balancing losses, and other mechanisms to suppress hot spots.
• Hierarchical/grouped routing: Preferentially route tokens to same-node experts to reduce cross-node traffic.

All of these approaches require high-bandwidth interconnect support at the hardware level. Multi-node GPU clusters stitched together over standard Ethernet or PCIe interconnects typically fail to achieve usable throughput on trillion-parameter MoE models due to communication bottlenecks.

6. Cost Estimation Framework: Two Paths

6.1 Path One: Self-Built Hardware

Self-built costs can be broken into four components:

1. Accelerators: Starting at 64 GPUs, with total VRAM covering 1.5 TB+ of weights plus all overheads, and headroom to spare.
2. Interconnect network: High-bandwidth switches, RDMA NICs, cabling—in supernode scenarios, network costs are a significant share.
3. Servers and supporting infrastructure: CPU, memory, local storage, racks, power, and cooling—a trillion-parameter model typically requires liquid cooling or high-power air-cooling retrofits.
4. Operational costs: Data center space, electricity, failure rates, and spare parts. MTTR of a large cluster directly affects usable compute.

A rough magnitude estimate: one-time hardware investment typically ranges from tens of millions to over one hundred million RMB, depending on accelerator unit prices, interconnect solutions, and redundancy levels. This figure fluctuates significantly with supply chains—use current quotes for decisions; this is only a magnitude reference.

6.2 Path Two: Cloud Rental

For most teams, renting from a cloud provider is a more realistic option. The cost structure includes:

• On-demand instances: Billed by accelerator-hours; suitable for validation and short-term tasks.
• Reserved/Annual plans: Unit costs drop significantly for sustained, stable workloads.
• Supernode specifications: Bare-metal or cluster instances with multi-node high-bandwidth interconnects, typically rented as whole nodes.
• Storage and network: Model weights exceed 1.5 TB, requiring high-throughput object storage and internal bandwidth for loading and distribution.
• Supporting services: Monitoring, alerting, logging, container platforms—billed as standard cloud services.

The core variable in cost estimation is utilization: rented GPUs sitting idle are pure loss. We recommend first validating the inference pipeline and throughput on a small-scale instance, then calculating the required GPU count and rental duration based on measured tokens/s and business demand.

6.3 Three Engineering Recommendations to Reduce Cost

1. Ask “Do we really need the full model?”: Many business scenarios do not require the full 2.8-trillion-parameter model. Distillation, pruning, or using a smaller model in the same family can reduce costs by an order of magnitude.
2. Be cautious about quantizing to lower bit widths: MXFP4 is already the result of quantization-aware training. Further compression requires re-evaluating accuracy loss—do not assume it is safe.
3. Treat KV cache as a first-class citizen: In long-context, high-concurrency scenarios, KV cache can exceed weights as the largest VRAM consumer. Paged management, prefix cache reuse, and reasonable max-concurrency limits often save more money than buying better hardware.

7. An Executable Pre-Deployment Checklist

Before placing orders or provisioning resources, verify the following items in order:

[ ] 1. Define precision requirements: Is MXFP4/MXFP8 acceptable? Is the accuracy loss within business tolerance?
[ ] 2. Calculate weight footprint: Total params × effective bits / 8—make sure to include block quantization scale.
[ ] 3. Add non-weight major items: Non-quantized tensors, metadata, alignment, workspace.
[ ] 4. Estimate KV cache: Max concurrency × max context length × KV bytes per token per layer.
[ ] 5. Sum all items and add a 15% safety margin to derive the VRAM floor.
[ ] 6. Verify interconnect bandwidth: Can the all-to-all traffic be carried by the high-bandwidth fabric?
[ ] 7. Stress-test actual throughput: Use tokens/s, not theoretical FLOPs, as the acceptance metric.
[ ] 8. Compare 3-year TCO between self-built and cloud rental, factoring in utilization.

8. Conclusion

Deploying a trillion-parameter MoE model locally is fundamentally a problem of balancing VRAM, bandwidth, and cost. VRAM sets the hardware floor, bandwidth determines the actually usable compute, and cost depends on utilization. Calculate weights based on total parameters, include quantization scale overhead, treat KV cache as a major consumer, and regard all-to-all communication as the primary bottleneck—if you get these four points right, your estimates will not deviate by an order of magnitude. As for “how much it costs,” the answer is not a fixed number but a curve that shifts with precision requirements, concurrency scale, and deployment form. Calculate VRAM first; only then talk about money. The order cannot be reversed.