Deploying Trillion-Parameter MoE Models Locally: A Complete Analysis of VRAM and Costs

1. Premises and Conclusions: Estimates Before Weight Release

This analysis uses a recently announced MoE model with approximately 2.8 trillion total parameters as a working sample. While the architectural details are public, the complete weights and technical report have not yet been officially released. Therefore, any figures regarding inference speed, VRAM, and hardware configurations presented here are engineering estimates based on available public information and API pricing. Once the weights are released, these figures must be verified through empirical benchmarking before finalizing procurement and deployment plans.

This article focuses on answering three core questions:

1. How is the VRAM budget calculated?
2. Beyond VRAM, what are the other bottlenecks?
3. Where exactly does the money go?

2. The VRAM Budget: First Principles of MoE Deployment Costs

2.1 Total Parameters Determine the VRAM Floor, Activation Ratio Determines Compute

The model features 2.8 trillion total parameters. Under its MoE architecture, it has 896 experts, with 16 experts activated per token, resulting in an expert activation ratio of approximately 1.79%.

A common misconception needs clarification: the 1.79% expert activation ratio does not equate to a strict parameter activation ratio. Non-expert components, such as attention modules and embedding layers, use full parameters and participate in every step. More importantly, a low activation ratio does not save VRAM. MoE routing is dynamic; any token can hit any of the 896 experts. The inference system must keep all expert weights resident in VRAM, fetching the corresponding experts for computation based on routing results.

Thus, VRAM usage is dictated by total parameters. The activation ratio affects the per-token compute load. Approximately 50 billion parameters (2.8T × 1.79%) participate in computation (attention aside), a figure that determines compute requirements and throughput limits, not VRAM demand.

2.2 The “Hidden Tax” of Quantization Formats: Scales, Metadata, and Workspace

On the positive side, this model natively employs a 4-bit weight and 8-bit activation micro-scaling floating-point scheme, having undergone quantization-aware training since the SFT stage. Let’s calculate the raw floor: 2.8 trillion parameters × 4 bits ÷ 8 ≈ 1400GB.

However, the calculation doesn’t end there. For block-scaling formats, every 32 4-bit weight elements requires an 8-bit scaling factor. The effective bit width per element becomes 4 + 8/32 = 4.25 bits. Consequently, the weight footprint increases to 2.8T × 4.25 ÷ 8 ≈ 1487.5GB. This is strictly for the weights themselves, excluding non-4-bit tensors (embeddings, specific attention and normalization parameters), quantization metadata, alignment padding, runtime inference workspaces, and the KV cache, which grows with context and concurrency. In engineering practice, it is safer to plan for 1.5TB of VRAM for weights alone, and a total system capacity of around 2TB.

# 1) Weight VRAM estimation: block-scaled quantization
def weight_mem_gb(params_b: float, w_bits=4, group=32, scale_bits=8) -> float:
    """params_b: parameter count in billions; returns GB"""
    eff_bits = w_bits + scale_bits / group   # Effective bit width per element
    return params_b * eff_bits / 8
print(weight_mem_gb(2800))                    # ≈ 1487.5 GB
print(weight_mem_gb(2800, scale_bits=0))      # ≈ 1400 GB (Raw floor without scale)

2.3 KV Cache: The Second Largest Head, Growing Linearly with Concurrency and Context

The general estimation formula for KV cache is: 2 × layers × KV heads × head_dim × context_length × batch_size × bytes_per_element.

# 2) KV cache estimation (substitute actual model configuration)
def kv_cache_gb(layers, kv_heads, head_dim, ctx, batch, bytes_per_elem=2):
    return 2 * layers * kv_heads * head_dim * ctx * batch * bytes_per_elem / 1024**3
# Example configuration: 100 layers, 8 KV head groups, head_dim=128, 32K context, concurrency 1
print(kv_cache_gb(100, 8, 128, 32768, 1))     # ≈ 12.5 GB

Using the hypothetical 100-layer, 8-KV-head configuration above, a single 32K context session consumes about 12.5GB. At 32 concurrent requests, this scales to roughly 400GB. In long-context scenarios, the KV cache can easily rival the weight footprint in size and must be allocated its own column in capacity planning.

2.4 Reference Point: Deployment Watermarks for 671B Models

To calibrate intuition, consider an open-source 671B parameter MoE model: in FP8 precision, the main parameters plus the multi-token prediction module occupy about 685GB, exceeding the 640GB total VRAM of 8 80GB-class accelerators. Switching to 4-bit weight quantization reduces the weights to about 335GB, and with activations and runtime overhead totaling around 400GB, it fits on a single 8-card node. The 2.8 trillion parameter model is over four times this size, elevating the VRAM floor for full deployment by an entire tier.

3. Compute and Communication: The Other Half of the Equation

3.1 All-to-All: The Hidden Tax of Large-Scale MoE

With 896 experts distributed across multiple cards, the activations of each token must be transferred to the card hosting the target expert for computation, and then aggregated back. The communication volume is proportional to the number of tokens, hidden dimensions, and top-k. When interconnect bandwidth is insufficient, massive compute power sits idle waiting for data, resulting in low throughput, low per-card utilization, and high latency jitter. Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth—this is the biggest difference from deploying dense models.

3.2 Why the Provider Recommends a 64+ Card Supernode

The model provider’s technical materials suggest deploying on a supernode consisting of no fewer than 64 accelerators. The essence of a supernode is to compress the all-to-all communication required for large-scale expert parallelism into a single high-bandwidth, low-latency interconnect domain. Accompanying engineering optimizations include compute-communication overlap, low-precision communication, topology-aware expert placement, and load balancing. Purchasing the cards is only the first step; the quality of the inter-card interconnect directly dictates how much compute power those cards can actually deliver.

3.3 Throughput Expectation Management

The decoding speed of a single request is constrained by VRAM bandwidth and the number of activated parameters. Even if the weights fit into VRAM, the actual throughput of a small-scale deployment may fall far below expectations. Only sufficiently high concurrency can saturate the compute power of such a cluster. The correct approach is to define your business’s concurrency and latency targets first, and then reverse-engineer the required hardware scale—not the other way around.

4. Where the Money Goes: Cost Tiers Across Four Routes

The following are order-of-magnitude estimates provided for budgeting purposes. Actual figures will vary with procurement channels and market conditions.

Route	Hardware Config (Estimate)	VRAM Level	Investment Magnitude	Applicable Scenarios
Full Self-Hosted	Supernode with 64+ accelerators	≥5TB	Tens of millions (hardware) + annual O&M	High-traffic production
Streamlined Deploy	24–32 80GB-class cards	2–2.5TB	Millions	Medium/low traffic, deep validation
Offloaded Inference	1–2 entry-level accelerators + large RAM + NVMe	—	Hundreds of thousands	Functional testing
Cloud Rental	On-demand instances	On-demand	Pay-per-hour	Short-term stress testing, elastic scaling

1) Full Self-Hosted: A single 8-card node costs in the millions, requiring at least 8 nodes to start. GPU servers alone will run into the tens of millions. Add high-speed networking, parallel storage, facility upgrades, and electricity (annual O&M typically runs hundreds of thousands to millions). Suitable for organizations with sustained, high-traffic production needs.

2) Streamlined Deploy: This assumes the provider releases official 4-bit weights. The weights take about 1.5TB; planning for a 2TB VRAM tier requires 24–32 80GB-class accelerators (3–4 8-card machines), representing a multi-million hardware investment. The trade-off is limited concurrency and throughput. Suitable for technical validation and medium-to-low traffic scenarios.

3) Offloaded Inference: Offloading some expert weights to CPU memory or high-speed NVMe drives reduces the hardware barrier to a workstation costing in the hundreds of thousands. However, decoding speed will drop to single-digit tokens per second. Suitable only for functional testing, unable to support production.

4) Cloud Rental: Mainstream cloud providers (such as Baidu AI Cloud) generally offer high-end accelerator instances, container platforms, and monitoring/alerting capabilities. Pay-as-you-go instances are friendly to short-term stress testing and elastic scaling. For long-term, full-load operations, self-hosting may become more cost-effective. A more pragmatic strategy is a hybrid approach: validate locally on a small scale, and burst to the cloud to handle peak loads.

5. Verification Checklist Post-Weight Release

1. Information Verification: Confirm total parameters, expert count and top-k, native quantization format and precision, attention layers and KV design, and official deployment recommendations and reference implementations.
2. VRAM Budget Review: Benchmark actual weight resident footprint, activation peaks, and KV cache growth. Adjust the formulas in this article item by item.
3. Performance Baseline: Record five core metrics: time to first token, per-step decode latency, steady-state throughput, all-to-all communication percentage, and VRAM watermark.
4. Phased Rollout: Single-node smoke test → small-scale expert parallelism → full supernode. Stress test communication and compute utilization at each step, ensuring no bottleneck displacement before proceeding.

6. Conclusion

Local deployment of a trillion-parameter MoE model is fundamentally a VRAM floor problem dictated by “total parameters × quantization format,” compounded by a communication problem dictated by “expert parallelism × interconnect bandwidth.” The VRAM budget can be calculated before purchasing, but communication capabilities dictate how much value you extract from the purchased cards. Before the weights are officially released, treat all numbers in this article as an estimation framework rather than a procurement basis. Once the weights are available, verify them with benchmarks—it’s never too late to sign the purchase order after that.