Trillion-Parameter Model Local Deployment Cost Breakdown: From VRAM Calculation to Hardware Selection

1. Understanding the Memory Logic of a Trillion-Parameter MoE Architecture

When discussing a mixture-of-experts (MoE) model with roughly 2.8 trillion total parameters, one core distinction matters first: total parameters and activated parameters are two completely different accounting systems.

Consider a typical trillion-scale MoE architecture with 896 experts. For each token, the router activates 16 of those experts, giving an expert activation ratio of about 1.79%. However, this is an expert activation ratio rather than a strict parameter activation ratio, because shared components such as attention layers and embedding layers always participate in computation. The model backbone may use a KDA-style attention mechanism with Attention Residuals, with Gated MLA as one component of the full inference path.

What does this mean for VRAM? In MoE deployment, memory usage is determined by total parameters, while the activation ratio mainly affects compute consumption. In other words, all 896 expert weights must be loaded into storage, and only the corresponding 16 experts are dynamically retrieved for computation during inference. This is a pattern of full storage with sparse computation.

Therefore, the first step in memory planning is to calculate weight storage requirements based on total parameter count, not activated parameter count.

2. Quantization Strategy and the Lower Bound of VRAM

2.1 Quantization Awareness Starts During Training

Trillion-scale models typically introduce quantization-aware training (QAT) during the supervised fine-tuning (SFT) stage, using a mixed-precision scheme with MXFP4 weights and MXFP8 activations. The advantage is that the model adapts to the numerical characteristics of low-precision inference during training, so deployment does not require an additional calibration dataset, and post-quantization accuracy loss is easier to control.

2.2 Calculating the VRAM Lower Bound

For a 2.8-trillion-parameter model converted directly to 4-bit:

• Raw data lower bound: 2.8T x 4 bit / 8 = 1,400GB

However, MXFP4 is not a pure 4-bit format. According to the OCP specification, MXFP4 attaches one 8-bit scale factor for every 32 4-bit elements. The actual per-parameter storage is therefore:

• 4 bit + 8 bit / 32 = 4.25 bit
• Total weight storage: 2.8T x 4.25 bit / 8 = approximately 1,487.5GB

That covers only the weights. Real deployments also require additional memory for:

• Non-4-bit tensors such as LayerNorm and embedding layers, which often keep higher precision
• Metadata and alignment padding
• Runtime workspace such as temporary buffers and communication buffers
• KV cache, which scales with sequence length and concurrency

2.3 Comparison with a Mid-Scale Model

To put this scale in perspective, consider a mid-scale MoE model with about 671B total parameters plus 14B MTP parameters. In FP8 format, it requires roughly 685GB. Even with AWQ 4-bit quantization, the weights take about 335GB and activations add 37GB, totaling around 400GB to run on eight 80GB accelerators with 640GB of aggregate VRAM.

A 2.8-trillion-parameter model is more than four times that size. This determines the lower bound for full deployment: at least about 1.5TB of pure weight storage, with real requirements going higher once runtime overhead is included.

3. Hardware Configuration and Cost Estimation

3.1 Feasibility Boundaries of Single-Node Deployment

A single accelerator server typically provides between 640GB and 1.5TB of VRAM depending on the accelerator model and count. For a 2.8-trillion-parameter model:

• FP8/INT8 precision: about 2.8TB of weights, requiring at least two to three 8-accelerator servers in a cluster
• MXFP4 precision: about 1.5TB of weights, which can barely fit into one high-end 8-accelerator server, but runtime space will be extremely tight
• More aggressive 2-bit quantization: about 700GB of weights, feasible on a single node, but accuracy loss needs careful evaluation

3.2 The Practical Choice of Supernode Deployment

For scenarios that require the full non-degraded performance, the recommended approach is typically a supernode consisting of 64 or more accelerators. The logic is straightforward:

1. VRAM capacity: 64 80GB accelerators provide 5,120GB of total memory, enough for weights, KV cache, and runtime overhead
2. Interconnect bandwidth: high-bandwidth interconnects within a supernode, such as NVLink or similar technologies, are essential for MoE routing communication
3. Compute redundancy: although sparse activation reduces per-token compute, total compute demand remains substantial under high concurrency

3.3 Cost Estimation Framework

The cost of deploying a trillion-scale model locally can be estimated across several dimensions:

Cost Item	Estimated Range	Notes
Hardware procurement	Millions to tens of millions of RMB	Depends on accelerator model and quantity
Power and cooling	Hundreds of thousands of RMB per year	A 64-accelerator cluster consumes roughly 30-50kW
Data center modifications	Hundreds of thousands of RMB	High-density racks, liquid cooling systems
Operations labor	Hundreds of thousands of RMB per year	Requires a professional MLOps team
Model licensing	Depends on the license	Commercial use requires attention to license terms

For most teams, the direct cost of deploying a trillion-scale model locally is enough to cover several years of cloud API usage. Therefore, the decision to deploy locally should be based on non-cost factors such as data privacy, latency requirements, and customization needs.

4. Deployment Path Recommendations

4.1 Quantization Trade-Offs

# Pseudocode: VRAM requirement estimation framework
def estimate_vram(total_params_b, bits_per_param, kv_cache_gb, runtime_overhead_gb):
    """
    total_params_b: total parameter count in billions
    bits_per_param: effective bits per parameter including scale overhead
    kv_cache_gb: estimated KV cache size
    runtime_overhead_gb: runtime overhead
    """
    weight_gb = total_params_b * 1e9 * bits_per_param / 8 / 1e9
    total_gb = weight_gb + kv_cache_gb + runtime_overhead_gb
    return weight_gb, total_gb
# MXFP4: 4.25 bit/param
w, t = estimate_vram(2800, 4.25, kv_cache_gb=100, runtime_overhead_gb=50)
print(f"Weights: {w:.0f}GB, Total: {t:.0f}GB")
# Output: Weights: 1488GB, Total: 1638GB

4.2 Phased Deployment Strategy

1. Validation phase: use a higher-precision quantization format such as INT8 on a small cluster to verify functionality and accuracy
2. Pilot phase: deploy with MXFP4 quantization on a supernode and evaluate real throughput and latency
3. Production phase: decide whether multi-node distributed inference is needed based on actual workload

4.3 Considering Alternatives

For most teams, on-demand cloud usage remains the more pragmatic choice. Local deployment is only recommended in the following scenarios:

• Data compliance requirements mandate private deployment
• Deep customization of model behavior is required
• There is long-term, stable, large-scale inference demand
• A professional MLOps team is available

5. Summary

The core challenge of deploying a trillion-scale MoE model locally lies in VRAM capacity and interconnect bandwidth. A 2.8-trillion-parameter model still requires about 1.5TB of pure weight storage under MXFP4 quantization, and actual requirements approach 2TB once runtime overhead is included. This makes single-node deployment extremely tight, with a 64-accelerator supernode being the more realistic starting point.

In terms of cost, hardware procurement, power and cooling, and operations labor constitute the main expenditure items, and total cost of ownership is usually far higher than cloud-based usage. Therefore, local deployment should be treated as a strategic infrastructure investment rather than a cost-optimization measure. Before making a decision, it is advisable to validate business value with cloud resources first and then assess whether private deployment is truly necessary.