Local Deployment of Trillion-Parameter Models: VRAM Estimation and Cost Tiers

1. Start with the Math: How Much VRAM Does a Trillion-Parameter Model Actually Consume?

When discussing local deployment, the first step is not choosing a GPU but figuring out how much space the weights themselves occupy.

Take an MoE (Mixture of Experts) model with roughly 2.8 trillion total parameters as an example. Its sparse structure is: 896 total experts, with each token activating 16 of them, giving an expert activation ratio of about 1.79%. Two concepts must be distinguished here—expert activation ratio is not the same as parameter activation ratio. MoE VRAM usage must be calculated based on total parameters, because all experts must reside in storage; the activation ratio only affects compute consumption: during inference, 16 experts are routed in for computation, then returned.

For the backbone structure, such models typically adopt a linear attention variant plus attention residual connections, including components such as gated multi-head latent attention. These details affect the computation graph and the shape of the KV cache, but not the order of magnitude of total weight size.

The key variable is the quantization format. Assume the model underwent quantization-aware training from the supervised fine-tuning stage, with weights in 4-bit MXFP4 and activations in MXFP8:

• Raw data lower bound: 2.8T × 4 bit ÷ 8 ≈ 1400GB
• However, MXFP4 carries an 8-bit scale for every 32 4-bit elements. Per the Open Compute Project specification: 2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB

This is only the weights. It does not yet account for non-4-bit tensors (such as normalization layers and embedding layers), metadata, memory alignment, runtime workspace, and KV cache.

For a cross-reference: a 671B-class MoE model in FP8 format has about 685GB of main parameters plus multi-token prediction parameters; converted to 4-bit weight quantization, it has about 335GB of weights plus 37GB of activations, totaling about 400GB to run on 8 × 80GB accelerators (640GB VRAM). A 2.8-trillion-parameter model is more than 4× that size, which determines the lower bound for a full deployment.

Conclusion: the VRAM lower bound for a full deployment is on the order of 1.5TB, and that is only a starting point.

2. Why the Official Recommendation Is a Supernode with 64+ GPUs

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth.

With 896 experts distributed across multiple GPUs, each token’s activations must be moved to the GPU hosting the target expert after routing, computed, and then moved back. This transfer latency directly eats into compute—no matter how many GPUs you add, if communication can’t keep up, the GPUs sit idle waiting for data.

The value of a supernode is this: use high-bandwidth interconnect (such as NVLink or an equivalent high-speed bus) to enlarge the communication domain so that expert transfers stay within a single communication domain. If cross-node traffic goes over ordinary Ethernet, all-to-all latency multiplies, and decode-phase throughput collapses.

So “64+ accelerators” is not an arbitrary number—it is the engineering lower bound determined by communication domain size.

3. Tier 1: It Runs, Speed Doesn’t Matter

The only goal: fit nearly 1500GB of weights into addressable storage and get the model to emit tokens, even at single-digit tokens per second.

The most direct approach is GPU + CPU heterogeneous inference:

• Each MoE layer has a shared attention part (compute-intensive) that stays on the GPU for high-bandwidth computation;
• The 896 sparse experts account for most of the storage, but only 16 are activated at a time, so they are all offloaded to CPU memory and fed by a large-memory server.

A representative implementation of this approach is the CPU-offload inference frameworks common in the community. The high-performance numbers in their official tutorials typically come from a dual-socket server CPU paired with a single consumer flagship GPU, running 671B-class 4-bit quantized weights (about 377GB):

• prefill: 200+ tokens/s
• decode: about 14 tokens/s
• But with expert-selection strategy constraints; real-world general scenarios will be lower

Cost profile (Tier 1):

Item	Specification	Notes
GPU	1 consumer flagship GPU	Handles attention and routing
Memory	1.5TB+ DDR5	Holds all expert weights
CPU	Dual-socket high-core-count server CPU	Main expert compute engine
Total budget	~100,000–200,000 RMB	Varies with memory capacity and channel count

This tier is positioned for validation and debugging: confirming that model weights are complete, routing logic is correct, and output quality is usable. Throughput is not production-grade, but it is sufficient for evaluation and prompt tuning.

4. Tier 2: It Runs Well Enough to Be Usable

When decode needs to hold steady at tens of tokens/s and support a small amount of concurrency, CPU memory bandwidth becomes the bottleneck. At this point, some experts need to move back to the GPU.

A typical configuration is multi-GPU + large memory + quantized weights:

• 8 × 80GB accelerators provide 640GB VRAM, enough for the attention backbone, KV cache, and some hot experts;
• Remaining experts stay in CPU memory and are fetched on demand over PCIe;
• Weights stay at 4-bit, and KV cache is also quantized.

The key optimization at this tier is the expert caching strategy: based on routing history statistics, keep frequently activated experts resident in VRAM and leave low-frequency ones in memory. Every 10% improvement in hit rate can produce a visible improvement in decode throughput.

Cost profile (Tier 2):

Item	Specification	Notes
GPU	8 × 80GB accelerators	640GB VRAM
Memory	1TB+	Stores cold experts
Interconnect	PCIe 4.0/5.0	The transfer bottleneck
Total budget	~1.5–2.5 million RMB	Includes server platform and networking

This tier can support internal tools, low-frequency Q&A, and similar scenarios, but concurrency is limited, and the prefill phase remains a weak point.

5. Tier 3: It Runs Fast—Full Supernode

To keep a trillion-parameter MoE at low latency under high concurrency, you must return to the officially recommended architecture: a supernode of 64+ accelerators.

Core characteristics:

1. All weights resident in VRAM: 1487.5GB of weights spread across 64 GPUs is about 23GB per GPU; with KV cache and runtime overhead, 80GB per GPU still leaves headroom;
2. All experts within the high-speed communication domain: all-to-all goes over NVLink-class interconnect with controllable latency;
3. Combined tensor parallelism + expert parallelism + pipeline parallelism: attention layers are split with tensor parallelism, MoE layers are distributed with expert parallelism, and pipeline parallelism fills between layers;
4. Tiered KV cache management: KV for hot tokens stays in VRAM; cold tokens are offloaded to host memory or high-speed storage.

Cost profile (Tier 3):

Item	Specification	Notes
GPU	64+ accelerators	Typical topology: 8 nodes × 8 GPUs
Per-GPU VRAM	80GB class	Total VRAM 5TB+
Interconnect	NVLink / equivalent high-speed bus	Communication domain covers all experts
Network	High-bandwidth RDMA	Inter-node all-to-all
Total budget	Tens of millions of RMB	Includes data center, power, cooling

This tier is what “full local deployment” really means, suitable for scenarios with hard requirements on data sovereignty and latency.

6. Easily Overlooked Hidden Costs

VRAM and GPU counts are only the numbers on paper; real deployment involves several additional costs:

1. Power and cooling: a 64-GPU supernode draws over 40kW at full load; data center retrofit costs cannot be ignored;
2. KV cache growth: in long-context scenarios, KV cache can consume tens of GB of VRAM, requiring advance planning;
3. Weight loading time: loading 1.5TB of weights from storage to VRAM takes minutes even over high-speed storage; cold-start experience must be designed for;
4. Quantization accuracy loss: the degradation of 4-bit weights on tasks such as mathematical reasoning and code generation must be empirically evaluated, and key layers may need to retain higher precision;
5. Operational complexity: fault recovery, load balancing, and canary releases under expert parallelism all require a dedicated inference scheduling layer.

7. Selection Decision Tree

Can you accept decode < 20 tokens/s?
├─ Yes → Tier 1: single GPU + large-memory CPU offload, budget ~100K–200K RMB
└─ No → Do you need concurrency?
        ├─ Low concurrency (< 5) → Tier 2: 8 GPUs + memory offload, budget ~1.5–2.5M RMB
        └─ Production-grade concurrency → Tier 3: 64-GPU supernode, budget in the tens of millions RMB

8. Summary

Local deployment of a trillion-parameter MoE is fundamentally an arithmetic problem of storage bandwidth and communication bandwidth:

• Weights start at 1.5TB, which determines that the lower bound cannot be low;
• The all-to-all communication of 896 experts determines that the upper bound must rely on a supernode;
• Every tier in between is a trade-off between these two constraints.

Figure out whether you want “it runs,” “it runs well enough,” or “it runs fast” first, then subtract from your VRAM and budget—that is more cost-effective than stacking GPUs from the start.