Local Deployment of Trillion-Parameter Models: Cost and VRAM Requirements Explained

1. Clearing Up a Common Misconception: Activation Ratio Is Not VRAM Ratio

When discussing local deployment of very large Mixture-of-Experts (MoE) models, the most commonly misread concept is the “activation ratio.” Take an MoE model with 2.8 trillion total parameters: it contains 896 experts, and each token activates an average of 16 of them, giving an expert activation ratio of roughly 1.79%.

Many people’s first reaction on seeing this number is: “If only 1.79% is activated, does that mean VRAM usage is also only 1.79%?” The answer is no. Two completely different resource dimensions must be distinguished here:

• VRAM capacity: Determined by the total parameter count. All 896 experts must reside in storage at all times, so that during inference the selected 16 can be called up at any moment based on routing results. You cannot predict which experts the next token will hit, so you cannot load only a subset.
• Compute consumption: Determined by the activated parameters. Only the activated experts actually participate in matrix multiplications, and only this part affects FLOPs and throughput.

In other words, MoE trades storage for compute: it enlarges the parameter count to increase model capacity, but keeps the per-token compute within an acceptable range through sparse activation. The price is that VRAM usage must be calculated based on total parameters.

At the architectural level, the backbone of such models typically consists of linear attention-style mechanisms (such as KDA, Kimi Delta Attention) combined with Attention Residuals, with gated MLA as one of the components. These structural designs affect compute efficiency and long-context capability, but they do not change the deployment prerequisite that “all experts must reside in memory.”

2. How Much VRAM Does 2.8 Trillion Parameters Actually Take?

2.1 Starting From the Theoretical Lower Bound

Suppose we convert all 2.8 trillion parameters to 4-bit storage. The bare data lower bound is:

2.8 × 10^12 × 4 bit ÷ 8 = 1400 GB

This is the ideal case — with no additional overhead whatsoever. But in reality, 4-bit quantization formats are not “pure 4-bit.”

2.2 The Hidden Overhead of Quantization Formats

Take the industry-common MXFP4 (weights) plus MXFP8 (activations) scheme as an example. According to the OCP microscaling specification, MXFP4 attaches one 8-bit scale factor for every 32 4-bit elements. Therefore the effective bit width per weight element is not 4 bit, but:

4 + 8/32 = 4.25 bit

Recalculating the weight footprint:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This 1487.5 GB is still only the weights themselves. A real deployment must also stack the following overheads:

• Non-4-bit tensors: Embedding layers, normalization layers, routing gates, output heads, etc. usually remain at higher precision (FP8/FP16). Their parameter counts are small, but their precision is high.
• Metadata and alignment: Tensor alignment, padding, descriptors, and so on.
• Runtime workspace: Temporary buffers, communication buffers, operator workspace.
• KV cache: In long-context scenarios, this part can reach hundreds of GB and grows linearly with concurrency and sequence length.

A useful reference point: an MoE model at the 671B total-parameter level takes about 685 GB for main parameters plus multi-token prediction (MTP) parameters in FP8 format; even with 4-bit weight quantization, the weights are about 335 GB, plus about 37 GB of activations, totaling roughly 400 GB just to run on 8 80GB accelerators (640 GB of VRAM).

The 2.8-trillion-parameter scale is more than 4 times that of 671B. Extrapolating roughly by proportion, even with an equivalent quantization strategy, the weights alone easily exceed 1.4 TB. Adding activations, KV cache, and runtime overhead, the VRAM lower bound for a full-featured deployment is far higher than what a single 8-GPU machine can provide.

2.3 An Estimation Table

Item	Estimated Value	Notes
Pure 4-bit weight lower bound	~1400 GB	Ideal value, no additions
MXFP4 weights (including scale)	~1487.5 GB	Calculated per OCP spec
Non-4-bit tensors + metadata	Tens of GB and up	Embeddings, gates, alignment, etc.
Runtime workspace	Tens of GB and up	Communication and operator buffers
KV cache	Scenario-dependent	Significant with long context and high concurrency
Total lower bound	Well over 1.5 TB	Leave ample headroom for real deployment

3. Why the Official Recommendation Is a Supernode With 64+ Accelerators

3.1 All-to-All Communication Is the Real Bottleneck

The performance bottleneck in large-scale MoE inference is often not per-GPU compute, but all-to-all communication in expert parallelism.

The inference flow is roughly as follows:

1. Each token passes through the gating network, which computes which 16 experts it should be routed to;
2. These experts are distributed across multiple accelerators, so activation values must be sent to the devices hosting the target experts;
3. After the target devices finish computing, the results are sent back to the original devices.

Since 896 experts are partitioned across dozens or even hundreds of GPUs, nearly every token’s activation values must cross device boundaries. Communication latency directly eats into compute — if interconnect bandwidth is insufficient, accelerators spend much of their time waiting, and utilization drops sharply.

3.2 The Value of a Supernode: Making the Communication Domain Larger

The core idea of a supernode is to use NVLink or equivalent high-speed interconnect technology to bring dozens of accelerators into a single high-bandwidth communication domain. The benefits are:

• Communication stays in-domain: Activation transfers between experts are completed within the domain, avoiding slow cross-node networks;
• Controllable latency: In-domain bandwidth is far higher than general-purpose networks, significantly reducing all-to-all latency;
• Better scalability: Experts can be distributed more evenly, making load balancing easier to achieve.

The official recommendation of 64+ accelerators forming a supernode is essentially finding a feasible solution between two constraints — VRAM capacity and communication bandwidth: VRAM must hold all experts, and bandwidth must sustain the all-to-all traffic.

4. A Decision Framework for Local Deployment

Based on the analysis above, evaluating whether a trillion-scale MoE model can be deployed locally can proceed in the following steps:

Step 1: Do the VRAM math

• Total parameters × equivalent quantization bit width ÷ 8 = weight lower bound
• Add non-quantized tensors, KV cache, and runtime overhead
• Reserve 20%–30% headroom for peaks

Step 2: Do the communication math

• Number of experts ÷ experts per GPU = communication domain size
• Estimate the volume of activation values moved per token
• Compare interconnect bandwidth against target throughput to judge whether it becomes a bottleneck

Step 3: Do the cost math

• Number of accelerators × per-GPU cost
• Additional investment in high-speed interconnect (supernode)
• Operational costs such as power, cooling, and data center retrofitting
• Compare TCO against on-demand cloud rental options

Step 4: Choose a path

• If pursuing full performance with sufficient budget: build a supernode cluster in-house
• If pursuing flexibility: use elastic cloud compute, scaling on demand
• If some accuracy loss is acceptable: consider more aggressive quantization schemes, but evaluate the resulting quality regression

5. Summary

Local deployment of trillion-parameter MoE models is essentially a dual-constraint optimization problem of VRAM capacity + interconnect bandwidth. The activation ratio only determines compute consumption, not VRAM usage; 4-bit quantization is also not simply “divide by 4” — scale factors, metadata, and KV cache all add overhead.

For most teams, running through an estimate using the framework in this article before purchasing hardware can often avoid the awkwardness of “buying it and not being able to run it” or “running it but with terrible throughput.” Calculating the VRAM lower bound, evaluating the communication bottleneck, and comparing the TCO of in-house versus cloud deployment are the starting points for a rational decision.