Local Deployment of Trillion-Parameter Models: VRAM Thresholds and Full Cost Analysis

1. Understanding Trillion-Parameter Models: What Actually Consumes VRAM

Before discussing local deployment, it is essential to clear up a common misconception: the activated parameter ratio does not equal the VRAM usage ratio.

Consider a Mixture of Experts (MoE) architecture with 2.8 trillion total parameters. It contains 896 experts, and only 16 are activated per token during inference, giving an expert activation ratio of approximately 1.79%. Many people see this number and intuitively assume “VRAM only needs 1.79% of total parameters”—that is incorrect.

Here is how VRAM consumption actually works for MoE:

• All expert weights must reside in storage at all times. During inference, the routing network determines which experts participate in computation, but every expert’s parameters must remain addressable.
• The activation ratio only affects compute consumption, not the total weight storage. Routing to 16 experts means those 16 experts’ weights are read and computed, but the remaining 880 experts’ weights still occupy storage space.

In other words, MoE is an architecture that “trades storage for compute”—it pays the cost of larger parameter storage so that each forward pass only computes a small fraction of parameters, achieving stronger model capability at equivalent compute.

Beyond expert weights, several other VRAM overheads must be accounted for:

Overhead Category	Description
Non-quantized tensors	Embedding layers, normalization layers, routing networks, etc. typically maintained at higher precision
Metadata and alignment	Tensor alignment, memory pool metadata, framework runtime overhead
KV Cache	Proportional to context length and concurrency; non-negligible in long-context scenarios
Runtime workspace	Operator temporary buffers, communication buffers, etc.

Combined, these typically add another 10%–20% of VRAM demand on top of the weights.

2. Quantization Format Determines the Storage Floor

Quantization is the primary means of lowering the VRAM threshold. Among current mainstream low-precision formats, 4-bit quantization is widely adopted because it strikes a good balance between precision loss and compression rate.

Using 2.8 trillion parameters as an example, here are estimates at several levels:

Pure theoretical lower bound (4-bit, no additional overhead):

2.8T × 4 bit ÷ 8 = 1400 GB

Accounting for quantization scale overhead:

Common 4-bit formats in the industry (such as MXFP4 under the OCP specification) attach one 8-bit scale per 32 4-bit elements. Converted:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

Adding non-quantized tensors, metadata, alignment, and runtime overhead:

In actual deployment, overhead beyond weights typically falls in the 100–200 GB range. Adding KV Cache (which depends on context length and concurrency), the VRAM floor for a full deployment lands roughly in the 1.6–1.8 TB range.

For comparison: a 671B-parameter MoE model requires approximately 685 GB for main parameters plus multi-token prediction parameters in FP8 format; even with 4-bit quantization, weights plus activations require about 400 GB to run on 8 high-bandwidth accelerators (640 GB total VRAM). A trillion-parameter model is more than 4× that size, so the deployment floor rises accordingly.

3. Supernodes: The Inevitable Choice for Large-Scale MoE Inference

VRAM is only the first threshold—communication is the second.

With 896 experts distributed across multiple accelerators, each token’s activations must be moved to the card hosting the target expert for computation after routing, then moved back. This all-to-all communication pattern is extremely sensitive to interconnect bandwidth—if the communication domain spans low-speed interconnects (such as standard Ethernet), communication latency directly eats into compute, and GPU utilization can drop below 20%.

Therefore, official-grade deployment plans typically recommend supernodes composed of 64 or more accelerators, using high-bandwidth interconnects (such as NVLink or equivalent high-speed buses) to enlarge the communication domain so that activation transfers between experts stay within a single communication domain.

This also explains why the hardware cost of large-scale MoE inference is not simply “total VRAM divided by per-card VRAM”—interconnect bandwidth and topology also determine whether usable performance can be achieved.

4. Tiered Deployment Options and Cost Estimates

Not every scenario requires a full deployment. Based on performance expectations, three tiers can be defined:

Tier 1: It Runs, Speed Doesn’t Matter

Goal: Fit nearly 1500 GB of weights into addressable storage so the model can emit tokens, even at single-digit tokens per second.

Approach: GPU + CPU heterogeneous inference. Each MoE layer has a shared attention component (compute-intensive) that stays on the GPU with high-bandwidth compute; the 896 sparse experts account for most storage but only 16 are activated at a time, so they are offloaded to CPU memory and fed by a large-memory server.

Typical configuration:

• Dual-socket server CPU (high memory channel count)
• 512 GB–1 TB system memory
• 1–2 high-VRAM consumer or professional-grade GPUs

Performance expectations: Prefill phase can reach hundreds of tokens/s; decode phase around 10–20 tokens/s (limited by expert selection strategy and memory bandwidth).

Cost range: Approximately $7,000–$21,000 USD (depending on memory capacity and GPU model).

Tier 2: Single-Machine Multi-GPU, Targeting Usable Speed

Goal: Decode speed of 50–100 tokens/s to meet interactive application requirements.

Approach: Use 8–16 high-bandwidth accelerators in a single-machine communication domain, with expert weights distributed across each card’s VRAM and all-to-all communication completed through high-bandwidth interconnects. Some infrequently activated experts can be offloaded to host memory as a second-level cache.

Typical configuration:

• 8–16 high-VRAM accelerators (80 GB-class per card)
• High-speed inter-card interconnect
• 1–2 TB system memory

Performance expectations: Decode can reach 50–100 tokens/s, depending on the locality of expert routing and the degree of communication optimization.

Cost range: Approximately $140,000–$420,000 USD.

Tier 3: Full Supernode Deployment

Goal: Decode exceeding 200 tokens/s with support for high-concurrency serving.

Approach: 64+ accelerators forming a supernode, with NVLink-class interconnects enabling a large communication domain, experts evenly distributed, and all-to-all communication staying within the domain. Combined with a hybrid strategy of expert parallelism, tensor parallelism, and pipeline parallelism.

Typical configuration:

• 64+ high-bandwidth accelerators
• Supernode-level interconnect (NVLink Switch or equivalent)
• Multi-TB system memory
• High-speed distributed storage

Performance expectations: Decode 200+ tokens/s, supporting multiple concurrent requests.

Cost range: Millions of USD.

5. Beyond Hardware: Overlooked Hidden Investments

Hardware procurement is only the beginning. Several items in actual deployment are easily underestimated:

1. Inference framework adaptation: Trillion-parameter MoE requires dedicated inference engines that support expert parallelism, dynamic routing, and communication optimization. General-purpose frameworks often require deep customization.
2. Power and cooling: A full deployment can consume tens of kilowatts; data center retrofitting and cooling systems represent a significant expense.
3. Operational complexity: Troubleshooting across multiple cards and machines, VRAM fragmentation management, and routing load balancing all require dedicated operations tools and accumulated experience.
4. Quantization precision tuning: Although 4-bit quantization compresses VRAM, it may introduce precision loss, requiring calibration and compensation on specific tasks.

6. Summary: First Ask “How Fast,” Then Ask “How Much”

The local deployment cost of a trillion-parameter model is fundamentally a question of performance expectations determining hardware configuration:

• If you only need to validate feasibility or run offline batch inference, a heterogeneous inference approach can get you started for under $10,000;
• If you need to support interactive applications, a single-machine multi-GPU approach requires investment in the hundreds of thousands of dollars;
• If you need to provide high-concurrency online serving, supernode deployment costs are in the millions of dollars.

Before getting started, clarify three questions: How fast does decode need to be? How much concurrency must be supported? What is the context length? These three answers determine the minimum requirements for VRAM, bandwidth, and interconnect topology—and ultimately the final cost range.

A trillion-parameter model is not a toy you can “run with a few GPUs,” but it is not out of reach either—the key is choosing the right deployment tier based on actual requirements rather than blindly pursuing a full configuration.