Deploying a Trillion-Parameter MoE Model Locally: VRAM Estimation and Three Hardware Tiers

1. Start with the Numbers: How Total Parameters Set the Deployment Floor

When planning on-premises deployment of an ultra-large model, the first step is never picking a GPU—it is calculating memory. Using a recently discussed trillion-parameter MoE architecture as a reference: total parameters are approximately 2.8 trillion, the MoE backbone contains 896 experts, 16 experts are activated per token, and the expert activation ratio is about 1.79%.

Two concepts must be kept separate. The expert activation ratio is not the same as the parameter activation ratio. The former determines the magnitude of compute consumed; the latter determines the per-forward-pass computation. VRAM usage must be calculated against total parameters, because all 896 experts’ weights must reside in addressable storage, with the routing mechanism fetching the 16 activated ones at inference time.

The architecture typically alternates linear-attention-style components with feed-forward expert layers. Attention parts are compute-heavy but small in parameter count; expert parts are parameter-heavy but only a small fraction is active per pass. This structural trait shapes every subsequent deployment strategy: keep dense components on high-bandwidth devices, and place sparse expert weights in large-capacity storage.

Quantization-Aware VRAM Baseline

At a bare-minimum 4-bit quantization:

• 2.8 T parameters × 4 bit ÷ 8 ≈ 1,400 GB

Real 4-bit floating-point formats are not pure 4-bit. Taking the commonly used MXFP4 as an example, every 32 four-bit elements carry an 8-bit scale factor. Under open-compute conventions, weights plus scales come to roughly:

• 2.8 T × 4.25 bit ÷ 8 ≈ 1,487.5 GB

That covers weights only. Non-4-bit tensors (embeddings, normalization parameters, router gating), metadata, memory-alignment padding, runtime workspaces, and KV Cache are all still missing from the count.

For comparison, a 670B-class MoE model at FP8 needs roughly 685 GB for main and MTP parameters; even with 4-bit weights plus low-bit activations (~335 GB weights + ~37 GB activations), the total of ~400 GB barely fits on eight 80 GB accelerators (640 GB). A trillion-parameter model is more than 4× larger, so a full deployment’s memory floor is locked above 1.5 TB, with another 20–30% headroom needed for activations, communication buffers, and KV Cache.

The Interconnect Bandwidth Constraint

Another easily overlooked factor is interconnect bandwidth. Large-scale MoE inference is extremely sensitive to all-to-all communication: 896 experts spread across many devices mean every routed token must shuttle its activations to the target expert’s device and back. Communication latency directly erodes compute utilization. Official guidance typically recommends super-nodes of 64+ accelerators with high-bandwidth interconnects, keeping expert movement inside a single communication domain. For on-premises setups, this means simply stacking GPUs does not equal usability—the interconnect topology and communication domain size are part of the solution.

2. Tier 1: “Just Make It Run”—Speed Is Not the Priority

The sole goal here is to load roughly 1.5 TB of weights into addressable memory and get the model to stably generate tokens, even at single-digit tokens per second. This tier suits precision validation, pipeline verification, and application-layer integration testing.

The most straightforward approach is GPU + CPU heterogeneous inference: the shared attention components in each MoE layer stay on the GPU for high-bandwidth compute; the 896 sparse experts, which dominate storage, are offloaded to CPU memory served by a large-memory server. Mature open-source inference frameworks already support this expert-offload pattern. Their published reference config runs a 670B-class 4-bit MoE (~377 GB weights) on a dual-socket server CPU plus a single consumer flagship GPU with acceptable time-to-first-token.

Scaling that ratio to a trillion-parameter model, a typical Tier 1 build looks like:

• Memory: 2 TB+ DDR5 ECC (1.5 TB weights + system/runtime overhead); 3 TB is safer
• GPU: 1–2 cards with 24–48 GB VRAM each, handling only attention layers and KV Cache
• Storage: 4 TB+ NVMe SSD for weight loading and page swapping
• CPU: Dual-socket high-core-count server platform; memory bandwidth matters more than clock speed

Cost range: roughly USD 20,000–43,000 (varies with memory capacity and platform age). The bottleneck is clear: CPU memory bandwidth and PCIe bandwidth. Measured throughput is often single-digit tokens/s, and long-context scenarios squeeze more with KV Cache pressure. The value is in validating model capability and business feasibility at minimum cost, not serving real traffic.

3. Tier 2: Single User, 25 Tokens/s Output

Twenty-five tokens per second is near human reading speed and marks the line between “usable” and “frustrating.” Moving from single digits to 25 t/s means shifting expert compute back onto GPUs while keeping expert weights efficiently accessible.

VRAM demand returns to the ~1.5 TB range. Three paths are available:

Path A — Multi-GPU, high-VRAM single node. Eight 80 GB cards (640 GB total) cannot hold all weights and still require CPU offload—essentially an enhanced Tier 1. To keep experts in GPU memory you need 16–24 × 80 GB accelerators, or fewer cards with larger VRAM. Sixteen cards (1,280 GB) still need partial offload; twenty-four (1,920 GB) covers it comfortably. System cost: roughly USD 280,000–560,000.

Path B — Super-node appliance. Vendor-supplied rack-scale super-nodes use high-bandwidth interconnects to scale the communication domain to dozens of cards, keeping per-token all-to-all overhead amortized. Delivered as full racks, costs run into the multi-hundred-thousand-dollar range. Pros: communication efficiency and operational uniformity. Cons: high procurement threshold.

Path C — Deeper quantization compression. Applying sub-4-bit or mixed-precision quantization to non-critical layers can shrink weights to ~1.2 TB and meaningfully lower the VRAM bar. But quantization depth has a floor: weights below 4-bit paired with 8-bit activations are roughly the current acceptable precision boundary—pushing further degrades long-context and chain-of-thought quality noticeably. Any quantization plan must be validated benchmark by benchmark, not by file size alone.

KV Cache planning is critical at this tier. With 32K–128K context lengths at 25 t/s, KV Cache can consume tens of gigabytes and must be reserved separately in the parallelism strategy. Batch size 1 with PagedAttention-style memory management is standard practice.

4. Tier 3: 10 Concurrent Users, 25+ Tokens/s Each

Ten users at 25 t/s each means total throughput above 250 t/s, with quantization no lower than 4-bit. The fundamental shift here is from “fits in memory” to “can compute fast enough.”

MoE compute scales with activated parameters, but 10 concurrent requests enlarge the active-expert set in each batch—different requests route to different experts, so more experts participate than in a single-request scenario. Two consequences follow:

1. More active expert copies must reside in memory, or swapping frequency increases
2. All-to-all traffic grows roughly linearly with concurrency; insufficient communication domains let latency swallow the added compute

Engineering mitigations include:

• Higher expert parallelism: distribute 896 experts across more devices to reduce per-card expert count and increase routing flexibility
• Larger communication domains: use high-bandwidth super-nodes so expert movement stays in-domain
• Continuous batching and request scheduling: merge tokens from different requests into the same batch to raise utilization
• Tiered KV Cache: hot data in VRAM, warm data in high-bandwidth memory, controlling VRAM growth with a分级 strategy

A typical build uses 32–64 × 80 GB accelerators in a super-node, landing in the USD 700,000–1,400,000 range (or similar totals with fewer, higher-VRAM cards). Supporting infrastructure—high-speed storage for weight loading and logging, redundant power, and cooling upgrades—typically adds another 10–20%.

5. Three Often-Overlooked Hidden Costs

Weight distribution and versioning. Trillion-parameter weight files are 1.5 TB each. Every model update means a full redistribution and reload. On-premises operations need a distribution pipeline and integrity checks, or version management becomes an operational burden.

Benchmarking and tuning effort. All the speed and memory figures above are estimates based on publicly available architecture details. Real performance must be verified against standard benchmarks: time to first token, decoding throughput, long-context degradation, and P99 latency under concurrency. Expect 1–2 inference-framework-savvy engineers spending several weeks on this.

Power and cooling. Dozens of high-TDP accelerators can draw tens of kilowatts. The facility needs matching electrical capacity and liquid or forced-air cooling. In smaller data rooms, this retrofit cost often exceeds the hardware line item itself.

6. Decision Guide

Tier	Goal	Memory/VRAM Threshold	Cost Range (USD)	Typical Use Case
1	Just run	2–3 TB RAM + modest GPU	~20K–43K	Validation, integration, demos
2	25 t/s single user	~1.5 TB VRAM	~280K–560K	Heavy single-user, internal tools
3	10 users × 25 t/s	3 TB+ VRAM + super-node	~700K–1.4M	Team service, production traffic

A critical reminder: before official weights and technical reports are released, every number here is an estimate based on public architecture information. Real quantization overhead, actual router distributions, and framework-hardware compatibility will all shift results. The pragmatic path is to start with a Tier 1 minimum viable environment, run real weights through benchmarks, then decide on upgrading to Tier 2 or 3 based on measured data. For most teams, Tier 1 plus on-demand cloud API calls in a hybrid model is the highest-value starting point.