Local Deployment of Trillion-Parameter MoE Models: VRAM Estimation, Hardware Selection, and Cost Breakdown

When a model reaches the trillion-parameter scale, local deployment is no longer a matter of “buying a few GPUs.” It becomes a systems engineering problem involving storage capacity, interconnect bandwidth, quantization formats, and inference scheduling. This article uses a typical trillion-parameter sparse mixture-of-experts (MoE) architecture as a reference to derive hardware requirements and cost ranges from first principles, then outlines practical deployment paths across three tiers. Specific weight details and official architecture specifics are subject to the official release; the throughput, memory, and hardware figures below are estimates based on publicly available information and general engineering experience. They should be verified through benchmarking once official weights are available.

A Critical Starting Point: MoE Memory Scales with Total Parameters

Trillion-scale models generally adopt sparse MoE architectures: the model contains a very large number of expert networks, but each token only routes to a small subset of them during inference. In one typical configuration, the model has roughly 2.8 trillion total parameters and 896 experts, with only 16 experts activated per token—an expert activation ratio of about 1.79%.

Two concepts are often confused and should be clearly separated:

• Expert activation ratio: the fraction of experts selected by the router, roughly 1.79% in this case.
• Parameter activation ratio: the fraction of total parameters actually involved in a single forward pass. This is usually higher than the expert activation ratio because dense components such as the attention backbone and shared layers are used for every token.

For deployment cost, what determines the memory floor is the total parameter count, not the activation ratio. The reason is straightforward: all 896 experts’ weights must reside in addressable storage at all times. During inference, the router fetches the weights of the 16 selected experts for computation. The activation ratio affects compute cost and memory bandwidth pressure, but not the total amount of weight storage. This is the starting point for all capacity planning.

Precise Weight Volume Calculation: Don’t Just Look at “4-bit”

This model incorporates quantization-aware training from the supervised fine-tuning stage, using MXFP4 for weights and MXFP8 for activations. A naive calculation—2.8 trillion × 4 bits ÷ 8—gives roughly 1400 GB as the raw lower bound, but that number is optimistic.

MXFP4 is a block-scaled format: every 32 4-bit elements carry an additional 8-bit scale factor. Using the microscaling specification from the Open Compute Project (OCP), the effective bits per weight element is:

Effective bits per element = 4 bits + (8 bits / 32) = 4.25 bits
Weight volume ≈ 2.8e12 × 4.25 ÷ 8 ≈ 1487.5 GB

This only accounts for weight elements and scale factors. It does not yet include:

• Non-4-bit tensors, such as some normalization layers and embedding layers that may retain higher precision;
• Metadata and tensor alignment padding;
• Runtime workspace memory, including temporary buffers and routing intermediates;
• KV cache, which grows linearly with concurrency and context length.

As a reference point, the previous generation of large MoE models (roughly 671B total parameters) in FP8 requires about 685 GB for main parameters plus multi-token prediction parameters. Even with 4-bit weight quantization, weights still occupy about 335 GB, plus about 37 GB for activations—around 400 GB total—to run on a single 8-GPU high-bandwidth server with roughly 640 GB of aggregate VRAM. A trillion-scale model has more than four times that volume, which puts the realistic VRAM floor for full deployment at approximately 1.5 TB.

Interconnect Topology: The Underappreciated Cost Driver

The official deployment guidance for models of this class recommends a supernode of 64 or more accelerators. This is not conservative hardware advice; it follows directly from the communication patterns of MoE inference.

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts spread across multiple GPUs, each token’s activations must be transferred to the GPUs hosting the selected experts, computed, and transferred back. Communication latency directly erodes compute utilization throughout this process. A supernode expands the communication domain through high-bandwidth interconnect such as NVLink or equivalent high-speed fabrics, keeping expert-to-expert transfers within the same domain as much as possible and bringing communication overhead down to acceptable levels.

For self-hosted deployments, this means the cost structure includes not only accelerators but also switches, high-speed cables, rack power delivery, and cooling. Coarse estimates suggest interconnect and supporting infrastructure can account for 20% to 30% of the total hardware budget in supernode configurations, and rack-level power density often requires liquid cooling or forced-air upgrades.

Tiered Deployment Paths

Tier 1: It Runs, But Don’t Expect Speed

The only goal here is to fit nearly 1.5 TB of weights into addressable storage and have the model produce tokens reliably, even at single-digit tokens per second. The core approach is GPU-CPU heterogeneous inference.

Each MoE layer has a shared attention component—in this case, a linear attention backbone with a gated multi-head latent attention component. This part is compute-intensive and stays on the GPU. The 896 sparse experts, while accounting for the overwhelming majority of storage, are activated only 16 at a time and can be offloaded entirely to CPU memory on a high-capacity server.

A common heterogeneous inference approach in the industry, based on ideas similar to the KTransformers project, has published reference numbers on the previous-generation 671B model: a dual-socket high-core-count server CPU paired with a single flagship consumer GPU running approximately 377 GB of 4-bit quantized weights achieved over 200 tokens/s in prefill and about 14 tokens/s in decode. These figures come with expert-selection caveats and will fluctuate with routing distribution.

Mapped to a trillion-scale model, the rough hardware outline for this tier is:

• Memory: at least 2 TB DDR5, with 4 TB recommended for headroom;
• GPU: 1–2 high-VRAM cards for attention and routing computation;
• Storage: 2 TB or more NVMe SSD for weight loading and paging;
• Budget range: roughly 100,000–250,000 RMB, excluding data center infrastructure.

For performance expectations, decode speed may drop to single-digit tokens per second, making this tier suitable only for offline batch processing, capability validation, and prototyping.

Tier 2: Usable Throughput

Building on Tier 1, adding more GPUs, keeping hot experts resident in VRAM, and introducing expert caching with prefetching can lift decode throughput to tens of tokens per second. This tier typically requires an 8-GPU high-bandwidth server paired with 2–4 TB of system memory. Budget falls in the 800,000–2,000,000 RMB range. It suits internal tools, knowledge-base Q&A, and other moderate-concurrency workloads.

Tier 3: Full-Scale Supernode

Following the official recommendation of a 64+ accelerator supernode with high-speed interconnect and liquid-cooled racks delivers full throughput capability. Hardware investment for this tier is typically in the tens of millions of RMB, plus dedicated data center modifications and an operations team. It is only justified for organizations with clearly defined large-scale inference needs.

Cost Reference Table and Decision Guidance

Tier	Accelerator Count	System Memory	Weight Placement	Budget Range	Use Case
Tier 1	1–2 GPUs	2–4 TB	All experts offloaded to CPU	100K–250K RMB	Validation, offline batch
Tier 2	8 GPUs	2–4 TB	Hot experts in VRAM	800K–2M RMB	Moderate-concurrency internal services
Tier 3	64+ GPU supernode	Distributed	Fully distributed across VRAM	Tens of millions RMB	Large-scale online inference

Key decision points:

1. Plan capacity before compute. The primary constraint for trillion-scale MoE is always storage. First confirm that weights plus KV cache can fit, then evaluate GPU models and interconnect.
2. Quantization formats must be converted according to spec. Block-scaled format scale overhead is not negligible. Estimating at 4.25 bits per element is closer to reality than 4 bits.
3. Heterogeneous deployment is the low-cost validation path. Before committing to a supernode, validate outputs and routing distributions with GPU-CPU heterogeneous inference to generate real data for subsequent capacity planning.
4. Reserve headroom for KV cache and concurrency. Long contexts and high concurrency significantly increase memory usage. Capacity planning should reserve at least 30% buffer on top of the weight volume.
5. Re-verify against official weights after release. All figures in this article are engineering estimates. Once official weights are available, calibrate against actual load size, prefill/decode throughput, and routing hit rates.

In summary, the local deployment threshold for trillion-parameter MoE models is defined by two constraints: the storage floor determined by total parameter count, and the communication ceiling determined by expert routing. Understanding these two constraints is more valuable than any single hardware figure.