Local Deployment of Trillion-Parameter MoE Models: Tiered Estimation of GPU Memory and Hardware Costs

As Mixture-of-Experts (MoE) architectures become the mainstream choice for ultra-large models, more teams are evaluating the feasibility of moving models back into their own data centers. But when a model’s total parameter count reaches the trillion scale, deployment is no longer as simple as “buy a GPU with a lot of memory.” It becomes a systems engineering problem involving memory capacity, quantization formats, communication bandwidth, and concurrency capability. This article uses a publicly referenced trillion-parameter MoE architecture as a baseline (approximately 2.8T total parameters, 896 experts, 16 experts activated per token) and presents a reusable estimation methodology and cost framework across three requirement tiers.

A necessary disclaimer upfront: the speeds, memory figures, and hardware counts below are engineering estimates based on publicly available architecture information. Actual numbers must be validated against benchmarks once weights are released. Any performance claim made without real-world testing should not be trusted.

1. First, Understand What Trillion Parameters Actually Mean for GPU Memory

1.1 Total Parameters Determine Memory; Activation Ratio Only Determines Compute

The most common pitfall in MoE deployment is mistaking the “expert activation ratio” for a “parameter activation ratio.” Take a model with 2.8T parameters and 896 experts, where 16 experts are activated per token. The expert activation ratio is roughly 1.79%. But this does not mean only 1.79% of parameters need to fit in memory — all 896 experts must reside in storage at all times, and during inference the router pulls the relevant expert weights for computation.

In other words:

• Memory capacity is determined by total parameters; you must fit all experts.
• Compute consumption is determined by activated parameters; each forward pass only computes 16 experts.
• Communication overhead is determined by expert distribution; tokens must be shuttled back and forth across multiple GPUs.

This distinction is the starting point for all cost estimation.

1.2 The “Hidden Overhead” of Quantization Formats

Assume the model undergoes quantization-aware training from the SFT stage, with weights in 4-bit MXFP4 and activations in MXFP8. Many people would simply calculate: 2.8T × 4 bit ÷ 8 = 1400 GB. But actual occupancy is higher than this number.

According to the OCP specification, MXFP4 attaches one 8-bit scale factor for every 32 4-bit elements. The equivalent bit width per weight element works out to:

4 bit + 8 bit / 32 = 4.25 bit
2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

And that is only the raw weight data. A complete deployment must also account for:

• Non-4-bit tensors (embedding layers, normalization layers, routing gates, etc., which typically retain higher precision);
• Framework metadata and tensor alignment padding;
• Runtime workspace;
• KV Cache (which grows linearly with concurrency and context length).

1.3 A Familiar Point of Reference

A mid-scale MoE helps make this more intuitive: a model with roughly 670B primary parameters plus 14B multi-token prediction parameters has about 685 GB of weights in FP8. Even compressed to 4-bit quantization, weights still require roughly 335 GB, plus about 37 GB of activations, totaling about 400 GB to run on a single 8-GPU high-end server (80 GB per GPU, 640 GB total).

The 2.8T parameter scale is more than 4× that. This means the lower bound for a full-fidelity deployment already exceeds what a single 8-GPU machine can handle, and must move into the territory of multi-node, multi-GPU super-nodes.

2. Hardware Tiers for Three Requirement Levels

Tier 1: Just Make It Run — Speed Doesn’t Matter

Goal: Validate model usability, build functional demos, get the inference pipeline working end to end.

Estimation logic: This tier only requires that weights fit and a single forward pass can complete. Using the ~1487.5 GB of 4-bit weights as a baseline, adding non-quantized tensors, metadata, and a minimal workspace, actual requirements land in the 1.6–1.8 TB memory range. Reserving a minimal KV Cache (single request, short context), a memory pool of around 2 TB is a reasonable starting point.

Hardware form factor: This can be composed of multiple 8-GPU servers. With 80 GB per GPU, roughly 24–32 accelerators are needed. At this tier, interconnect bandwidth is not a priority; expert distribution can be relatively concentrated, and some experts may even traverse slower cross-node channels — at the cost of high latency and low throughput. A single request might wait a long time before the first token appears.

Cost magnitude: The primary cost at this tier is total memory capacity, not interconnect. Estimated at high-end accelerator full-system procurement, the overall investment is in the millions of dollars. For short-term validation, renting multi-node multi-GPU instances on an hourly basis is more cost-effective.

Caveats: Running is not the same as being usable. Inference frameworks at this tier often need CPU offload or disk paging to compensate for memory shortfalls, resulting in a very poor experience. This tier is suitable for validation only.

Tier 2: Single-User, 25 tokens/s Output Speed

Goal: A single developer gets a near-interactive experience.

Estimation logic: A decode speed of 25 tokens/s imposes two hard constraints on a trillion-parameter MoE.

First is compute. The decode phase is memory-bound: every generated token requires reading the activated 16 expert weights from memory once. Activated parameters are roughly 2.8T × 1.79% ≈ 50B, which at 4-bit means about 26 GB of effective reads, plus attention and routing overhead. Completing all computation and memory access for one token within 40ms puts pressure on both per-GPU bandwidth and compute, typically requiring multi-GPU parallelism to distribute the load.

Second is communication. The 16 experts are spread across different GPUs, and every token requires an all-to-all activation transfer. If cross-node communication runs over a standard network, latency will directly eat into compute. Therefore, this tier must keep the communication domain small and fast — prioritizing expert parallelism within a single node or a single high-speed interconnect domain.

Hardware form factor: A single machine with high interconnect density — 8 or 16 GPUs — is recommended as one communication domain, using NVLink-class interconnect to keep all-to-all latency manageable. On the memory side, since concurrency is only 1, KV Cache pressure is minimal. The memory baseline from Tier 1 can be reused, but more budget should go toward interconnect and per-GPU performance.

Cost magnitude: One step above Tier 1, with the main increment in high-speed interconnect backplanes and system integration. For individual developers, a more realistic path is renting multi-GPU instances that support high-speed interconnect, paying by usage time to avoid a large one-time investment.

Validation method: After weights are released, measure Time to First Token (TTFT) and decode throughput (TPS), and observe the proportion of all-to-all communication. If communication time exceeds 30% of total per-token time, the interconnect configuration is insufficient.

Tier 3: 10 Concurrent Users, Each Still Getting 25+ tokens/s

Goal: Shared use by a small team, with quantization no lower than 4-bit.

Estimation logic: The difficulty at this tier is not weights but KV Cache and batch scheduling.

• With 10 concurrent requests decoding simultaneously, KV Cache occupancy grows linearly with concurrency. In long-context scenarios, a single request’s KV Cache may reach several GB, meaning tens of GB of additional memory for 10 concurrent requests — this must be reserved separately.
• Batching amplifies compute demand: although MoE expert activation is per-token independent, different tokens in a batch hit different experts, so the actual activated expert set is the union — increasing both memory access and communication volume.
• Maintaining 25 tokens/s per request means total throughput must reach 250 tokens/s, placing high demands on decode scheduling and continuous batching capability.

Hardware form factor: Memory requirement = weight baseline + tens of GB of KV Cache reservation + workspace. The overall memory pool should have sufficient headroom above 2 TB. Meanwhile, the communication domain must cover all GPUs participating in parallelism, making super-node-class interconnect (scaling the communication domain to dozens of GPUs) a necessity. This is precisely why the official recommendation calls for “64 or more accelerators forming a super-node” — large-scale MoE inference is extremely sensitive to all-to-all bandwidth, and only by enlarging the communication domain can expert transfers stay within the domain.

Cost magnitude: This is the most expensive of the three tiers, with hardware investment reaching the tens of millions of dollars. The corresponding cloud rental approach also requires multi-node, multi-GPU clusters with high-speed interconnect, with significant daily/monthly costs. It is advisable to first validate scheduling efficiency at small concurrency (e.g., 2–4 concurrent) and then extrapolate linearly.

Key tuning points:

1. Matching expert parallelism degree with the communication domain: Expert placement strategy should keep frequently co-occurring experts in the same domain.
2. KV Cache quantization and management: Apply low-bit quantization to KV Cache within acceptable precision bounds, or use paged attention to reduce fragmentation waste.
3. Continuous batching and scheduling: Use a mature inference framework for request-level scheduling to avoid tail latency caused by static batching.

3. A Reusable Estimation Workflow

Abstracting the analysis above into a workflow, any trillion-parameter MoE can follow it:

Step 1 Weight memory = total parameters × equivalent bit width ÷ 8
       Equivalent bit width = quantization bits + scale bits / group size
Step 2 Total memory = weight memory × (1 + non-quantized tensor factor)
                + metadata and alignment reservation
                + workspace
                + KV Cache (concurrency × context length)
Step 3 Determine communication domain: activated experts × per-expert size — does it fit within a single node?
       If not → super-node interconnect is mandatory
Step 4 Determine compute: target TPS × per-token memory access vs available bandwidth
Step 5 Use real-world benchmarks to regress and correct all coefficients above

4. Conclusion

Local deployment of a trillion-parameter MoE is essentially a three-variable equation of “memory × bandwidth × interconnect.” Tier 1 solves “does it fit,” Tier 2 solves “is it fast enough,” and Tier 3 solves “can it handle concurrency.” The cost difference across the three lies not in the weights themselves but in interconnect density and scheduling capability. For most teams, the lowest-risk path is to first use multi-node multi-GPU cloud instances for tiered validation, obtain real benchmark data, and then decide whether to build on-premises. Once weights and test reports are officially released, be sure to verify every estimation coefficient in this article against measured data before locking in the final configuration.