Local Deployment of a Trillion-Parameter MoE Model: A Full Breakdown of Three Hardware Tiers and Costs

As of late July 2026, the complete weights and technical report for a certain frontier model have not yet been officially released; the official timeline states they will be published before July 27. Apart from the architecture description and API pricing already disclosed by the vendor, all figures below—throughput, VRAM usage, hardware configurations—are engineering estimates based on public information. Once the weights are officially released, each item will need to be verified through benchmarking. The goal of this article is not to provide a “standard answer” but to offer a reusable estimation framework—so that when you receive any trillion-parameter MoE model, you can calculate for yourself which GPUs to buy and how much to spend.

---

1. Start with the Math: What 2.8 Trillion Parameters Mean for VRAM

The model has approximately 2.8 trillion total parameters and uses a MoE (Mixture of Experts) architecture: of 896 experts, 16 are activated per token, giving an expert activation ratio of about 1.79%. There is a point here that is very easy to misread—the expert activation ratio is not the same as the parameter activation ratio. VRAM usage for MoE deployment must be calculated based on total parameters; the activation ratio only affects compute consumption, not storage requirements.

The reason is straightforward: you must first load all 896 experts into storage, then dynamically route to 16 of them during inference based on the router network. Experts do not disappear from VRAM just because they were “not selected this time.”

The architecture backbone consists of KDA (a linear attention variant) and Attention Residuals, with Gated MLA as one of its components. The model introduced quantization-aware training starting from the SFT stage, using MXFP4 for weights and MXFP8 for activations.

1.1 Raw Lower Bound of the Weights

Converting 2.8 trillion parameters to 4-bit gives a raw data lower bound of approximately:

2.8e12 × 4 bit ÷ 8 = 1400 GB

But MXFP4 is not “pure 4-bit.” According to the OCP specification, every 32 4-bit elements carry an additional 8-bit scale factor. Therefore the actual footprint of weight elements plus scales is:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This is only the weights themselves. Not yet counted are: non-4-bit tensors (such as norm, bias, router weights), metadata, memory alignment padding, runtime workspace, and KV Cache.

1.2 A Useful Reference Point

Take a widely deployed 671B MoE model as a reference: in FP8 format, 671B main parameters plus 14B MTP parameters total about 685 GB; even with 4-bit weight quantization, it still requires about 335 GB of weights plus 37 GB of activations—roughly 400 GB total—to run on 8 80GB accelerators (640 GB VRAM).

The 2.8 trillion parameter model is more than 4 times the size of 671B. This determines the lower bound for a full deployment.

1.3 Official Recommendation and the Communication Bottleneck

The vendor recommends deploying this model on a supernode composed of 64 or more accelerators. This is not “over-provisioning”—it is an architectural choice forced by all-to-all communication.

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth: 896 experts are distributed across multiple cards, and after each token is routed, activations must be moved to the card hosting the target expert for computation, then moved back. Communication latency directly eats into compute. The value of a supernode lies in using high-speed interconnect (such as NVLink or an equivalent solution) to enlarge the communication domain, so that this movement never leaves a single communication domain.

With this understood, the cost differences among the three deployment tiers become clear: the gap is not just in “the number of cards” but in “the grade of interconnect.”

---

2. Cost Breakdown of the Three Deployment Tiers

The following estimates assume: weights in 4-bit quantization-aware training format, KV Cache in FP8, and a context length of 32K. Actual figures will vary with inference framework, batching strategy, and context length.

Tier 1: It Runs, Speed Doesn’t Matter

Goal: The model loads and produces output; speed is not a constraint.

VRAM requirements:

• Weights: about 1488 GB
• Runtime overhead: about 100–150 GB
• KV Cache (single request, short context): about 20–40 GB
• Total: about 1600–1680 GB

Hardware options:

• Option A: 24 × 80GB accelerators (1920 GB VRAM), interconnected via PCIe or standard Ethernet
• Option B: 21 × 96GB accelerators (2016 GB VRAM)

Cost range:

• With previous-generation 80GB cards, priced according to the second-hand/inventory market, the full system cost is in the range of several hundred thousand RMB
• With new 96GB cards, the cost rises significantly

Key limitation: This tier will most likely only run batch size = 1, with output speed possibly in the single-digit tokens/s range. It is suitable for functional validation, weight integrity checks, and quantization quality evaluation—not for any production scenario.

Engineering note: Even for “it just needs to run,” confirm that the inference framework supports expert parallelism. If the framework only supports tensor parallelism, 896 experts cannot be distributed across cards, and VRAM will immediately overflow.

---

Tier 2: Single-User, Output Speed ≥ 25 tokens/s

Goal: Single-user interactive use with output speed of 25 tokens/s or higher, and quantization no lower than FP4.

This is the most easily underestimated tier. Many assume “single-user just means batch size = 1,” but 25 tokens/s output for a trillion-parameter MoE means that every layer’s forward computation, routing, expert scheduling, and all-to-all communication must complete within an extremely short time.

VRAM requirements:

• Weights: about 1488 GB (must all reside in VRAM, cannot be swapped out)
• Activations and workspace: about 150–200 GB
• KV Cache (32K context, FP8): about 40–60 GB
• Total: about 1680–1750 GB

Hardware options:

• 24–32 × 80GB accelerators, high-speed interconnect mandatory (NVLink or equivalent)
• Or 18–22 × 96GB accelerators, also requiring high-speed interconnect

Why high-speed interconnect is mandatory:
During single-token inference, 16 of 896 experts are activated. If experts are spread across 24 cards, each token requires several cross-card communications on average. PCIe 5.0 bandwidth becomes a bottleneck in all-to-all scenarios, driving actual output speed well below 25 tokens/s. Only NVLink-grade interconnect (hundreds of GB/s bidirectional bandwidth per card) keeps communication from becoming the bottleneck.

Cost range:

• 24 × 80GB cards plus high-speed interconnect backplane/switch: full system cost enters the million-RMB range
• Stacking 8-card supernode units with 3 units interconnected yields a similar cost

Key trade-offs:

• Reducing context length from 32K to 8K compresses KV Cache to under 15 GB, but has little impact on total VRAM (weights dominate)
• Lowering quantization from FP4 to a lower bit width (e.g., 3-bit) compresses weights to about 1100 GB, but the accuracy loss must be evaluated. The original requirement explicitly states “quantization no lower than FP4,” so this tier’s weight lower bound is locked at around 1488 GB

---

Tier 3: 10 Concurrency, ≥ 25 tokens/s per User

Goal: 10 concurrent requests, each with output speed of at least 25 tokens/s, and quantization no lower than FP4.

This is the entry point for production deployment. 10 concurrency means:

1. Compute demand doubles: Although the MoE activation ratio is only 1.79%, the 10 concurrent requests activate different expert sets per token, so actual computation far exceeds a linear 10× stacking of single-request load
2. KV Cache demand grows linearly: 10 requests with 32K context each require about 400–600 GB of FP8 KV Cache
3. Communication pressure surges: Tokens from 10 requests are routed simultaneously, multiplying all-to-all communication volume

VRAM requirements:

• Weights: about 1488 GB
• Activations and workspace: about 300–400 GB
• KV Cache (10 × 32K, FP8): about 400–600 GB
• Total: about 2200–2500 GB

Hardware options:

• 32–40 × 80GB accelerators (2560–3200 GB VRAM), supernode interconnect
• Or 24–28 × 96GB accelerators (2304–2688 GB VRAM), supernode interconnect

Cost range:

• 32 × 80GB cards plus supernode interconnect: full system cost enters the multi-million-RMB range
• A 64-card supernode (the official recommended minimum) costs more but provides more headroom

Why the vendor recommends 64+ cards:
A 64-card supernode provides about 5120 GB VRAM (80GB cards) or 6144 GB (96GB cards). After deducting 1488 GB for weights, 3600–4600 GB remains for activations, KV Cache, and runtime overhead. This leaves ample headroom for 10 concurrency and beyond. If you provision only 32 cards at the minimum VRAM requirement, the system runs at a high water mark for extended periods, and burst traffic or long-context requests can easily trigger OOM.

Engineering recommendations:

• Use an inference framework that supports continuous batching to dynamically merge the 10 concurrent requests
• Enable paged attention for KV Cache to reduce fragmentation
• Monitor P99 latency of all-to-all communication—this is the key metric for holding 25 tokens/s

---

3. Beyond Cost: Three Easily Overlooked Variables

3.1 The Accuracy Cost of Quantization

FP4 quantization-aware training is not “lossless.” 4-bit weights incur accuracy loss on complex reasoning tasks; the exact magnitude must be verified once the official technical report and benchmarks are released. If your workload is accuracy-sensitive, you may need to return to FP8 or mixed precision, which doubles weight VRAM to about 2800 GB and moves the entire hardware plan up one tier.

3.2 Maturity of the Inference Framework

Framework support for trillion-parameter MoE varies widely. Key capabilities include:

• Expert parallelism
• Expert offloading/prefetching
• Paged KV Cache
• Continuous batching
• Communication primitive optimization for high-speed interconnect

If the framework does not support expert parallelism, 896 experts cannot be distributed across cards, and the deployment plan is not viable.

3.3 Power and Cooling

A full system with 32 × 80GB accelerators typically draws 10–15 kW, and a 64-card supernode may reach 20–30 kW. This requires a dedicated machine room, industrial-grade power supply, and liquid cooling or high-capacity air cooling. Power and cooling account for a substantial share of three-year TCO and should not be overlooked.

---

4. Summary: Three-Tier Quick Reference

Tier	Goal	VRAM requirement	Card count (80GB)	Interconnect requirement	Cost magnitude
Tier 1	It runs	1600–1680 GB	24 cards	PCIe/Ethernet	Hundreds of thousands of RMB
Tier 2	Single user, 25 tokens/s	1680–1750 GB	24–32 cards	High-speed interconnect	Million-RMB range
Tier 3	10 concurrency, 25 tokens/s	2200–2500 GB	32–40 cards	Supernode interconnect	Multi-million-RMB range

Key conclusions:

1. The VRAM lower bound for a trillion-parameter MoE is determined by total parameters; the activation ratio affects only compute, not storage
2. Under FP4 quantization, weights are about 1488 GB—the starting point for every plan
3. Going from “it runs” to “single user at 25 tokens/s,” the bottleneck is interconnect bandwidth, not card count
4. Going from “single user” to “10 concurrency,” the bottleneck is KV Cache and all-to-all communication pressure
5. The vendor-recommended 64-card supernode is not over-configuration—it leaves headroom for burst traffic and long context

Once the weights are officially released, it is advisable to verify each of these estimates through actual benchmarking. Until then, this framework can help you establish reasonable budget expectations and avoid underestimating the deployment threshold.