Local Deployment of Trillion-Parameter Models: VRAM and Cost Estimation
Local deployment of a trillion-parameter MoE model is fundamentally an engineering problem of memory and communication: every expert must be stored in full, so VRAM is driven by total parameters rather than the activation ratio. Based on the publicly disclosed architecture of a 2.8-trillion-parameter model (896 experts, 16 activated per token, MXFP4 weights with MXFP8 activations), this guide builds a general estimation framework: the quantization math behind a ~1.5 TB weight floor, a cross-check against a 671B MoE reference deployment, and the resulting hardware picture — roughly 1.8–2.1 TB of total VRAM, at least 24–27 80 GB GPUs, and supernodes of 64+ GPUs as the official recommendation. It also explains the memory, communication, and parallelism walls behind the supernode form factor, and closes with a benchmark checklist for verifying every estimate once the weights are released.
Local Deployment of Trillion-Parameter Models: A Guide to VRAM, Hardware, and Cost Estimation
As of July 2026, a MoE large language model with 2.8 trillion total parameters has yet to fully release its weights and technical report; the official word is that it will do so by July 27. So, apart from the publicly disclosed architecture and API pricing, every speed, memory, and hardware configuration figure below is an engineering estimate based on public information, and each one will need to be verified with benchmarks once the weights are out.
The goal of this article is not to hand down a deployment verdict for any specific product, but to establish a general estimation framework for trillion-scale MoE models, so that developers can plan hardware budgets sensibly before the weights become available.
1. What 2.8 Trillion Parameters Mean for VRAM
Start with the architectural fundamentals. The model has 2.8 trillion total parameters in a mixture-of-experts (MoE) design with 896 experts, 16 of which are activated per token — an expert activation ratio of about 1.79%. One point worth stressing: this is the expert activation ratio, not a strict parameter activation ratio.
The backbone uses KDA (Kimi Delta Attention) plus Attention Residuals, with Gated MLA as one of its components. The goal of this hybrid attention design is to reduce attention computation and KV-cache pressure in long-context scenarios — but it does not change the fundamentals of weight memory.
The key insight: MoE deployment memory is counted against total parameters, and the activation ratio only affects compute. You must first fit all 896 experts into storage or VRAM, and only then does inference routing select 16 of them. In other words, the memory floor is set by total parameter count, not by active parameter count.
2. From Parameter Count to VRAM: A Layer-by-Layer Breakdown
2.1 Basic Quantization Math
The model has undergone quantization-aware training from the SFT stage onward, with MXFP4 weights and MXFP8 activations. Converting 2.8 trillion parameters to 4-bit gives a raw data floor of roughly:
2.8e12 × 4 bit ÷ 8 = 1400 GB
But in MXFP4, every 32 4-bit elements also carry an 8-bit scale. Counting weight elements plus scales per the OCP specification:
2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB
Even that is only the ideal lower bound for the weights — it still excludes non-4-bit tensors, metadata, alignment, the runtime workspace, and the KV cache.
2.2 A Reference Point for Comparison
Take a MoE model with 671B total parameters as a reference: in FP8 format, its 671B main parameters plus 14B MTP parameters total about 685 GB; even with AWQ 4-bit, you still need roughly 335 GB of weights plus 37 GB of activations — about 400 GB in total — to run on eight 80 GB GPUs (640 GB of VRAM).
At more than 4× the size of that 671B model, the 2.8-trillion-parameter model sets its full-strength deployment floor accordingly. Rough extrapolation: the weights alone require about 1.5 TB of VRAM, and actual demand will be higher once runtime overhead is included.
2.3 A Cost Estimation Template
Assuming today’s mainstream datacenter GPUs with 80 GB of VRAM each:
| Item | Estimate |
|---|---|
| Weight VRAM lower bound | ~1.5 TB |
| Runtime workspace + KV cache | 0.3–0.6 TB |
| Total VRAM requirement | ~1.8–2.1 TB |
| Number of 80 GB GPUs | ≥ 24–27 |
| Official recommended supernode size | 64+ GPUs |
At typical rental rates on the order of one to two dollars per GPU-hour for high-end datacenter GPUs, a 64-GPU supernode costs on the order of a hundred dollars per hour — several thousand dollars per day. Self-hosting means budgeting for complete systems, networking, power, and cooling as well, an investment on the order of several million dollars.
3. Why Supernodes Are the Recommended Form Factor
The official recommendation is to deploy this model on a supernode built from 64 or more GPUs. There are three reasons:
- The memory wall. A single 8-GPU machine cannot hold the weights, so VRAM has to be aggregated across nodes.
- The communication wall. MoE routing makes all-to-all communication frequent, and ordinary Ethernet can hardly meet the latency requirements — a high-bandwidth, low-latency interconnect is needed.
- The parallelism strategy. A combination of expert parallelism (EP), tensor parallelism (TP), and pipeline parallelism (PP) is required, which places high demands on topology-aware scheduling.
4. A Pre-Deployment Verification Checklist
Once the weights are released, verify in roughly this order:
# Pseudocode: pre-deployment verification workflowchecklist = ["Confirm the actual quantization format of the weights and the scale overhead","Measure peak VRAM on each GPU after loading its weight shard","Stress-test all-to-all communication bandwidth and latency","Run end-to-end inference benchmarks and record TTFT (time to first token) and TPOT (time per output token)","Verify KV cache usage at the target context length","Calculate the actual inference cost per million tokens",]
5. Summary
Local deployment of a trillion-scale MoE model is, at its core, an engineering problem of VRAM and communication. The key takeaways: estimate VRAM by total parameters, quantization by its format, and cost by the supernode. Before the weights are officially released, use the framework above for budget planning so you don’t underestimate the hardware bar; once they are released, verify each item with benchmarks to converge these estimates into a deployment plan you can actually execute.
