How to Deploy a Multi-Trillion-Parameter MoE LLM Locally—and What It Costs
Deploying a recently open-sourced Mixture-of-Experts (MoE) LLM with roughly 2.8 trillion total parameters—only about 104 billion of which activate per token—is a server- or small-cluster-scale project, not something a high-end workstation can handle. This article explains why MoE routing forces all expert weights to stay resident in VRAM, works through the memory math (~1.4TB of native MXFP4 weights, ~1.5TB with runtime overhead for a single user), and maps three deployment tiers onto concrete hardware: "it runs" and a single user at 25 tokens/s both require an 8-GPU node of new-generation datacenter GPUs, while 10 concurrent users at 25 tokens/s each is likely still a single node but demands real benchmarking. It also estimates the running costs—about $330K per year for a 24/7 cloud-rented 8-GPU node—and explains when self-hosting beats using a managed cloud inference service, and when it doesn't.
1. The Short Answer: This Is Not a “Buy a Few Flagship GPUs” Problem
A recently open-sourced ultra-large Mixture-of-Experts (MoE) LLM has drawn a lot of attention: about 2.8 trillion total parameters, with roughly 104 billion activated per token. Many developers’ first instinct is: “Only ~100B parameters are activated, so a few GPUs should be enough, right?” The conclusion is exactly the opposite: even the lowest bar of “it runs” pushes past the ceiling of an ordinary workstation. This is a server-class, or even small-cluster-class, project.
The reason traces back to a first principle of MoE deployment: the routing mechanism can direct any token to any expert, so the complete set of expert weights must reside in VRAM at all times. You cannot trim away the other 96% just because “on average, less than 4% of parameters activate.” Keep one guiding rule in mind: the activated parameter count sets the ceiling on speed, while the total parameter count sets the floor on memory. The two must be budgeted separately.
2. Start with the VRAM Math
The weights themselves: ~1.4TB
The model ships natively in MXFP4, a block-scaled 4-bit floating-point format. This is the official designed precision—not FP16 weights compressed to 4 bits after the fact. The 4-bit precision loss was absorbed during the design phase, so there is none of the degradation typical of post-hoc quantization.
The memory estimate is plain arithmetic:
Total parameters × bits per parameter ÷ 8= 2.8 trillion × 4 bits ÷ 8≈ 1.4 TB
Some readers will ask: can’t you quantize harder to save memory? The official release precision is 4-bit, and no lower official tier exists. More aggressive community quantization schemes, if they appear, would need to be validated on your own—and they should never be the premise of a deployment plan.
Adding runtime overhead: ~1.5TB for a single user
Raw weights are only the theoretical floor. Real-world operation also adds:
- Activations: intermediate tensors from the forward pass;
- Framework runtime: scheduling, communication buffers, operator and computation-graph caches, and similar overhead;
- KV cache / state space: grows linearly with context length, which matters a lot for long-text workloads.
Combining these, a single active user needs on the order of 1.5TB of GPU memory. For comparison: consumer flagship cards carry only 24–32GB each. Even setting aside multi-GPU interconnect, power, and cooling, capacity alone would take dozens of cards. That route is a dead end.
3. Three Requirement Tiers, Three Hardware Thresholds
Tier 1: It just has to run—speed is secondary
From pure capacity, the theoretical floor is about:
- 19× 80GB-class datacenter GPUs (the previous mainstream accelerator generation), or
- 11× 140GB-class high-memory variants.
But here’s a common pitfall: those two GPU generations support older numeric formats and cannot execute MXFP4 computation natively. They must dequantize at runtime—the weights fit in memory, but decoding keeps performing format conversions, which noticeably drags down speed. That’s “it fits,” not “it runs well.” In theory you could also pool capacity across more older cards on multiple machines, but multi-node deployment requires high-speed RDMA networking, and the complexity of tensor-parallel / expert-parallel sharding rises accordingly. It isn’t any easier than a single-node setup.
The more realistic minimum is a single node with 8 new-generation datacenter GPUs in the 256–288GB class. Public deployment references from mainstream inference engines likewise start at 8-way parallelism. In other words, the entry ticket for the “it runs” tier is a complete multi-GPU server node.
Tier 2: A single user, a stable 25 tokens/s
The hardware starting point remains 8 new-generation GPUs, for a blunt reason: the 1.4TB of weights must fit first. MoE memory demand is independent of request concurrency—you can’t shrink to two or three cards just because only one person is using the model. Remove even one card, and the weights no longer fit.
Can 8 cards hold a steady 25 tokens/s? A sanity check from the decode-stage bandwidth bottleneck: for MoE models, generating each token requires moving roughly “activated parameters × 4 bits” worth of weights—about 52GB (excluding KV). Single-stream decoding is therefore a textbook bandwidth-bound scenario, roughly:
Single-stream throughput ceiling ≈ effective VRAM bandwidth ÷ bytes moved per token
An 8-GPU node has ample aggregate bandwidth, so “tens of tokens per second” per stream is the right order of magnitude. 25 tokens/s is a realistic goal, but not one you can promise on paper—long contexts, sampling parameters, and framework implementation differences all move the actual number. Stress-test with real context lengths before going live.
Tier 3: 10 concurrent users at 25 tokens/s each
The aggregate throughput requirement is 10 × 25 = 250+ tokens/s.
Interestingly, the memory configuration for this tier is not necessarily higher than Tier 2. Weights are shared across all concurrent requests; what grows with concurrency is only per-request KV cache and scheduling overhead. Per public estimates, an 8-GPU node can host at least ~40 concurrent requests, and the theoretical residency count is even higher if contexts stay below the million-token level.
So 10-way concurrency doesn’t, by itself, force a second server. The real question is: when 10 streams generate simultaneously, can each still sustain 25 tokens/s? That is a throughput-and-scheduling problem, determined by batch scheduling efficiency and the communication efficiency of MoE expert dispatch. It can only be answered by measurement, not inferred backward from memory capacity.
4. Counting the Money: What a Year of Operation Burns
These datacenter GPUs are essentially unavailable through retail channels. In practice, the realistic path is cloud rental. Using one cloud provider’s publicly listed 12-month reserved-instance pricing as a reference: new-generation GPUs of this class cost roughly $4–5 per GPU-hour, so an 8-GPU node totals about $38 per hour. Extrapolating:
Per hour: ~$38Per day (24 hours): ~$900Per month (30 days): ~$27,000Per year (365 days): ~$330,000
That is bare compute rental only. It excludes building and maintaining the inference service stack, monitoring and alerting, failure recovery, and the engineering time spent on benchmarking and tuning.
If you self-host instead: an 8-GPU new-generation server typically costs on the order of several hundred thousand dollars up front, plus roughly 10kW-class power and cooling, colocation, depreciation, and operations staffing. For the vast majority of teams, building in-house doesn’t pay off in either money or manpower.
5. Before You Start, Answer Three Questions
1. Do you actually need local deployment? Self-hosting only makes sense when compliance rules require the model to stay on-premises, when token volume is large enough that pay-as-you-go API billing is clearly more expensive, or when latency and availability requirements are extreme. For most workloads, using a cloud inference service or a metered API is more economical—for example, the large-model inference services offered by Baidu AI Cloud can offload the underlying compute and operations burden entirely.
2. Did you pick the right GPU generation? Choose new-generation GPUs that natively support 4-bit floating-point formats. Older generations “fit but run slowly”—dequantization overhead continuously slows decoding, which means paying for 8 GPUs while getting the experience of 4.
3. Have you benchmarked? The memory math only answers “does it fit.” Performance must be measured. A recommended benchmark checklist:
- Test with real business workloads at short, medium, and long context lengths;
- Measure both single-stream latency and per-stream throughput at your target concurrency;
- Observe how KV cache growth under long contexts affects the number of resident requests;
- Make sure sampling parameters match real usage patterns.
6. Conclusion: One Table Says It All
| Requirement tier | Minimum hardware | Key constraint | Benchmarking needed? |
|---|---|---|---|
| It just runs | 8× new-generation datacenter GPUs | 1.4TB weight residency | Lenient |
| Single user, 25 tokens/s | Same as above | Bandwidth and decode efficiency | Mandatory |
| 10 concurrent × 25 tokens/s | Single node very likely sufficient | Aggregate throughput and scheduling | Mandatory |
To put it in one sentence: memory sets the floor, bandwidth and scheduling set the ceiling, and the bill decides whether to self-host at all. At current cloud prices, running an 8-GPU node around the clock costs on the order of $300K per year. Unless your workload is large enough to amortize that, a cloud inference service is always the more pragmatic first choice.
