Local Deployment of Trillion-Parameter MoE LLMs: Hardware Requirements and Cost Breakdown
Can you run a trillion-parameter mixture-of-experts (MoE) model on a few consumer GPUs? Not really. This article breaks down the hardware floor for a 2.8-trillion-total-parameter MoE model across three tiers, from "it just loads" to "ten concurrent users at 25 tokens/s each." It shows how the 1.4TB weight footprint is calculated, why sparse activation cuts compute but not memory, why 4-bit native weights and memory bandwidth decide whether the model merely fits or actually runs fast, and how to convert GPU hourly rates into a cost per million output tokens. It closes with a self-host vs. cloud GPU vs. inference API comparison and a pre-deployment checklist.
1. The short answer: three hardware tiers
Let’s start with the conclusion. For a MoE model with 2.8 trillion total parameters, “buy a few consumer GPUs and run it locally” is not realistic. Even the bare minimum of “it loads and runs” already sets a high hardware bar. Roughly, there are three tiers:
| Target scenario | Hardware starting point | Key constraint |
|---|---|---|
| Load and run at all | ~19 cards in the 80GB-per-card class, or ~11 cards in the 141GB-per-card class (counted in 8-card nodes, so 3 or 2 nodes respectively) | Neither generation of data-center GPUs supports native 4-bit float weights; dequantization happens at runtime and throughput suffers |
| One user at 25 tokens/s | 8 GPUs in the 288GB-per-card class (single node) | All 1.4TB of weights must fit in memory first. This is a capacity problem, unrelated to concurrency |
| 10 concurrent streams at 25 tokens/s each | Usually the same 8-card node, possibly more nodes depending on context length | Aggregate output must hold at 250 tokens/s — a scheduling and throughput problem that must be measured |
2. The memory math: where the 1.5TB comes from
To understand the barrier, start with one point: MoE sparse activation reduces compute, not memory footprint.
Out of 2.8 trillion parameters, roughly 104 billion are activated per generated token — about 3.7% of the total. So the compute per step is comparable to a hundred-billion-parameter dense model. But whether an expert is activated or not, all expert weights must stay resident in GPU memory. During autoregressive generation, the activation at each step is determined by the attention output and the router, and the router depends on only a handful of tokens in the current context — effectively a fixed input. As a result, activations during decoding are highly concentrated at each position and cannot be amortized across many tokens the way prefill can. So the idea of “load experts on demand and keep them in host memory the rest of the time” degenerates, in interactive generation, into moving nearly all weights on every single step. The speed simply won’t work.
Weight footprint can be estimated directly:
# Weight memory ≈ total parameters × bytes per parameter# bytes per parameter = quantization bit width ÷ 8bytes = 2.8e12 * (4 / 8) # ≈ 1.4e12 bytes, i.e. about 1.4TB
The 4-bit figure here is the model’s native weight precision: from training or release onward, weights are stored in a 4-bit float format, with scale factors shared across small blocks to preserve numerical accuracy at low bit width. That is different from “pseudo-quantization,” which compresses high-precision weights down to 4 bits after the fact — the former is part of the design, the latter usually comes with noticeable accuracy loss.
1.4TB is just the weights. An actual deployment also needs headroom for:
- Runtime: the inference framework’s own memory pool, routing tables, and temporary buffers;
- Activations: scaling with batch size and sequence length;
- KV / state cache: scaling with context length and concurrency.
Taken together, a single user needs roughly 1.5TB of GPU memory in practice. To be clear, 1.5TB is an order-of-magnitude estimate, not a precise ceiling: the longer the context and the higher the concurrency, the larger this overhead grows, and final capacity has to be settled by measurement.
3. The compute and bandwidth math: fitting the model isn’t the same as running it well
If the model doesn’t fit in memory, the service won’t even start. Fitting is not enough either, for two reasons.
(1) Bandwidth: decoding is a memory-bandwidth-bound workload. Even if each token activates only 104 billion parameters, that slice of weights has to be read in full every step. At 4-bit storage:
- 104 billion × 0.5 bytes = 52GB, meaning roughly 52GB of weights must be read from memory per generated token;
- For a single stream at 25 tokens/s, weight reads alone require about 1.3TB/s of aggregate bandwidth.
At that scale, current-generation single cards — with memory bandwidth ranging from several TB/s to nearly 10TB/s — can still cover it. So the deciding factor for a single stream at 25 tokens/s is not raw compute; it’s whether you can fit the entire model across cards with enough memory each. The first value of an 8-card setup is capacity, and parallel compute comes second.
(2) Format: without native 4-bit float weight support, throughput drops sharply. Some data-center GPUs do not support 4-bit float weight compute at the hardware level, so weights must be dequantized to a 16-bit format before running. The consequences are twofold:
- For the same single token, effective bandwidth demand grows from about 52GB to about 208GB, so one stream at 25 tokens/s needs more than 5TB/s of aggregate bandwidth;
- Dequantization itself consumes extra instructions and register resources, and the weights stored in memory get expanded, squeezing available space further.
That is the gap between “it fits” and “it runs well”: the older generation can load and run inference, but struggles to support smooth interactive generation, and is better suited to offline batch processing or functional validation.
4. Tier 1: the theoretical minimum
Working backward from capacity:
- 80GB per card: 1.5TB ÷ 80GB ≈ 18.75, rounded up to 19 cards;
- 141GB per card: 1.5TB ÷ 141GB ≈ 10.64, rounded up to 11 cards.
But data-center servers are generally built as 8-card nodes, so those counts have to be rounded up to whole nodes:
- 3 nodes of 8 × 80GB-class cards: 24 cards, roughly 1.92TB of memory;
- 2 nodes of 8 × 141GB-class cards: 16 cards, roughly 2.26TB of memory.
This raises two new problems:
- Cross-node communication overhead. Large-model parallelism requires cross-card synchronization at every layer. Inside a node you typically get high-bandwidth interconnect; between nodes you are limited to general-purpose networking, usually one to two orders of magnitude lower. Once the split crosses nodes, communication can become the main bottleneck, and the failure domain widens too.
- Limited throughput. Neither GPU generation supports native 4-bit float weights, so runtime dequantization is required.
As a result, this tier is only good for “proving it runs”: offline evaluation, algorithm validation, or scenarios where response speed doesn’t matter at all. It is not recommended as the foundation for an interactive service.
5. Tier 2: 25 tokens/s for a single user
A more realistic minimum is 8 GPUs in the 288GB-per-card class. A single node then totals 8 × 288GB = 2304GB (about 2.25TB), which fits the 1.4TB of weights and still leaves several hundred GB for the runtime and KV cache. Current-generation data-center GPUs generally support 4-bit float weights natively, so no runtime dequantization is needed, and the official deployment guidance from mainstream inference frameworks is mostly built around 8-card parallelism.
One common misconception here: “only one person will use it, so I can buy fewer cards.” It’s the opposite. Whether 1.4TB of weights fit has nothing to do with the number of online users — only with model size. Two or three cards will not hold this model no matter what.
To be clear, this article has no measured data on whether an 8-card configuration reliably reaches 25 tokens/s, so it can’t promise it. From engineering experience, though, single-stream inference on large models typically lands in the tens of tokens per second, so 25 tokens/s is a reasonable target. The prerequisites are listed below, and load testing should be completed at the target context length:
- High per-card memory bandwidth;
- High-bandwidth interconnect inside the node;
- Context kept as short as possible;
- A sensible parallelism split.
6. Tier 3: 10 concurrent users at 25 tokens/s each
This tier requires aggregate output to hold steadily at 10 × 25 = 250 tokens/s. Counterintuitively, the memory configuration may not need to grow beyond tier 2. Capacity determines how many request states you can hold, not how many tokens you can produce per second. By a medium-context estimate, one 8-card node can theoretically hold roughly 40 concurrent requests; if the context isn’t in the million-token range, the theoretical number of requests it can accommodate goes up further. So 10 concurrent users by itself does not force you to add a second server.
The real difficulty is whether all 10 streams can each sustain 25 tokens/s while generating simultaneously. That is a throughput and scheduling question, and again it can only be answered by measurement, not inferred from memory capacity. Three reasons:
- Every decode step reads roughly 52GB of weights, and the result is shared across the multiple requests in the current batch. Higher concurrency improves hardware utilization but also lengthens each step.
- MoE routing can be uneven: a few popular experts concentrate on some cards, creating imbalanced compute load. Overall speed is set by the slowest card.
- Long-context scenarios also have to account for prefill overhead and KV cache growth.
If measurement shows single-stream speed degrading noticeably as concurrency rises, these optimizations are worth considering:
- Speculative decoding: a small model drafts several tokens, then the large model verifies them in one pass. Suits low-concurrency, latency-sensitive scenarios;
- Architectures such as grouped-query attention (GQA) that reduce KV cache overhead;
- Prefill/decode disaggregation: splitting compute-heavy prefill from bandwidth-bound decode onto separate resource pools. The payoff at high concurrency is significant, but implementation and operational complexity rise too.
7. Cost estimation: from GPU-hour price to cost per million tokens
Data-center-class GPUs like these are hard for ordinary users to buy outright, so the most direct route is renting cloud GPU compute. The example price below walks through the full conversion chain.
① Base price. A cloud provider’s 12-month reserved price for a GPU instance is $4.76 per GPU-hour (an illustrative figure only; actual pricing varies by provider, region, configuration, and contract term — check the provider’s official real-time pricing).
② Per-node hourly cost.
Per-node hourly cost = $4.76/GPU-hour × 8 cards = $38.08/hour
③ Time cost. Roughly, at 24 hours a day and 30 days a month:
- About $913.92 per day;
- About $27,417.6 per month;
- About $333,581 for the full 12-month reserved contract.
④ Per-token cost. The conversion formula:
Cost per token = node hourly cost ÷ (aggregate output tokens/sec × 3600)Cost per million tokens = cost per token × 1,000,000
Plugging in both scenarios (both theoretical ceilings; real figures will be higher):
- Full load (250 tokens/s): 38.08 ÷ (250 × 3600) ≈ $0.0000423 per token, or about $42.3 per million output tokens;
- Single user (25 tokens/s): cost is 10× higher, about $423 per million tokens.
Even at full load, compute alone can cost tens of dollars per million output tokens, which is higher than common pay-as-you-go inference API pricing. Of course, the latter is also typically billed per token, so in the end you compare unit price against service quality.
From this, a more important conclusion follows: for self-deployment, utilization is the first-order cost variable. The same node serving 1 user versus 10 users differs in cost by 10×. That is also the practical reason clouds offer multiple billing models — reserved, on-demand, and spot — and why you should choose based on how stable your load is and how tolerant it is of interruption.
Also, the GPU hourly price is not total cost of ownership. Building your own data center means accounting for racks, power and cooling, high-speed networking, egress bandwidth, spare parts, and operations staff. Cloud deployment means adding object storage, load balancing, logging, and monitoring services.
8. Self-host, cloud GPUs, or an API?
Beyond cost, there’s a prior question: do you actually need local deployment? The tradeoffs:
| Dimension | Self-built data center | Cloud GPU instance | Inference API |
|---|---|---|---|
| Cost structure | Large upfront investment plus ongoing operations | Hourly or contract-based | Pay per usage |
| Elasticity | Poor; long scaling cycles | Good; fast to adjust | Best; pay as you go |
| Data control | Strongest | Strong | Depends on terms of service |
| Operational complexity | Highest | Medium to high | Nearly zero |
| Best fit | Stable long-term use, strict compliance | Fluctuating load, fast validation | Volatile demand, no special compliance needs |
From a pure economics standpoint, a reserved cost of roughly $330,000 per year only makes sense under long-term, high-load, high-utilization conditions. If usage is occasional, or you’d rather focus on business logic than on compute operations, a mature inference API is often the more pragmatic choice. But hard requirements — data compliance, deep control over model behavior, enforceable service-level objectives — can make local deployment the only option.
For organizations that genuinely need self-deployment, cloud platforms such as Baidu AI Cloud can let you validate and load-test at a smaller scale first, then decide whether to commit to a long-term reserved contract.
9. Pre-deployment checklist
- Size the capacity: estimate memory as “weights + runtime + KV cache at the target context length and concurrency,” not weight size alone;
- Confirm format support: make sure your GPUs support 4-bit float weights natively, to avoid runtime dequantization dragging down throughput;
- Check the interconnect topology: large-model parallelism is highly sensitive to intra-node interconnect bandwidth, so prefer a single-node 8-card configuration with complete interconnect;
- Build a load-test matrix: cover different context lengths (short, medium, long) and concurrency steps (1/2/4/8/10…), recording time to first token, single-stream speed, P99 latency, and GPU utilization;
- Calculate cost on effective load: use measured throughput rather than theoretical peak for cost per million tokens, and choose reserved, on-demand, or API billing accordingly.
10. Summary
Local deployment of a trillion-parameter MoE model is fundamentally a “capacity first” problem. The 1.4TB of weights has to fit in GPU memory as a whole, which sets the hardware floor at an 8-card, large-memory node; native 4-bit weights and memory bandwidth determine whether you reach an acceptable generation speed; concurrency and scheduling determine the experience when several people share the node; and real utilization ultimately determines whether the investment pays off. Do the memory math first, load-test throughput next, and calculate cost last — that’s how you avoid “affordable to buy, painful to use.”
