Trillion-Parameter LLM Local Deployment: Cost Breakdown and How to Choose
Deploying a trillion-parameter MoE model locally is far harder than simply fitting it into memory. This article breaks down three target tiers—from a bare "it runs" setup for functional validation, to single-user 25 tps, to 10 concurrent users—and the very different hardware each one demands. It explains why memory must be sized by total parameters rather than activated ones, why weight storage alone needs at least 1.5TB, why inter-card and inter-node communication becomes the hidden bottleneck, and how heterogeneous GPU+CPU inference can cut storage costs for low-load cases. It also provides realistic cost estimates for accelerators and infrastructure, a rent-vs-build framework based on annual full-load hours, six practical cost-saving strategies, a decision checklist, and a reusable Python script for estimating memory and bandwidth needs.
1. The bottom line first: the hard part isn’t “fitting it in”
Deploying a trillion-parameter MoE model locally is not really about whether you can load it. The real challenges are whether memory capacity, inter-card interconnect, the inference engine, batching efficiency, operating costs, and your business SLA all line up.
If you only need it to “run,” a handful of GPUs plus a server with very large memory may be enough. If you need a single user at 25 tps, you’re typically looking at a multi-node GPU cluster. If you need 10 concurrent users with each user at no less than 25 tps, it turns into a data-center-scale project.
So judging whether a trillion-parameter MoE model is suitable for local deployment is not about the parameter count alone—it’s about the target tier: functional validation, single-user usability, or multi-user concurrency.
2. Three tiers, three costs: the same model
| Tier | Goal | Deployment form | Hardware threshold | One-time investment (order of magnitude) |
|---|---|---|---|---|
| Tier 1 | Runs, speed not a concern | A few GPUs + large-memory CPU server; expert weights held in memory | Tens of GB of VRAM to start; memory ≥1.5TB | ~RMB 200K–600K |
| Tier 2 | Single-user output ≥25 tps | Multi-node GPU cluster with high-speed interconnect | Total VRAM ≥1.5TB; typically 20+ 80GB-class accelerators | ~RMB 2M–7M |
| Tier 3 | 10 concurrent and single-user ≥25 tps | Tier 2 plus added compute, VRAM, and scheduling headroom | 20+ 80GB-class accelerators with ample compute and VRAM headroom | ~RMB 3M–10M |
All three tiers point to the same conclusions:
- Tier 1: technically feasible, but only good for functional validation.
- Tiers 2 and 3: essentially data-center-scale engineering, where cost rises quickly with “concurrency × per-user speed × context length.”
- Economic judgment: if annual full-load operation is under roughly 4,000 hours and compliance allows, renting is usually more cost-effective than building. If data must stay on-premises, offline inference is required, or the compute can be reused for training, fine-tuning, and batch jobs, local self-hosting is more valuable.
3. Reading the architecture: first figure out “how many parameters are activated”
A representative public configuration of a recently announced ultra-large-scale MoE model looks like this:
| Item | Parameter |
|---|---|
| Total parameters | ~2.8 trillion |
| Architecture | Sparse mixture-of-experts (MoE) |
| Number of routed experts | 896 |
| Experts activated per token | 16 |
| Expert activation ratio | 16 ÷ 896 ≈ 1.79% |
| Weight/activation precision | 4-bit floating-point weights + 8-bit floating-point activations |
| Attention backbone | Linear attention, gated low-rank attention, residual attention |
| Quantization approach | Quantization-aware training from the supervised fine-tuning stage onward |
3.1 Expert activation ratio ≠ parameter activation ratio
The 16/896 figure is only the activation ratio at the expert level; it is not the parameter activation ratio for the whole model. Each layer also has a shared attention component that is always involved in computation, including linear attention, gated low-rank attention, and residual attention. These parameters are never skipped by routing—every token passes through them.
So 1.79% should be treated as the lower bound of the actual activation ratio. If we assume the dense parts such as attention make up about 5% of total parameters, then the parameters a single token actually touches are roughly:
2.8 trillion × (1.79% + 5%) ≈ 0.19 trillion
That’s about 6.8% of the total. Note that this ratio is an illustrative estimate; the real figure depends on the model’s published structure. This correction affects both the compute estimate and the “expert weight read volume” estimate.
3.2 Memory is sized by all parameters; activation ratio only affects compute
All 896 experts must be resident in storage, and only the 16 selected by routing are pulled out for computation during inference. This yields a rule you must keep in mind:
MoE memory usage is sized by total parameters; the activation ratio only affects compute consumption.
Plenty of “a trillion-parameter model only needs a few cards” claims come precisely from confusing these two things.
4. Memory accounting: why weights need at least 1.5TB
4.1 1400GB is only the ideal figure
Storing 2.8 trillion parameters at 4-bit gives a raw-data floor of roughly:
2.8 × 10¹² × 4 bit ÷ 8 = 1.4 × 10¹² bytes ≈ 1400 GB
But the 4-bit floating-point format needs a group-shared scaling factor: one 8-bit scale for every 32 4-bit elements, used to restore the value range. Under the microscaling format spec, each weight element needs on average:
4 + 8/32 = 4.25 bit
Which works out to:
2.8 × 10¹² × 4.25 bit ÷ 8 ≈ 1.49 × 10¹² bytes ≈ 1487.5 GB
So the theoretical floor for the weight portion is closer to 1.5TB than 1.4TB.
4.2 Additional memory a complete deployment needs
| Item | Description |
|---|---|
| Non-4-bit tensors | Token embeddings, output head, normalization parameters, etc. may stay at higher precision |
| KV cache | Grows linearly with concurrency and context length |
| Runtime workspace | Temporary operator buffers, communication buffers, framework overhead |
All told, the theoretical memory floor for minimum hardware is about 1.5TB. A real deployment also needs headroom for startup, loading, inference, and failover.
4.3 Side-by-side comparison: scale and threshold
Using a common ~0.7-trillion-parameter MoE model as a reference point:
- Under FP8, the main parameters are about 671GB, plus a multi-token prediction module at about 14GB, for roughly 685GB total.
- After 4-bit quantization, the weights are about 335GB; with activations included, about 400GB.
- Such a model can therefore run at full capability on 8 × 80GB accelerators, i.e., 640GB of VRAM.
The target model is roughly 4× the size of this class of model, so the memory threshold jumps from “a single 8-card machine” to “multiple nodes and 20+ accelerators.”
5. Why you need super-nodes: communication is the hidden tax
The official deployment recommendation for this model is a super-node of 64 or more accelerators. The reason isn’t compute—it’s communication.
5.1 Every token triggers an “expert gathering”
The 896 experts are spread across different accelerators. After a router picks 16 experts for a token, one of two things usually happens:
- most of those 16 experts are not on the current card; or
- the current card holds many experts this token doesn’t use.
Either way, two global data exchanges are required:
- Send the token’s hidden vector to the cards hosting the target experts (dispatch).
- Collect the results back after computation (combine).
5.2 Small-message latency is the hidden tax
Take a hidden dimension of 7168 as an example. Represented in FP16, each token is about 7168 × 2 = 14336 bytes, i.e., roughly 14KB.
With 896 experts spread across cards, it’s highly likely that the 16 target experts land on different cards. In that case, each global exchange may require every card to send many small messages to the other cards. Small single-hop messages sent in large numbers mean the bottleneck is often communication latency, not link bandwidth.
5.3 Intra-node vs. inter-node bandwidth differ greatly
- Cards within a node, connected by a proprietary high-speed interconnect bus, can reach hundreds of GB/s to the TB/s range in one direction.
- Across nodes, relying on InfiniBand or RoCE, the network bandwidth per card is typically only 25–50 GB/s.
That’s a gap of about 20× to over 100×.
5.4 The latency budget is very tight
Single-user 25 tps means only 40ms per token. If every global exchange crosses nodes, and there are two such exchanges across 60+ MoE layers, the accumulated latency is enough to push single-user speed down to single-digit tps.
Common engineering countermeasures include:
- Enlarge the communication domain: choose machines that pack 16 or more accelerators into a single node or rack with full inter-card interconnect, minimizing inter-node exchanges as much as possible.
- Continuous batching: combine tokens from multiple requests into one batch so a single exchange carries more useful computation.
- Raise the compute-to-communication ratio: when batch size is small, attention compute is roughly fixed while expert compute grows linearly with token count. At batch = 1, communication dominates; as batch grows, expert compute can “cover” the communication overhead.
6. Heterogeneous inference: amortizing storage costs with large memory
The most expensive part in terms of capacity is those 896 sparse experts. Open-source “GPU + CPU heterogeneous inference” approaches already exist in industry: keep the sparse experts in CPU memory, supplied on demand by a large-memory server.
The division of labor works like this:
| Component | Compute characteristics | Recommended placement |
|---|---|---|
| Shared attention | Compute-intensive; every token must pass through it | GPU, leveraging high compute and high VRAM bandwidth |
| Routed experts | Account for the vast majority of storage; only 16 used per token | Large CPU memory, prefetched by layer and by expert |
That said, whether this route can hit the 25 tps target needs to be checked against the “expert weight read volume.”
6.1 Expert weight read per token
1487.5 GB × 16 ÷ 896 ≈ 26.6 GB
In other words, on average each token needs to read about 26.6GB from the expert weights.
6.2 Converting to bandwidth demand
- At 25 tps: 26.6 GB × 25 = 665 GB/s. That’s the lower bound for CPU memory bandwidth for the expert-weight portion alone.
- The dense parts such as attention can’t simply be folded into CPU memory bandwidth. If dense parameters are about 5% of total, i.e., roughly 0.14 trillion:
- 4-bit storage is about 70GB;
- FP16 storage is about 280GB.
- The dense parts are better placed in GPU VRAM, read by GPU HBM; the expert parts are prefetched by CPU memory. The two bandwidths can’t be conflated.
So under a heterogeneous setup, the main data path per token is: CPU memory reads expert weights, GPU VRAM reads dense parameters. At 25 tps, CPU memory needs at least ~665 GB/s of expert-read capability; the GPU side must handle roughly 280GB/step of dense-weight reads, on top of which KV cache, runtime overhead, and scheduling losses pile up.
6.3 Comparison with a real configuration
Published measurements from an open-source heterogeneous inference project offer a reference point:
- Configuration: a dual-socket high-end server CPU with 12-channel DDR5, a theoretical single-socket memory bandwidth of 460.8 GB/s, plus one 24GB consumer-grade GPU.
- Model: a 4-bit quantized version of a ~0.7-trillion-parameter MoE, weights about 377GB.
- Result: short-prompt prefill at 200+ tokens/s, decode speed around 14 tokens/s, using an expert-selection optimization strategy.
By measured or inferred estimates, if each token needs to read on the order of 53GB of weights, then 14 tps corresponds to about 742 GB/s of total reads. A dual-socket server’s total memory bandwidth can reach the hundreds of GB/s to about 900 GB/s range, so 742 GB/s is in the same order of magnitude; but if you look only at single-socket or effective bandwidth, the actual speed will be noticeably limited.
6.4 Mapping to the target model
The target model’s expert-weight read per token is about 26.6GB, roughly half of the reference model’s 53GB. On the same configuration, the single-user decode speed should be on the order of 2× the reference model, i.e., 20–30 tokens/s.
But for 10 concurrent users at single-user ≥25 tps, at a per-step batch of 10, the expert weights read per second would be:
26.6 GB × 10 × 25 = 6650 GB/s
That far exceeds mainstream dual-socket server memory bandwidth.
Conclusion: the heterogeneous approach is only suitable for Tier 1—functional validation and low-frequency single-user use. It cannot meet Tier 3’s high-concurrency target.
7. Tier-by-tier breakdown: configuration and overhead
Tier 1: just needs to run
- Memory side: the GPU handles attention and routing logic; memory holds the expert weights.
- CPU server: at 64GB per channel, 12 channels single-socket gives 768GB, and dual-socket totals 1536GB. That already covers the 1487.5GB of expert weights, but not the runtime buffers, so you need to populate every channel or choose high-density memory.
- Safe configuration: a 12-channel platform fully populated with 24 × 64GB DIMMs, or a dual-socket high-density memory server. A 4-channel or 8-channel setup with 16 × 64GB usually only reaches 1024GB, which falls short.
- GPU side: if the dense part uses FP16, you need about 280GB of VRAM, so at least 4 × 80GB accelerators; if you go 4-bit or another low-precision scheme and control precision loss, that drops to about 70GB, so at least 2 × 80GB accelerators.
- Expected performance: decode speed roughly from single digits to the low 20s in tokens/s, highly sensitive to prompt length and the expert-placement strategy; long-prompt prefill is very slow.
- The real weakness: high and unpredictable latency. Per-token latency is often hundreds of milliseconds or even seconds, which can’t support real-time conversation.
Tier 2: single user at 25 tps
VRAM is the hard constraint. At 80GB per card:
1500 GB ÷ 80 GB ≈ 19
So you need at least 20 cards to hold both the weights and runtime overhead—and that’s before redundancy and failover margin.
At any given moment, only about 0.19 trillion parameters take part in computation. That works out to roughly 380 GFLOPs of matmul per token; at 25 tps, a single card needs only about 9.5 TFLOPS, less than a fraction of a high-end card.
It’s clear that the bottleneck for single-user decode is weight-read bandwidth, not compute:
| Setup | Estimated effective read capability | Illustrative single-user decode speed |
|---|---|---|
| 20 × 80GB, effective HBM bandwidth ~30 TB/s | Estimated ~300 GB weight read per token | ~100 tps |
| 20 × 80GB, effective HBM bandwidth ~48 TB/s | Same as above | ~150 tps |
| Dual-socket 12-channel DDR5, effective bandwidth ~300 GB/s | Expert weights only, 26.6 GB/token | ~11 tps |
The GPU approach leaves a large margin to handle three cases:
- KV reads driven by long context;
- the attention compute;
- stalls from global exchanges.
Configuration recommendations:
- Count: at least 20 × 80GB accelerators.
- Networking: prefer machines that pack 16+ cards per node or rack with full inter-card interconnect, minimizing inter-node exchanges.
- Engine: the inference engine must support expert parallelism and efficient global exchanges—a capability most consumer-grade deployment setups currently lack.
Tier 3: 10 concurrency, single user ≥25 tps
At this tier, compute finally becomes a real constraint.
Per-step compute (batch = 10, context 4096, ~60 layers estimated):
- Weight matmul: 0.19T × 2 × 10 ≈ 3.8 TFLOPs;
- Attention part: usually much smaller than the weight term, totaling about 4 TFLOPs/step.
At 25 steps/s, that’s about 100 TFLOPS. In theory a few high-end cards could meet the arithmetic demand, but that doesn’t solve VRAM capacity, weight-read bandwidth, or global-exchange overhead.
Bandwidth is the bottleneck:
- each step needs to read about 1500GB of weights, and at 25 steps/s that’s about 37.5 TB/s;
- at 80% efficiency, you need at least 15–20 × 80GB accelerators.
On KV cache:
- if you compress KV with a low-rank attention approach, each token needs only on the order of hundreds of bytes per layer;
- roughly computing at 512 bytes per token per layer, 100 concurrent, 4096 context:
60 × 512 × 100 × 4096 ≈ 12.6GB
With only 10 concurrent, that’s about 1.26GB. With traditional multi-head attention under the same conditions, it could balloon tens to hundreds of times, and a 20-card setup would be stretched thin.
Training-phase KV pressure is even greater. Assume a training sequence length of 128K:
- roughly at 512 bytes per token per layer for a single sequence:
60 × 512 × 128K ≈ 4GB
If K and V are 512 bytes each, that’s about 8GB. In training, a batch often contains dozens of sequences, so the total scales up to tens of GB or more.
That’s why training this class of model depends on large-scale sequence parallelism, splitting the sequence and head dimensions across more cards. From the training configuration, a super-node of 64+ cards serves training first, and inference deployment inherited the same form.
Summary:
- On the GPU configuration that delivers “25 tps at batch = 1,” simply raising the batch can in theory substantially increase total throughput and thus serve 10 concurrent users.
- But constrained by compute intensity, global-exchange overhead, and the inference engine’s scheduling ability, reserving compute headroom is necessary.
The three tiers at a glance: single-user speed vs. batch
| Concurrency | Heterogeneous estimate | 20-card GPU (illustrative) |
|---|---|---|
| 1 | ~20–30 tps | ~100–150 tps |
| 4 | Bandwidth-limited; far below 4 × 25 tps | Depends on batching efficiency |
| 10 | Cannot meet single-user 25 tps | Must reserve compute and VRAM headroom |
Note: the heterogeneous figures are estimated under the assumption that “decode speed is limited by CPU memory bandwidth”; the GPU figures are illustrative values, meant to show that bandwidth determines single-user speed while compute and scheduling determine the concurrency ceiling.
8. Cost and benefits: three more accounts beyond hardware
8.1 Accelerator cards: the biggest line item
- Price ranges widely: from entry-level to data-center-grade, a gap of nearly 20×. By recent public-channel ballpark, data-center-grade cards run roughly RMB 35,000–120,000 each.
- Consumer cards cost about RMB 8,000–30,000 but have no high-speed inter-card interconnect or cross-node RDMA, so cross-card parallel efficiency drops sharply.
- High-end training cards cost about RMB 150,000–250,000, and their cost-performance is usually worse than the former.
- 20 cards total roughly RMB 600,000–2,400,000.
8.2 Supporting infrastructure
| Item | Description |
|---|---|
| Servers | An 8-GPU server runs ~RMB 300K–800K each |
| Networking | A high-end server or 200G/400G NIC runs ~RMB 2,000–8,000 per card (per actual procurement channel); one per node, plus switches |
| Storage | Weights are ~1.5TB; at 3 replicas that’s ~4.5TB, a modest investment for enterprise NVMe drives |
| Data center | A 20-card system draws ~8–12 kW; adding CPU, memory, networking, and storage brings it to ~10–16 kW, requiring dedicated power and cooling |
These are recent public-channel estimates; prices fluctuate with supply and demand, taxes, channels, and configuration, and the final figures depend on actual procurement.
8.3 Electricity and long-term costs
- Power consumption: at an average draw of 12 kW, annual consumption is roughly 12 × 24 × 365 ≈ 105,120 kWh. At a data center electricity rate of about RMB 0.7–1.2 per kWh, that’s roughly RMB 70,000–130,000 per year.
- Depreciation: depreciating accelerators over 3–5 years gives about RMB 120,000–800,000 per year.
- Operations: routine inspections, fault replacement, driver and framework upgrades, model and engine tuning.
- Opportunity cost: tying up 20 high-end accelerators means losing the chance to use them to train or fine-tune other models.
9. Why “renting” often beats “buying”: annual full-load hours
Let the one-time investment for the 20-card configuration be A (in units of RMB 10,000), the rental price for the same-spec compute be P (RMB per card-hour), and the annual full-load hours be H.
- Self-build annualized total cost: A ÷ 5 + 10
- Rental cost: 20 × P × H ÷ 10000
Setting them equal and solving for the annual full-load hours at which self-building wins:
H ≈ (A ÷ 5 + 10) × 10000 ÷ (20 × P)
- With P ≈ RMB 60 and A = RMB 4.5M (i.e., 450): H ≈ (90 + 10) × 10000 ÷ 1200 ≈ 833 hours/year.
- With P ≈ RMB 30 and A = RMB 2.5M (i.e., 250): H ≈ (50 + 10) × 10000 ÷ 600 = 1000 hours/year.
Factor in utilization, the operations team, and hidden costs, and the guidance becomes:
| Scenario | Situation | Recommendation |
|---|---|---|
| Working hours or test environment only | Less than 2,000 full-load hours per year | Rent |
| 7×24 external service, cloud allowed by compliance | ~8,000 full-load hours per year, close to full utilization | Self-building is usually more economical, but renting is also fine if you’d rather not handle operations |
| Data can’t leave the intranet, or offline inference is required | Compliance mandates on-premises | Build locally |
| Spare compute available for reuse | Training, fine-tuning, batch offline inference | Build locally to amortize costs |
10. Save where you can: six cost-reduction strategies
- Match the model scale: when the workload is mainly general Q&A, summarization, or rewriting, a MoE model in the tens to hundreds of billions of parameters covers most scenarios. For the same “single-user 25 tps” goal, a 100B-class MoE can hit it with one or two 80GB cards, a cost gap of 1–2 orders of magnitude. Only tasks like very-long-document understanding, complex code, and reasoning genuinely need a trillion-class model.
- Choose the right quantization format: this model uses quantization-aware training from the fine-tuning stage, so 4-bit weights are essentially “native precision”; switching to post-training quantization would noticeably amplify precision loss.
- Reduce KV overhead: prefer low-rank attention architectures, or enable KV quantization in the inference engine. In high-concurrency, long-context scenarios, this often pays off more than buying more compute.
- Optimize expert placement: track how often each expert is routed, and place hot experts where VRAM or bandwidth is better, cutting the overhead of “can’t fetch it, shuffling back and forth.” This is the expert-selection strategy common in heterogeneous inference projects, and it applies equally to cluster deployments.
- Separate prefill and decode: long-prompt prefill is compute-intensive, while decode is bandwidth-intensive. Deploying them in separate pools with independent scaling avoids long requests blocking short ones and meaningfully improves overall utilization.
- Speculative decoding: use a small model or a lightweight module of the model itself to “guess” several tokens, then have the large model verify them in a single batched forward pass. The more guesses that hit, the more tokens each decode step produces, amortizing weight reads—precisely hitting MoE decode’s bandwidth bottleneck.
11. Before you start: a decision checklist
- Is local deployment a must?: data compliance, network isolation, and steady large-volume offline inference are the three legitimate reasons for local deployment. “Saving money” is rarely one of them.
- Define the SLA: is the target user count a concurrency peak or an average? What is the minimum acceptable per-user speed? How long is the context? These three directly determine the configuration tier.
- Validate in stages:
- Stage 1: get it running on a single machine with a heterogeneous setup, verifying weight integrity and output quality.
- Stage 2: rent a small-scale multi-card allocation and load-test with real business traffic to measure actual MFU and batching efficiency.
- Stage 3: decide how many cards to buy based on the measured data.
- Align budget cycles: hardware is a one-time investment, while electricity, depreciation, and operations are ongoing costs; if traffic is unstable, prefer turning fixed costs into variable ones.
- Use the cloud as a fallback: for scenarios where compliance allows and traffic fluctuates, use managed inference or elastic GPU compute in the cloud to absorb peaks, keeping only a small baseline inference setup on-premises.
- Leave a validation window: after new model weights are released, run the benchmark scripts before signing a procurement contract or expansion order.
- Risk note: insufficient memory or VRAM configuration can cause startup failures, inference interruptions, or incomplete model loading; back up weights and configuration before deploying.
12. A reusable estimation script
The script below computes the “memory–bandwidth–speed” estimates in one pass. It doesn’t depend on any specific framework—just swap the parameters to reuse it for other MoE models.
# -*- coding: utf-8 -*-"""Memory and bandwidth estimation for trillion-parameter MoE local deployment (illustrative)"""GB = 10 ** 9def estimate(total_params, num_experts, top_k, bits=4.25, target_tps=25):# 1. Weight memory: average bits per element (4-bit data + one 8-bit scale per 32 elements)weights = total_params * bits / 8# 2. Expert weights actually read per token (assuming roughly uniform routing)per_token = weights * top_k / num_experts# 3. Use decimal GB for display and round appropriately to align with the article's figuresweights_gb = int(weights // GB) # ~1487 GBper_token_gb = round(per_token / GB, 1) # ~26.6 GBbw_needed = per_token_gb * target_tps # ~665 GB/sreturn {"Weight memory": f"{weights_gb} GB","Expert read per token": f"{per_token_gb:.1f} GB",f"Bandwidth needed for {target_tps} tps": f"{bw_needed:.0f} GB/s","Heterogeneous single-user limit (effective 300GB/s memory bandwidth)":f"{300 / per_token_gb:.1f} tps","Memory bandwidth needed for 10 concurrency in heterogeneous setup":f"{per_token_gb * target_tps * 10:.0f} GB/s",}if __name__ == "__main__":result = estimate(total_params=2.8e12, num_experts=896, top_k=16)for key, value in result.items():print(f"{key}: {value}")
Output:
Weight memory: 1487 GBExpert read per token: 26.6 GBBandwidth needed for 25 tps: 665 GB/sHeterogeneous single-user limit (effective 300GB/s memory bandwidth): 11.3 tpsMemory bandwidth needed for 10 concurrency in heterogeneous setup: 6650 GB/s
Two final reminders:
- The numbers above cover expert weights only. The dense parts such as attention need another ~5% of parameter memory, plus KV cache and runtime overhead.
- Real speed also depends on global-exchange efficiency, batching strategy, and inference-engine implementation. Before finalizing a choice, be sure to benchmark with your target workload’s real traffic.
