Trillion-Parameter LLM Local Deployment: A Complete Cost and Configuration Breakdown
This article breaks down what it takes to deploy a trillion-parameter Mixture-of-Experts (MoE) model on your own hardware, covering VRAM (memory) requirements, network bandwidth bottlenecks, hardware configuration tiers, and total cost of ownership. It explains why MoE memory must be budgeted by total parameters rather than activated ones, walks through the memory-floor calculation (around 1.5 TB just for weights), clarifies why a 64+ accelerator super-node is recommended because of all-to-all communication, and compares self-hosted versus cloud economics through a break-even analysis. It also includes optimization strategies for memory and communication, a Python estimation script, and an acceptance checklist for verifying real benchmarks once the weights are released.
A note on assumptions: As of this writing, the full weights and technical report for this class of trillion-parameter model have not been officially released; the vendor has said they will follow. Except for publicly disclosed architecture parameters and API pricing, all figures below for VRAM, bandwidth, and hardware prices are estimates based on public information. They illustrate the estimation methodology and order of magnitude only. Once the weights are released, each figure should be verified against measured benchmarks.
1. Conclusions First: Three Configurations and Budget Ranges
| Tier | Accelerator scale | VRAM pool | Suitable scenario | Estimated 3-year TCO (RMB) |
|---|---|---|---|---|
| Validation tier | 16 cards (2 nodes, 80GB-class per card) | 1.28 TB | Weight loading, single-stream or low-concurrency debugging, accuracy evaluation | ~RMB 3–6 million |
| Recommended tier | 32 cards (4 nodes, 80GB-class per card) | 2.56 TB | Steady inference service at small-to-medium scale | ~RMB 7–14 million |
| High-throughput tier | 64 cards (8 nodes, super-node form factor) | 5.12 TB | High concurrency, long context, or multi-stream parallel service | ~RMB 14–28 million |
The table above assumes a single-card purchase price of RMB 50,000–100,000 (training-grade products typically cost more), a total cost of ownership of roughly 2.6–2.8x the hardware purchase price, and electricity at RMB 0.8/kWh. See sections 5 and 6 for the estimation methodology.
2. Why MoE VRAM Is Counted by Total Parameters
This model has about 2.8 trillion total parameters and uses a Mixture-of-Experts (MoE) architecture. Of its 896 experts, each token activates only 16—an expert activation ratio of about 1.79%.
Many teams’ first reaction is: “If only 1.8% of the parameters are used, can’t we also save 98% of the VRAM?” The answer is no, because capacity and compute are two separate problems:
- VRAM (a capacity problem): All 896 experts must stay resident in memory, because the router may select any of them at any time.
- Compute (a compute problem): A single token only participates in the matrix operations of 16 experts, so the activation ratio affects only compute consumption.
So what MoE saves is floating-point operations, not VRAM. Even after real throughput ramps up and different experts are hit at roughly uniform frequencies, this can only improve VRAM “reuse” at the batch level; it does not change the fact that all experts must be loaded.
When this article refers to “about 1.79% activated per token,” it means the expert activation ratio. Because experts also contain components that always participate in computation—such as attention and shared experts—this figure is not a strict parameter-activation ratio. It is only an approximate basis for estimating the number of activated parameters.
3. Memory-Floor Calculation: 1.5 TB Is Just the Starting Point
3.1 4-bit Weights with Block-Level Scaling Factors
Starting from the supervised fine-tuning stage, this model uses quantization-aware training, natively employing a numerical scheme of 4-bit weights with 8-bit activations.
The naive calculation is 2.8 trillion parameters × 0.5 bytes ≈ 1400 GB. But the block-level scaling format (one 8-bit scaling factor shared per 32 4-bit elements) adds overhead:
- Effective bit width: 4 + 8 ÷ 32 = 4.25 bit
- Raw weights: 2.8×10¹² × 4.25 ÷ 8 ≈ 1487.5 GB
In practice, it is safer to convert using binary units: 1487.5 GB ÷ 1.024 ≈ 1453 GiB.
3.2 What to Reserve Beyond the Weights
| Reserve item | Estimate | Notes |
|---|---|---|
| Non-quantized tensors | Tens of GB | Embedding layers, normalization layers, the output layer, routing, etc. usually keep higher precision, depending on the weight format |
| Framework and communication buffers | Hundreds of MB to a few GB | Inference engine runtime, communication buffers, CUDA graphs (graph capture), etc. |
| KV cache | Depends on architecture and context | Must be assessed item by item (see 3.3) |
| Compute peak and fragmentation headroom | ~5%–10% | Intermediate results of attention operators, temporary buffers, VRAM fragmentation |
3.3 KV Cache: The VRAM “Amplifier” of Hybrid Attention Architectures
The backbone of this class of model is composed of components such as linear attention, gated low-rank attention, and attention residuals.
- State-update layers: At runtime, only a fixed-size recurrent state needs to be maintained, so VRAM does not grow with sequence length.
- Full-attention layers: These still need to cache Key/Value, but gated low-rank attention has already compressed the KV.
So the VRAM overhead of long context depends mainly on how many full-attention layers are retained, the compressed KV dimension, and the cache precision. These parameters cannot be known before the weights are released, so when planning, you should make tiered assumptions based on “number of full-attention layers × compressed KV dimension.”
3.4 Comparison with a Dense Model
A useful point of reference is another open-source model with 715 billion parameters (671 billion main parameters plus ~14.5 billion multi-token prediction parameters):
- FP8 weights are about 685 GB; eight 80 GB cards (640 GB) cannot hold them directly.
- After switching to 4-bit weight quantization, the weights are about 335 GB, and adding activations and runtime overhead brings the total to about 400 GB—only then can it run stably on a single 8-card node.
A comparable 2.8-trillion-parameter model is more than 4x larger, which is the fundamental reason it “cannot fit in a single node.”
3.5 Memory Budget Summary
| Item | Per-instance estimate | Notes |
|---|---|---|
| 4-bit weights (with scaling factors) | ≈1453 GiB (1.49 TB) | Fixed cost, cannot be compressed |
| Non-quantized tensors and framework overhead | 20–60 GiB | Varies with engine and quantization scheme |
| KV cache and compute peak | Tens of GiB to start, higher for long context | Roughly proportional to concurrency and context length |
| Per-instance total | ~1.6 TB to start | Equivalent to the VRAM pool of 2–3 eight-card 80GB nodes |
4. Why the Official Recommendation Is 64+ Cards: Communication Is the Real Bottleneck
VRAM determines whether the model “fits,” while interconnect bandwidth determines whether it “runs fast.”
4.1 Large-Scale MoE Is Highly Sensitive to All-to-All Communication
The 896 experts are spread across many cards. Once routing selects 16 experts for a token, its activations must be sent to the cards holding the target experts and gathered back after computation. This “dispatch–compute–aggregate” process runs through every expert module in every layer, making it a classic all-to-all communication pattern.
Large-scale MoE inference in the industry typically uses expert parallelism (EP) to partition experts, combined with tensor parallelism, pipeline parallelism, and data parallelism—i.e., “multi-dimensional hybrid parallelism.” In this setup, each token may generate cross-node transfers at every layer; the more nodes there are, the higher the share of cross-node traffic.
4.2 The Bandwidth Gap Between Inter-Node and Intra-Node
- Inter-card interconnect: Within a single super-node, a private interconnect domain provides near memory-semantics access, with per-card unidirectional bandwidth reaching hundreds of GB/s to the TB/s range.
- Inter-node network: A conventional high-speed RDMA parameter plane offers 200 Gbps or 400 Gbps per port, or roughly 25–50 GB/s. High-end lossless networks can aggregate a node’s egress to the Tbps level, but costs rise sharply.
The two differ by one to two orders of magnitude. All cross-node traffic must pass through the NIC, and once cross-node communication becomes the bottleneck, no amount of compute can help—it just idles. This is the point of the “super-node” form factor: use a larger inter-card interconnect domain to keep as much all-to-all communication as possible inside the node.
4.3 Estimation: How Much Time All-to-All Can “Eat”
The following estimates use illustrative parameters: hidden dimension 8192, activations transferred at 8-bit, 60 layers, a single-card 400 Gbps NIC, with about 16 KB sent and received per token per card per layer.
| Scenario | Per-card, per-layer communication | Cross-node share | Per-card cross-node bandwidth | Theoretical communication time |
|---|---|---|---|---|
| 16 cards (2 nodes) | ≈256 KB | 50% | 50 GB/s | ≈2.6 ms |
| 32 cards (4 nodes) | ≈256 KB | 75% | 50 GB/s | ≈3.8 ms |
| 64 cards (8 nodes) | ≈256 KB | 87.5% | 50 GB/s | ≈4.5 ms |
A few observations:
- This is only a theoretical lower bound. After adding software-stack overhead, network congestion, and queuing delays, a millisecond-scale pause at a single layer can grow to over ten milliseconds, and this is more pronounced under high concurrency.
- The larger the batch, the more data each all-to-all transfer carries, and bandwidth pressure grows roughly linearly. The higher the throughput target, the more you depend on a high-bandwidth parameter plane.
- Configurations above 64 cards require all three of the following: an adequate memory budget, a high-bandwidth parameter plane (e.g., one 400 Gbps-class NIC per card, or Tbps-class node egress), and “compute–communication overlap” mechanisms.
The official recommendation to deploy on a super-node of 64+ accelerators is precisely to align expert distribution with the high-bandwidth interconnect domain, so that post-routing activations flow as much as possible within the super-node.
5. Self-Hosted vs. Cloud: How to Choose and How to Calculate
5.1 What Makes Up Self-Hosted TCO
The cost of self-hosting goes well beyond the accelerators themselves:
- Hardware procurement: Accelerator cards and full 8-card servers (CPU, memory, local disks, power supplies, etc.), inter-node switches and optical modules, storage equipment.
- Data-center retrofit: A fully loaded node can draw about 8 kW, and a full rack commonly exceeds 10 kW. This usually requires high-power racks, redundant power, in-row cooling, and even liquid cooling—an ordinary data center often cannot support it directly.
- Ongoing operations: Electricity, bandwidth, colocation, and hardware maintenance.
- Staffing: At least 1–2 engineers familiar with distributed inference to handle deployment tuning and day-to-day operations.
5.2 Full-Load Self-Hosting vs. On-Demand Cloud: Break-Even Analysis
Let C be the annual cost of full-load self-hosting, P the cloud on-demand price per card-hour, and n the number of cards. Then the break-even annual runtime hours are:
h* = C / (n × P)
Using the 32-card configuration as an example: annual self-hosting cost C is about RMB 2.3–3.5 million (hardware purchase of RMB 5–7.5 million, depreciated over 3 years and including electricity, maintenance, and staffing), with a cloud on-demand price P of RMB 80 per card-hour.
| Per-card cloud price | Break-even annual runtime | Equivalent utilization | When self-hosting wins |
|---|---|---|---|
| RMB 80/card·hour (on-demand) | ≈900 hours | ≈10% | Annual runtime exceeds ~900 hours |
| RMB 30/card·hour (committed-use discount) | ≈2400 hours | ≈27% | Annual runtime exceeds ~2400 hours |
Model inference cannot “run only half the time during business hours”: high concurrency requires reserved headroom, and troughs are hard to idle completely. At the discounted price, a workload that is “online 365 days a year at only 30% utilization” costs about the same whether self-hosted or on the cloud.
Two more variables matter in real decisions:
- Load fluctuation: When daytime peaks and nighttime troughs differ greatly, elasticity is more valuable.
- Iteration pace: Open-source large models are refreshed roughly every few months, while GPU depreciation is typically calculated over 3–5 years. The faster the model iterates, the higher the risk of hardware obsolescence—an implicit opportunity cost.
Cloud pay-as-you-go essentially converts capital expenditure into operating expenditure. You pay for what you use, avoid paying for peak capacity year-round, and take on no data-center retrofit or operations staffing.
5.3 Cloud Capacity Mapping
Taking Baidu AI Cloud as an example, a trillion-parameter model of this scale involves at least four types of resources:
| Resource type | Role in deployment | Capacity reference |
|---|---|---|
| GPU cloud servers | Host inference processes; can form a high-bandwidth interconnect domain of 32 or 64 cards | Assuming 80 GB VRAM per card, a single instance needs 32–64 cards |
| Parallel file storage / object storage | Store model weights and version snapshots of 1.5 TB or more | At least 1.5 TB for weights; 3–5 TB recommended for multiple versions |
| Container service | Orchestrate, schedule, and elastically scale inference instances in a containerized way | Planned by node count and instance replicas |
| Log service / monitoring & alerting | Collect metrics for VRAM, utilization, latency, and errors | Configured by instance count and collection frequency |
VRAM and interconnect are hard thresholds, but cloud resources offer finer adjustment granularity and faster elasticity. During validation, small-scale instances suffice to verify accuracy; scale out horizontally at business peaks; and after a model refresh, old instances can be released at any time.
5.4 Hybrid Architecture: Balancing Compliance and Elasticity
Many enterprises see clear daytime traffic peaks, and some data must remain on the internal network. A common approach is internal–external coordination:
- Deploy small-scale “baseline capacity” in the local data center to handle sensitive data and steady-state traffic on an ongoing basis.
- Use a high-speed cloud connection to form an “elastic capacity pool” of cloud GPU instances, with a unified gateway routing traffic by load and real-time price.
- When local instances saturate or during nighttime troughs, automatically scale cloud resources up or down to keep elastic cost and latency within budget.
6. Deployment Architecture: Four Categories of Optimization for VRAM and Communication
6.1 Parallelism Strategy: Align Expert Distribution with the Interconnect Domain
- Expert parallelism (EP): Each card holds only a subset of experts, minimizing VRAM pressure, but all-to-all is heaviest.
- Tensor parallelism (TP): Splits a single expert’s matrices across cards; every card participates in computation, so keeping communication inside the node is more cost-effective.
- Pipeline parallelism (PP): Splits by layer, reducing per-card VRAM, but introduces pipeline bubbles.
- Data parallelism (DP): Each replica holds the full model, suitable for scaling concurrency with multiple replicas.
Trillion-parameter models typically use a combination of “TP + EP within a node, PP / DP across nodes,” keeping communication within the highest-bandwidth interconnect domain as much as possible.
6.2 Quantization and Cache Optimization
- Keep the native quantization format: Since the weights are 4-bit from the training stage, loading 4-bit weights directly is the most VRAM-efficient at inference. Dequantizing back to FP8 or even BF16 before inference multiplies VRAM usage.
- KV cache quantization: For full-attention layers at low concurrency, if the cache is small, spare VRAM should go to batch size rather than precision.
- On-demand offloading: When VRAM is tight, cold experts can be offloaded to fast host memory, at the cost of an extra PCIe round trip for those tokens—suitable for low-concurrency validation scenarios.
6.3 Load Balancing: The “Bucket Effect” of Expert Parallelism
Routing distribution is naturally uneven, and a few hot experts may receive far more than the average number of tokens. Within the same layer, the card holding a hot expert becomes the slowest link for the whole layer—the bucket effect. Common countermeasures include:
- Setting up redundant replicas for hot experts;
- Using a dynamic capacity factor to cap the load on any single expert;
- Adopting grouped routing constraints to reduce ineffective activation of irrelevant experts.
6.4 Bandwidth Calculation: The Ceiling for Single-Stream Decoding
Inference computation can be split into two phases:
- Prefill: Processes the entire input at once, with large matrices and high parallelism—compute-intensive.
- Decode: Generates one token per step and must read the weights in full at every step—memory-access-intensive.
Estimating at about 1.79% of parameters activated per token, the weights that single-stream decoding must read per step are about:
2.8×10¹² × 1.79% × 0.5 bytes ≈ 25 GB
With per-card HBM bandwidth of about 3.5 TB/s, the aggregate bandwidth of 16 cards is about 56 TB/s, giving a theoretical lower bound of about 0.45 ms per step for single-stream decode memory access. But every step also adds the latency of dozens of layers of all-to-all, so measured single-stream speed is usually noticeably below this lower bound.
Estimating at 50 tokens/s single-stream and an output length of 1000 tokens, each request needs about 20 seconds for the generation phase.
- To support 100 concurrent requests: cluster output throughput must reach 100 × 50 = 5000 tokens/s, about 100x the single-stream capability.
- This requires batched inference, continuous batching, and multi-replica data parallelism working together, which further explains the origin of the official 64+ card recommendation.
7. A VRAM Estimation Script: One Ruler for Everything
Before the weights are released, you can use the script below to work out “whether VRAM is enough, how many cards you need, and how much it will cost”:
"""VRAM and configuration estimation for a trillion-parameter MoE model (illustrative parameters, for planning reference only)"""# ---------- Model assumptions (replace with measured values after weights are released) ----------TOTAL_PARAMS = 2.8e12 # Total parameter countBITS_WEIGHT = 4 # Weight bit widthBITS_SCALE = 8 # Block-level scaling factor bit widthSCALE_GROUP = 32 # Number of elements covered by each scaling factorLAYERS = 60 # Total number of layersFULL_ATTN_LAYERS = 8 # Number of full-attention layers (the rest are linear-attention layers)KV_HEADS = 8 # KV heads per full-attention layer (GQA/MQA assumption)HEAD_DIM = 128 # Compressed KV dimensionKV_BITS = 8 # KV cache precisionCARD_MEM_GB = 80 # VRAM per cardNODE_CARDS = 8 # Cards per nodeCARD_PRICE_WAN = 6.0 # Purchase price per card (in units of 10k CNY)NODE_EXTRA_WAN = 5.0 # Non-card cost of a full node (in units of 10k CNY)SWITCH_COST_WAN = 30.0 # Parameter-plane switches, optical modules, and cables (in units of 10k CNY)POWER_KW = 8.0 # Full-load power per node (kW)PUE = 1.4 # Power usage effectivenessELEC_PRICE = 0.8 # Electricity price (CNY/kWh)YEARS = 3 # Depreciation period (years)def mem_report(seq_len: int, batch: int) -> None:"""Print VRAM and node requirements for different concurrency/context levels"""weight_gb = TOTAL_PARAMS * (BITS_WEIGHT + BITS_SCALE / SCALE_GROUP) / 8 / 1e9kv_bytes = (2 * FULL_ATTN_LAYERS * KV_HEADS * HEAD_DIM* (KV_BITS / 8) * seq_len * batch)kv_gb = kv_bytes / 1e9reserve_gb = 40 + 0.05 * weight_gb # Framework overhead + 5% compute peak headroomtotal_gb = weight_gb + kv_gb + reserve_gbnodes = max(1, -(-int(total_gb) // (CARD_MEM_GB * NODE_CARDS)))cards = nodes * NODE_CARDSprint(f"Context {seq_len // 1024}K x concurrency {batch:>4} | weights {weight_gb:6.0f}GB "f"| KV {kv_gb:6.1f}GB | total {total_gb:6.0f}GB "f"| needs {nodes} nodes / {cards} cards | VRAM utilization {total_gb / (cards * CARD_MEM_GB):.0%}")def cost_report(nodes: int) -> None:"""Print procurement spend and 3-year total cost of ownership"""cards = nodes * NODE_CARDScapex_wan = cards * CARD_PRICE_WAN + nodes * NODE_EXTRA_WAN + SWITCH_COST_WANpower_kwh = nodes * POWER_KW * PUE * 24 * 365tco_wan = capex_wan + (power_kwh * ELEC_PRICE + 0.05 * capex_wan) * YEARSprint(f"Config {nodes} nodes / {cards} cards | procurement {capex_wan:,.0f} (10k CNY) "f"| annual power {power_kwh / 1e4:,.1f} (10k kWh) | 3-year total {tco_wan:,.0f} (10k CNY)")if __name__ == "__main__":for seq, bs in [(32768, 1), (131072, 8), (131072, 32)]:mem_report(seq, bs)for n in (2, 4, 8):cost_report(n)
Sample output (after the weights are released, replace the model assumptions with measured values):
Context 32K x concurrency 1 | weights 1453GB | KV 1.3GB | total 1527GB | needs 3 nodes / 24 cards | VRAM utilization 80%Context 128K x concurrency 8 | weights 1453GB | KV 41.9GB | total 1568GB | needs 3 nodes / 24 cards | VRAM utilization 82%Context 128K x concurrency 32 | weights 1453GB | KV 167.8GB | total 1694GB | needs 3 nodes / 24 cards | VRAM utilization 88%Config 2 nodes / 16 cards | procurement 197 (10k CNY) | annual power 29.4 (10k kWh) | 3-year total 321 (10k CNY)Config 4 nodes / 32 cards | procurement 379 (10k CNY) | annual power 58.8 (10k kWh) | 3-year total 621 (10k CNY)Config 8 nodes / 64 cards | procurement 743 (10k CNY) | annual power 117.6 (10k kWh) | 3-year total 1223 (10k CNY)
Combined with the ranges in the conclusions table, we can see:
- Although 16 cards can “barely fit,” they struggle to support long context and high concurrency at the same time.
- 32 cards is the common starting point for balancing cost and headroom under a single instance, and it happens to match the scale of one standard high-bandwidth interconnect domain.
- 64 cards suits production scenarios that need high throughput or multiple replicas.
8. Conclusions and Action Recommendations
- VRAM: budget by total parameters. At 2.8 trillion parameters with native 4-bit quantization, the raw weights alone are about 1.45–1.5 TB. A single instance needs at least ~1.6 TB, and a single 8-card 80GB node is far from enough.
- Nodes: budget by communication domain. The bottleneck of large-scale MoE is all-to-all, not compute. As nodes increase, the share of cross-node traffic rises, so you must rely on the high-bandwidth interconnect of a super-node form factor.
- Card count: budget by throughput. 16 cards suits validation, 32 cards suits steady-state service, and 64+ cards suits high-concurrency production.
- Cost: calculate the total bill. For self-hosting, look at 3–5 year total cost; for cloud, look at utilization and elasticity; most teams should adopt a hybrid form of “local baseline, cloud elasticity.”
After the weights are officially released, complete acceptance testing against the following checklist:
- Verify VRAM usage with the measured layer count, KV dimension, and non-quantized tensors
- Measure the actual effective bandwidth of single-card, single-node, and cross-node interconnects
- Test all-to-all efficiency under different batch sizes and concurrency levels
- Evaluate the accuracy difference before and after quantization on your business evaluation set
- Stress-test throughput with continuous batching to derive inference cost per million tokens
Only by grounding every number in real measurements can local deployment of a trillion-parameter model be made both calculable and affordable.
