Trillion-Parameter MoE Model Local Deployment: VRAM, Hardware, and Cost Analysis
This article uses a 2.8-trillion-parameter Mixture-of-Experts (MoE) model with 896 routed experts per layer, top-16 expert activation, and 4-bit floating-point weights as a case study. It explains why a 1.79% expert activation ratio does not reduce resident VRAM, breaks down the full memory budget—including weights, quantization scale factors, non-4-bit tensors, runtime workspace, and KV cache—and compares hardware topologies, deployment routes, and cost factors. It also provides practical optimization directions and a pre-launch checklist for teams considering local or self-hosted deployment.
Note: This article analyzes a model with approximately 2.8 trillion total parameters, 896 routed experts per layer, 16 experts activated per token, and 4-bit floating-point weights. Hardware configurations and prices are illustrative public-market examples used to explain the method. Actual values depend on the model documentation, hardware models, procurement timing, and measured results.
1. First, clarify a common misconception: 1.79% does not mean “56× less VRAM”
1.1 What 2.8 trillion parameters mean for memory
This model has approximately 2.8 trillion total parameters and uses a sparse Mixture-of-Experts architecture. Each layer contains 896 routed experts, and a routing network selects 16 experts to participate in computation for each token:
- Expert activation ratio = 16 ÷ 896 ≈ 1.79%
This ratio is often misread as “only 1.79% of the memory is needed” or “compute can be reduced by 56×.” In reality:
- VRAM depends on the total number of parameters that must remain resident, not on how many experts are activated per token.
- Compute depends on the number of parameters participating in computation, which is determined by the activation structure.
- VRAM and compute are governed by different factors. This is why trillion-scale MoE models have very large parameter counts, controllable per-token compute, but an extremely high deployment threshold.
1.2 Architecture requires memory to be budgeted for the full parameter set
Parameters in a MoE model can be roughly divided into two categories:
| Parameter category | Composition | Relation to tokens |
|---|---|---|
| Dense parameters | Attention projections, including low-rank compression and gating variants; normalization layers; embedding/output layers; router gating | Shared by all tokens |
| Sparse parameters | Routed experts in each layer, and shared experts if present | Used only by selected tokens |
During deployment, all 896 experts must remain resident in accelerator memory and wait for routing decisions. Inference can only call experts on demand; it cannot reliably load them on demand. If an expert is not already resident, its weights must be fetched from main memory or a remote node. Cross-node fetching introduces latency far higher than a single matrix multiplication and can directly degrade decoding speed. When memory and bandwidth are extremely abundant, a small number of cold experts may be swapped in on demand, but this is a special optimization rather than a default approach.
1.3 “Expert activation ratio” is not the same as “parameter activation ratio”
To calculate the parameter activation ratio, dense parameters that are used by all tokens must also be included:
- Parameter activation ratio ≈ dense parameter ratio + (1 − dense parameter ratio) × 1.79%
The following example code shows the difference between “how many parameters are computed” and “how much memory is actually occupied”:
TOTAL_PARAMS = 2.8e12 # Total parameters: dense parameters + all expertsNUM_EXPERTS = 896 # Number of routed experts per layerTOP_K = 16 # Number of experts activated per tokenDENSE_RATIO = 0.15 # Dense-parameter ratio: illustrative assumption; check model documentationBITS_PER_PARAM = 4.25 # 4-bit weights + one 8-bit scale factor per 32 elementsexpert_ratio = TOP_K / NUM_EXPERTSactive_ratio = DENSE_RATIO + (1 - DENSE_RATIO) * expert_ratioactive_params = TOTAL_PARAMS * active_ratioresident_params = TOTAL_PARAMS # Resident memory is calculated using the full parameter setprint(f"Expert activation ratio: {expert_ratio:.2%}")print(f"Parameter activation ratio: {active_ratio:.2%}")print(f"Per-token compute relative to a dense model with the same total parameters: {active_ratio:.1%}")print(f"Resident memory relative to a dense model with the same total parameters: {resident_params / TOTAL_PARAMS:.0%}")print(f"Resident weights: {resident_params * BITS_PER_PARAM / 8 / 1e12:.3f} TB")
Example output under the above assumptions:
Expert activation ratio: 1.79%Parameter activation ratio: 16.52%Per-token compute relative to a dense model with the same total parameters: 16.5%Resident memory relative to a dense model with the same total parameters: 100%Resident weights: 1.488 TB
Per-token compute is only about 16.5% of that of a dense model with the same parameter count, but resident memory remains 100%. If the dense parameter ratio is assumed to be between 0.05 and 0.30, the parameter activation ratio is roughly 7% to 26%. However, regardless of the assumed value, memory occupancy does not change.
Conclusion: 1.79% determines “how fast the model computes,” while 2.8 trillion parameters determine “whether it fits.” The first threshold for local deployment is the latter.
2. Memory budget for trillion-scale MoE: a breakdown
The total memory budget can be written as:
- M_total = W_weights + W_unquant + W_align + M_workspace + M_kv + M_reserve
- Decode latency lower bound ≈ weight bytes read per token ÷ aggregate memory bandwidth across cards
Unless otherwise stated, capacity units below use decimal GB, where 1 GB = 10⁹ bytes.
2.1 Raw weights: the theoretical floor of 4-bit quantization
The raw 4-bit data lower bound is:
- 2.8 × 10¹² × 4 bit ÷ 8 = 1.4 × 10¹² bytes = 1400 GB, or about 1.30 TiB
This is an unavoidable baseline. Any claim about “deploying a trillion-parameter model on a consumer-grade single machine” fails at this first step.
2.2 Quantization scale factors: the easily overlooked 6.25%
This type of model uses quantization-aware training from the supervised fine-tuning stage. Weights are 4-bit floating-point, and activations are 8-bit floating-point. A public micro-scaling floating-point specification uses a “small block shared scale” design: every 32 low-bit elements share one 8-bit scale factor, which extends the dynamic range. Therefore:
- Effective bits per parameter = 4 + 8 ÷ 32 = 4.25 bits, which is 6.25% more than raw 4-bit
- Weights plus scale factors = 2.8T × 4.25 bit ÷ 8 = 1487.5 GB, or about 1.385 TiB
| Accounting method | Bits per parameter | Capacity |
|---|---|---|
| Raw 4-bit values only | 4.00 bit | 1400 GB, or 1.30 TiB |
| Values plus scale factors, recommended | 4.25 bit | 1487.5 GB, or 1.39 TiB |
The difference is 87.5 GB, roughly the capacity of one large-memory accelerator card. If hardware planning only counts raw data, an out-of-memory failure after deployment becomes very likely.
2.3 Non-4-bit tensors and alignment overhead
Not all tensors are 4-bit. The following categories need to be accounted for separately:
- Embedding and output layers: If these layers are not quantized or not tied to the embedding layer, assume a vocabulary size of about 150,000 and hidden dimension of about 7,000; a single matrix in bf16 can be on the order of 2 GB.
- Attention modules: Some low-rank compression projections or gating parameters may remain at 8-bit or 16-bit. If they are outside the quantization scope, they must be counted as bf16, or 2 bytes per parameter. Every 1 percentage point increase in the dense parameter ratio adds roughly 56 GB to the budget.
- Normalization and router gating: Parameter counts are small and can be ignored in most estimates.
- Sharding alignment and router capacity alignment: Weights are partitioned using tensor parallelism or expert parallelism. Shards often need to be aligned to a fixed granularity, such as 128 elements. Inference routing may also use a capacity factor to control the batch size of a single expert. These can introduce small amounts of additional memory overhead.
2.4 Runtime workspace: the part frameworks do not report
The following components exist outside the “weight memory” statistics reported by some inference frameworks, but they still occupy real memory:
- Activations and temporary buffers: intermediate activations during long-context prefill, and temporary buffers for dequantization and reductions.
- Communication buffers: all-to-all send/receive buffers for expert parallelism, and all-reduce/all-gather buffers for tensor parallelism. These grow with batch size and parallelism degree.
- Framework and communication library overhead: hundreds of MB to several GB per card, including context, CUDA Graph capture, memory pool fragmentation, and similar items.
From engineering experience, this portion should be reserved at about 5% to 15% of weights. In a 16-card deployment, 2 GB to 5 GB per card corresponds to 32 GB to 80 GB in total.
2.5 KV cache: the main variable for concurrency and context length
KV cache size depends only on the attention structure and context length. It is unrelated to the MoE activation ratio. A general estimation formula is:
- KV bytes/token = 2 × number of full-attention layers × KV elements per token × bytes per element
- M_kv = KV bytes/token × context length × number of concurrent sequences
In low-rank compressed attention designs, such as those representing KV with a compressed latent vector plus decoupled positional dimensions, KV elements per token = compressed dimension + positional dimension. In hybrid architectures, linear attention layers maintain a fixed-size recurrent state that is independent of sequence length, so the KV cache is contributed only by full-attention layers.
Example assumptions: 60 layers in total, 40 full-attention layers, compressed dimension 512, positional dimension 64, bf16 storage:
- KV per token = 2 × 40 × (512 + 64) × 2 bytes = 92,160 bytes = 90 KiB
- 32K context × 16 concurrent streams ≈ 45 GB
- 128K context × 16 concurrent streams ≈ 180 GB
def kv_cache_gb(full_attn_layers, kv_lora_rank, rope_dims, dtype_bytes,seq_len, concurrency):per_token = 2 * full_attn_layers * (kv_lora_rank + rope_dims) * dtype_bytestotal = per_token * seq_len * concurrencyprint(f"Per-token KV: {per_token / 1024:.1f} KiB | "f"{seq_len // 1024}K context x {concurrency} concurrent streams: {total / 1e9:.1f} GB")# Example assumptions: 40 full-attention layers, compressed dimension 512 + positional dimension 64, bf16kv_cache_gb(40, 512, 64, 2, 32_768, 16)kv_cache_gb(40, 512, 64, 2, 131_072, 16)
Per-token KV: 90.0 KiB | 32K context x 16 concurrent streams: 45.1 GBPer-token KV: 90.0 KiB | 128K context x 16 concurrent streams: 180.3 GB
As context length and concurrency increase, KV cache can grow from tens of GB to over 100 GB. This directly determines how many cards are needed and what maximum batch size is feasible.
2.6 Summary budget
| Item | Estimated value | Notes |
|---|---|---|
| Raw 4-bit weights | 1400 GB | Theoretical lower bound |
| Quantization scale factors | 87.5 GB | One 8-bit factor per 32 elements |
| Unquantized/non-4-bit tensors | About 30–70 GB | Embedding/output layers, some attention parameters, etc. |
| Alignment and sharding overhead | About 15–45 GB | Roughly 1%–3% of weights |
| Runtime workspace | About 32–80 GB | Roughly 5%–10% of weights; varies with batch size and parallelism |
| KV cache | About 45–180 GB | Example: 32K–128K context × 16 concurrent streams |
| Subtotal, weights only | About 1.53–1.64 TB | Excludes KV cache and runtime workspace |
| Total, including KV cache and workspace | About 1.6–1.9 TB | Reserve at least 10% headroom |
One often overlooked detail: GPU capacities are usually labeled in GiB, while model weight byte counts are often calculated in decimal units.
- 1487.5 GB = 1487.5 × 10⁹ ÷ 2³⁰ ≈ 1385.1 GiB
- Both may be called “GB,” but the two conventions differ by about 7.4%
If the conventions are mixed, unit conversion alone can create a gap of more than 100 GB.
3. Hardware selection: card count, topology, and parallelism
3.1 Single node or multiple nodes
Mainstream training and inference accelerator cards typically provide about 80 GB to 300 GB of memory per card. The following evaluation assumes a single node with 8 cards:
| Per-card memory | Total for 8 cards | Can it fit the model, about 1.6–1.9 TB? | Deployment form |
|---|---|---|---|
| 80 GB class | 640 GB | Insufficient; about 24 or more cards are needed | 3 eight-card nodes |
| 140 GB class | 1.12 TB | Insufficient; 16 cards are needed | 2 eight-card nodes |
| 180 GB class | 1.44 TB | Enough only for weights, with no headroom for KV cache | At least 2 nodes |
| 280–300 GB class | 2.24–2.40 TB | Yes | Single node |
Key conclusion: accelerator cards with 140 GB class or lower per-card memory cannot fit this model in a single 8-card node. Even if enough cards are assembled, multi-node deployment is still required.
3.2 Parallel strategy: expert parallelism and tensor parallelism
| Parallelism method | Target | Main communication | Characteristics |
|---|---|---|---|
| Tensor parallelism, TP | Dense layers such as attention and embeddings | all-reduce / all-gather | Frequent communication; best kept within a node |
| Expert parallelism, EP | Routed experts | all-to-all | Main scaling method for MoE |
| Pipeline parallelism, PP | Partitioned by layers across nodes | point-to-point, P2P | Friendly to cross-node deployment, but introduces pipeline bubbles |
| Data parallelism, DP | Replicates the full model | Gradient synchronization, not needed for inference | Main method for inference scaling |
A typical configuration is “TP within a node + EP/PP across nodes.” The reasons are:
- During decoding, TP requires an all-reduce synchronization for every generated token, making it the most communication-intensive pattern.
- EP all-to-all traffic is large during prefill, but it is easier to overlap with computation.
Keeping high-frequency communication inside a node and leaving only low-frequency or overlappable communication to cross-node links is the basic topology principle for trillion-scale MoE deployment.
3.3 Why all-to-all is the core communication for MoE deployment
When experts are distributed across 16 cards, with about 56 experts per card, the 16 experts selected for a token are expected to be distributed across about 16 cards. This means:
- Almost every card must send data to all other cards.
- After computation, results must be sent back along the same paths. This is the dispatch and combine process.
Example: with hidden dimension 7168, bf16, and expert parallelism degree 16, expert communication per token is approximately:
- 2 × 7168 × 2 bytes = 28,672 bytes, or 28 KiB, for dispatch plus combine
For an 8K-token prefill, this communication pattern mainly affects prefill-side latency: TPOT, or time per output token, is not the direct metric for this phase, while TTFT, or time to first token, increases as batch size grows. Because latency requirements differ by workload, batch size should be increased only when it still satisfies latency constraints.
4. Cost breakdown: how much does it actually cost?
Local deployment costs can be divided into three parts: one-time capital expenditure, ongoing operational expenditure, and unit output cost.
4.1 One-time capital expenditure: starting at the multi-million CNY level
| Item | Example configuration | Estimated cost |
|---|---|---|
| Accelerator servers | 2 eight-card machines with 140 GB class per-card memory | 2–4 million CNY |
| Inter-node network | 8 high-speed NICs per node plus switching equipment | 0.2–0.5 million CNY |
| Storage | About 50 TB of usable all-flash storage for weights, checkpoints, and logs | 0.1–0.3 million CNY |
| Power and cooling | About 8–10 kW per node, requiring corresponding data-center power and cooling capacity | 0.1–0.3 million CNY |
| Total | About 2.5–5.0 million CNY |
If a single-node solution using 280 GB class or higher cards is selected, total hardware cost may be similar or even lower, but it can eliminate cross-node networking and the associated engineering complexity.
4.2 Ongoing operations: electricity and staffing are often underestimated
- Electricity: A 16-card system consumes about 12–16 kW. Including cooling and power distribution losses, with a PUE of 1.2–1.5, total power demand is about 15–24 kW. Annual consumption is about 130,000–210,000 kWh, and annual electricity cost is about 0.1–0.2 million CNY.
- Operations staffing: Cluster scheduling, hardware replacement, and model/framework upgrades require at least one dedicated owner. A shared staffing estimate is about 0.2–0.5 million CNY per year.
- Total: Ongoing operating cost is about 0.3–0.7 million CNY per year. Adding depreciation amortized over 3 years, about 0.85–1.65 million CNY, the annualized total cost is about 1.2–2.3 million CNY.
4.3 Unit cost: CNY per million tokens
Theoretical peak throughput is about 10,800 tokens/s. Let sustained throughput be T, in tokens/s, and annual utilization be U:
- Annual token output = T × U × 3600 × 24 × 365
- Cost per million tokens = annualized total cost ÷ (annual token output ÷ 10⁶)
ANNUAL_COST = 1.75e6 # Annualized cost (CNY): hardware amortized over 3 years + electricity + operationsdef cost_per_million(t_tokens, utilization):tokens_per_year = t_tokens * utilization * 365 * 24 * 3600return ANNUAL_COST / (tokens_per_year / 1e6)for t in (4_000, 8_000, 10_000):row = " | ".join(f"{cost_per_million(t, u):>6.1f}" for u in (0.3, 0.5, 0.7))print(f"{t:>6} tok/s | {row}")
Sustained throughput (tok/s) | 30% | 50% | 70%4,000 | 57.9 | 34.7 | 24.88,000 | 29.0 | 17.4 | 12.410,000 | 23.2 | 13.9 | 9.9Unit: CNY per million tokens
The pattern is clear: unit cost is approximately inversely proportional to sustained throughput multiplied by utilization. When utilization rises from 30% to 70%, unit cost drops by about 57%. Only clusters that are continuously busy and highly utilized can reduce cost per million tokens to the low tens of CNY or even single digits.
4.4 Three routes: self-built, reserved cloud, or on-demand cloud
| Route | Entry threshold | Fixed cost | Elasticity | Suitable scenario |
|---|---|---|---|---|
| Self-hosted cluster | Multi-million CNY CAPEX plus data-center capability | High | Expansion requires additional procurement | Data cannot leave the domain; 7×24 high load |
| Reserved cloud instances | Pay by term | Medium | Medium | Stable workload; want to avoid operations |
| On-demand or spot cloud instances | Pay for usage | Low | High | Variable workload; testing and validation |
If a cloud-based route is selected, high-memory GPU cloud servers and supporting infrastructure such as object storage, logging, and monitoring may be available from providers such as Baidu AI Cloud. Availability, pricing, and billing options should be checked before commitment.
A simple break-even rule, illustrative only and excluding network and storage costs, is:
- Self-hosting is more economical than renting when: self-hosted annualized total cost < cloud unit price × annual request volume × average tokens per call
- Otherwise, on-demand or reserved cloud resources are the better choice.
In addition, self-hosting means bearing hardware depreciation and technology iteration risk. Accelerator models and quantization formats continue to evolve. Unit compute cost may decline significantly in two to three years, so residual value should be assessed before making a long-term asset commitment.
5. Practical ways to reduce cost and improve efficiency
Model and quantization
- Mixed-precision quantization: Keep higher precision for sensitive modules such as attention, while keeping expert weights at 4-bit to reduce memory with controlled quality loss.
- Tiered weight placement: Keep frequently accessed experts resident on accelerators, and place cold experts in main memory or local all-flash storage for on-demand swapping. This is a special optimization and requires measuring routing hit rate and swap-in latency first.
Hardware and parallelism
- Prefer large-memory single-node configurations: Reduce cross-node hops and lower all-to-all latency.
- Topology-aware parallelism: Keep TP within a node and EP/PP across nodes. Choose collective communication implementations that support topology awareness.
- Network upgrades: Use full interconnect within nodes and high-speed RDMA between nodes. Select NICs according to card count when budget allows.
Serving and scheduling
- Scheduling optimization: Use continuous batching, aggregate requests by expert routing, and apply grouped routing constraints to improve expert hit rate.
- Length management: Use tiered context caching and pool scheduling by input length to control KV cache peaks.
- Capacity planning: Work backward from maximum context length and concurrency to estimate KV cache budget, avoiding forced concurrency reductions after launch.
Operations
- Monitoring metrics: Track memory usage, batch size, expert load distribution, all-to-all duration, TTFT, and TPOT.
- Automatic scaling: Use container platforms that can scale automatically based on queue depth.
- Off-peak scheduling: Run offline batch jobs during low-traffic periods to improve utilization.
6. Deployment checklist
Before launch
- Confirm total parameter count, number of experts, top-k, and dense parameter ratio
- Confirm compatibility of the quantization format, including weight bit width, block size, and scale-factor bit width, with the target accelerator
- Confirm that the memory budget includes scale factors, unquantized tensors, runtime workspace, and KV cache
- Confirm that card count and parallel topology match communication capabilities
- Complete capacity and concurrency stress testing
- Perform a break-even analysis of annualized total cost against business request volume
Conclusion
Local deployment of trillion-parameter MoE models is a combined problem: memory determines feasibility, bandwidth determines speed, and utilization determines cost.
- The 1.79% expert activation ratio only reduces compute consumption, while about 1.49 TB of resident weights cannot be reduced.
- Omitting scale factors and runtime overhead from the memory budget is one of the most common causes of failed launches.
- When sustained throughput and utilization are insufficient, centralized inference services or elastic cloud resources may be more economical.
Calculate these two budgets first using the method described above, then decide whether to self-host, reserve cloud capacity, or pay for usage. This is the way to spend the budget where it matters most.
