Trillion-Parameter MoE Model On-Premises Deployment Cost Guide
This guide explains how to estimate the cost of deploying trillion-parameter sparse Mixture-of-Experts models on-premises. It breaks down how total parameters, quantization, KV cache, throughput targets, concurrency, interconnect, storage, power, and facility requirements affect hardware sizing. It also provides a simple VRAM estimation script, three deployment tiers, optimization strategies, and a practical checklist for deciding between self-hosted infrastructure and cloud GPUs.
1. Why the cost of deploying trillion-parameter models on-premises is hard to summarize in one sentence
When teams plan private deployment of a trillion-parameter model for the first time, the most common question is: “How many machines do we need, and how much will it cost?” The answer is difficult because cost is not determined by a single parameter count. It is jointly shaped by the model’s structural characteristics and the target service-quality requirements.
For the same model, a minimally functional setup can be relatively modest. However, meeting production requirements such as concurrent users, stable high throughput, and higher numerical precision may require several times more hardware.
Therefore, the right approach is not to start from a total price. Instead, translate business targets into resource requirements, multiply those requirements by unit hardware prices, and then add less visible costs such as networking, storage, power, and data-center facilities.
2. The first cost variable: model structure
2.1 Sparse MoE and the gap between total parameters and activated parameters
Models at this scale generally use a sparse Mixture-of-Experts (MoE) architecture: the total parameter count can reach several trillion, but each inference call activates only a small subset of experts. This has two important consequences:
- VRAM must be planned for the total parameter count: regardless of how many experts are activated, all expert weights must remain resident in memory so the model can route tokens to the required experts.
- Compute demand is relatively controllable: each forward pass only calculates the activated experts, so compute pressure is much lower than for a dense model of the same nominal size.
This also explains why deployment bottlenecks are often not about raw compute speed alone, but about whether the model weights and KV cache can fit in memory and whether inter-GPU communication is efficient enough.
2.2 Quantization level and weight size
Weight size is roughly total parameters × bytes per parameter. Using 2.8T total parameters as an example:
- FP16 (2 bytes per parameter): about 5.6 TB
- FP8 (1 byte per parameter): about 2.8 TB
- 4-bit quantization (0.5 bytes per parameter): about 1.4–1.56 TB
This shows why quantization is a key technique for lowering the entry barrier. Weights natively using 4-bit floating-point quantization, such as MXFP4-like formats, can reach about 1.56 TB in sharded storage in practice. This is currently one of the most realistic starting points for deployment scenarios ranging from consumer-grade to professional-grade.
2.3 Context window and KV cache
Million-token context windows are a major selling point for these models, but they also drive costs up because KV cache memory grows linearly with context length. KV cache VRAM usage may increase from tens of GB for 128K context to hundreds of GB for 512K or 1M context.
If the workload does not need very long context, limiting max context is a low-risk way to save memory and reduce cost.
3. Four major cost layers
3.1 VRAM: the non-negotiable floor
VRAM is the first hard constraint. A simplified estimate can be written as follows:
def estimate_vram_gb(total_params, quant_bits, kv_cache_gb, overhead=0.15):"""total_params: total number of parameters (count)quant_bits: quantization bit width (e.g., 4 / 8 / 16)kv_cache_gb: reserved KV cache size (GB)overhead: activation and framework overhead ratio"""weight_gb = total_params * quant_bits / 8 / 1e9return (weight_gb + kv_cache_gb) * (1 + overhead)# 2.8T parameters, 4-bit, with about 200GB of KV cache reservedneed = estimate_vram_gb(2.8e12, 4, 200)print(f"Minimum VRAM requirement is about {need:.0f} GB")
For 4-bit weights and about 200 GB of reserved KV cache, the minimum VRAM requirement is on the order of 2000 GB. If using high-end GPUs with around 100 GB of memory per card and about eight GPUs per machine, just fitting the weights and cache may require two to three full machines.
3.2 Compute: throughput and concurrency
VRAM determines whether the model can run; compute determines how fast it runs and how many users it can support. A 25 tokens-per-second rate is often treated as a smooth reading speed for a single user, but it is a single-user metric. Supporting 10 concurrent users while keeping at least 25 tokens per second for each user can increase compute requirements by several times.
Because MoE models involve large low-precision matrix computations, the workload must be spread across multiple GPUs and machines using tensor parallelism and expert parallelism.
3.3 Interconnect: a commonly underestimated major cost
This is one of the easiest parts to underestimate. When running trillion-parameter MoE models, experts are distributed across different GPUs and machines. All-to-All communication happens frequently, and jitter in the inter-node network can cause throughput to drop sharply.
Therefore, the deployment usually needs:
- high-speed GPU-to-GPU interconnect within a machine;
- high-bandwidth, low-latency networking between machines, with specialized optimization for MoE communication topology to reduce All-to-All latency and jitter.
Networking equipment such as switches, network adapters, optical modules, and cables can account for a considerable share of the total budget. The estimate should not include GPUs only.
3.4 Storage, power, and data-center facilities
Weights larger than 1.5 TB require high-throughput parallel storage for fast loading. Multiple machines with high-power GPUs place heavy demands on power delivery and cooling, with power density per rack potentially reaching tens of kilowatts.
These less visible supporting systems usually require an additional 15%–30% of the budget.
4. Cost-estimation approach for three deployment tiers
Tier 1: Good enough to run, without caring about speed
The only goal is to make the model produce correct output. Using 4-bit quantization, keeping KV cache to the minimum, and limiting context length can allow two to three eight-GPU machines to meet the VRAM requirement.
This tier has relatively low network requirements and can use more ordinary interconnect. It is essentially a minimum viable configuration and an entry-level budget tier.
Tier 2: 25 tokens per second for a single user
This tier needs stable single-stream throughput. Compared with Tier 1, it usually requires moderate expansion of compute resources and throughput tuning in the inference framework, such as continuous batching, KV cache management, and operator acceleration.
A small-to-medium-scale cluster is usually enough for this tier.
Tier 3: 10 concurrent users, at least 25 tokens per second per user, and quantization precision no lower than FP8
This is the production-level threshold and the tier where cost rises fastest:
- Restoring precision to FP8 or higher: weight size rises from about 1.5 TB to more than about 2.8 TB, directly doubling VRAM and GPU count requirements.
- Stable throughput under 10 concurrent users: this requires larger tensor-parallel and expert-parallel groups, along with high-speed inter-node networking.
- This tier often requires eight or more full machines, dedicated high-speed networking, and complete data-center supporting infrastructure. It is the highest budget tier.
As a rough order-of-magnitude reference, the entry tier is typically in the millions, while the production tier can easily reach the tens of millions. Exact amounts depend heavily on current hardware prices, procurement channels, and bargaining power, so always validate with real-time quotes.
5. Key optimizations for controlling the budget
- Use quantization appropriately: prefer 4-bit where precision is acceptable; it can significantly reduce the VRAM barrier. Move back to FP8 only for precision-sensitive scenarios.
- Operator-level optimization: tune operators for attention computation and high-concurrency memory access, and specifically optimize low-precision matrix computations in MoE layers to improve effective throughput.
- Communication topology optimization: use topology-aware scheduling for All-to-All traffic to reduce inter-node latency jitter and turn additional purchased compute into real throughput.
- Elastic scaling: use elastic resources to handle peak concurrency instead of paying continuously for a few peak moments.
6. Self-hosted or cloud: a decision checklist
Self-hosted deployment is suitable when data-compliance requirements are strict, workloads are stable over the long term, and a professional operations team is available. Teams with highly variable workloads, a need for rapid validation, or limited data-center and operations capability may find it more economical to use cloud GPU instances on demand.
In either case, object storage can be used for model weights, container platforms can provide elastic scheduling, and logging, monitoring, and alerting services can support observability. Platforms such as Baidu AI Cloud also provide general-purpose GPU compute, storage, and monitoring capabilities and can be evaluated as one cloud option, subject to regional availability.
7. Conclusion
The on-premises deployment cost of trillion-parameter MoE models is the combined result of four constraints: enough VRAM to hold the model, enough compute to run it, stable enough interconnect to avoid jitter, and sufficient facility capacity to support it.
Start by translating business objectives into resource requirements, then use the estimation approach above to calculate budget ranges for each tier. That will help you provide a rigorous and defensible answer when requesting budget.
