How Much Does It Really Cost to Deploy a Large-Scale MoE Model Locally?
This article breaks down the real cost of deploying a trillion-parameter Mixture-of-Experts (MoE) model locally, using a ~2.8-trillion-parameter model as a working example. It argues that the binding constraint is VRAM capacity rather than compute, then builds a full memory budget—microscaling 4-bit weights (~1487.5 GB), KV cache, and runtime workspace—totaling roughly 1.6–1.9 TiB across a cluster. From there it shows how that memory floor translates into GPU count and parallel topology (TP/EP/PP/DP/CP), why decode is bandwidth-bound while prefill is compute-bound, and what the wider bill includes: hardware, network, storage, power, cooling, and manpower. It also provides a CapEx/OpEx estimation framework, a reusable VRAM estimation script, six practical cost-reduction levers, and guidance on weighing a self-built data center against the cloud. Readers come away with a clear, layered method for projecting total cost of ownership (TCO) before procurement.
Introduction: The Real Cost Barrier Is VRAM, Not Compute
The MoE (Mixture-of-Experts) architecture scales to trillions of parameters on a simple idea: “many parameters, but only a fraction used at a time.” Sparse activation drives down the per-token compute cost to a small share of the total parameter count. Many teams infer from this that although the model is huge, only a few percent is computed at each step, so a handful of accelerators should be enough to run it.
This intuition is only half right. Sparse activation determines the compute demand, but at inference time the weights of all experts must stay resident in VRAM or memory and be fetched on demand. What really determines “how much it costs” is where those weights are stored and how they are accessed.
Using a MoE model with roughly 2.8 trillion total parameters as a sample, this article starts from the VRAM budget and works through card count, topology, bandwidth, compute, peripherals, and manpower. It offers a reusable estimation method and cost-reduction paths so you can settle the full bill before procurement.
1. Break Down the Model Structure First: Which Numbers Drive Cost
| Structural parameter | Example value | Impact on deployment cost |
|---|---|---|
| Total parameters | 2.8 trillion | Sets the VRAM floor for resident weights; the single largest cost item |
| Attention backbone | Linear attention variant (Delta Attention) plus residual connections, with gated latent attention (MLA) in some layers | 1) Linear attention keeps the decode state roughly constant at long context; 2) compressed KV cache greatly reduces per-token cache cost |
| Expert count | 896 experts | Routing-table size; affects how parallelism is partitioned and the all-to-all communication volume |
| Activation strategy | 16 experts activated per token | Determines per-token compute; no effect on resident VRAM |
| Expert activation ratio | 16 ÷ 896 ≈ 1.79% | Easily confused with “parameter activation ratio”; the two must be distinguished |
| Weight format | Microscaling 4-bit float (MXFP4-like), with quantization-aware training from the SFT stage | Compresses weight size to about a quarter of BF16, but still around 1.4 TB |
1.1 Attention Backbone: Understand the Structure Before Talking Cost
The attention mechanism of mainstream large models—whether scaled dot-product attention (MHA), grouped-query attention (GQA), or gated latent attention (MLA)—has a per-token KV cache that grows linearly with context length. Ultra-large MoE models often use a hybrid backbone of “a linear attention variant + a few gated latent attention layers”:
- Linear attention variant: borrows the Delta rule, updating the state via “erase first, then write” to avoid interference from old information. Its per-token state size stays roughly constant over long sequences rather than bloating as context grows, and it supports chunked parallel training.
- Gated latent attention layers: first compress KV into latent vector caches in a low-rank manner, then use a gating mechanism to modulate information flow. Its compressed per-token cache is far smaller than standard MHA/GQA.
The most direct benefit for deployment is that the KV cache is no longer an uncontrollable variable. The VRAM budget below quantifies this benefit.
1.2 Expert Activation Ratio: What Does 1.79% Actually Save?
With 896 experts and 16 selected per token, the expert activation ratio is about 1.79%. But three points must be clear:
- 1.79% is the “expert activation ratio,” not the “parameter activation ratio.” The attention backbone and shared layers participate in computing every token, so the actual number of parameters involved per token is necessarily higher than 1.79% of the total.
- VRAM is counted by total parameters. The router only knows which experts the current token should go to; it cannot predict subsequent requests. All 896 experts must stay resident in the cluster, ready at all times. The activation ratio only affects compute.
- Cluster total vs. single node. Counting VRAM by total parameters refers to the whole serving cluster; with expert parallelism, a single node holds only one shard of all the experts.
Here is a corrected example: suppose experts account for 90% of the parameters and the remaining 10% is computed for every token. Then the parameters actually involved per token are approximately:
2.8T × 10% + 2.8T × 90% × 1.79% ≈ 325B
That is several times higher than the intuitive estimate of “50B” (2.8T × 1.79%). Ignore this and the compute budget will be severely underestimated.
2. The VRAM Budget: From 1487.5 GB to Actually Running
2.1 The Physical Floor of the Weights
Start with the ideal case: if the weights could be packed as tightly as ordinary 4-bit integers, the raw data floor would be:
2.8 × 10^12 × 4 bit ÷ 8 = 1.4 × 10^12 bytes = 1400 GB ≈ 1303 GiB ≈ 1.21 TiB
But the microscaling 4-bit format cannot be packed this tightly. Under the general specification, every 32 weight elements share one scale factor, which works out to:
- Each element actually occupies 4 + 8 ÷ 32 = 4.25 bit
- Elements plus scales total 2.8 × 10^12 × 4.25 ÷ 8 = 1.4875 × 10^12 bytes, i.e. 1487.5 GB ≈ 1385 GiB ≈ 1.35 TiB
VRAM is usually labeled in GiB/TiB, so switching to TiB gives a more intuitive view. This still excludes the non-4-bit tensors (token embeddings, normalization parameters, routing gates, etc.), framework metadata, shard alignment, and the runtime workspace.
2.2 Why “4.25 bit” Is Not an Optional Optimization
One seemingly clever approach is to store only the 4-bit elements and generate the scale on the fly at runtime. This does not work:
- The scale is not auxiliary data; it is part of the dequantization formula. The actual computation at inference is
y = (q × scale) ⊗ x: q is the 4-bit element, scale is the 8-bit scale factor for the corresponding block, and ⊗ is the block-scaled matrix multiplication. - Every step reads all expert shards. To save about 87 GB by dropping the pre-stored scales, you would have to regenerate them every step. This both consumes extra bandwidth and adds latency.
Therefore, 1487.5 GB should be treated as the planning baseline for weight VRAM, not a starting point for optimization.
2.3 The Dividend from Quantization-Aware Training
This sample model uses quantization-aware training (QAT) from the SFT stage: the forward pass simulates low-precision errors so the weights adapt to the 4-bit representation during training. Compared with “post-training quantization” done after training completes, it has two direct benefits:
- Controllable accuracy loss: the model has already learned to maintain performance at low bit widths.
- No extra calibration burden: there is no need to prepare a calibration dataset, and no worry about unexpected accuracy cliffs on certain tasks.
But QAT is no free lunch: the training stage must maintain both high-precision master weights and low-precision copies, which itself raises training cost. That cost has long since occurred on the model side; what the deployment side enjoys is “ready-to-use low-precision weights.”
2.4 The Second Commonly Underestimated Block: KV Cache
The KV cache is counted per token. Using the hybrid attention structure in the example:
- Suppose the backbone has L layers, of which L_mla layers use latent-attention compression, with a per-token cache dimension of d_c + d_rope. The example assumes about 576 elements per token per layer, i.e. 1152 bytes.
- The rest are linear attention layers, whose per-token state is constant, denoted S_lin. The example assumes about 2 KB in total.
So the single-token cache is roughly:
bytes_per_token ≈ L_mla × 1152 + S_lin
With L_mla = 3 and S_lin = 2048 bytes in the example, that is about 5.3 KB per token. The longer the context and the higher the concurrency, the faster this block grows:
| Scenario | Per-request/per-sequence KV cache |
|---|---|
| 128K context | ≈ 0.66 GiB |
| 1M context | ≈ 5.0 GiB |
| 32-way concurrency × 128K | ≈ 20.6 GiB |
| 32-way concurrency × 1M | ≈ 160 GiB |
This is where the value of the hybrid backbone lies. If you compute with a standard MHA cache instead—60+ layers, 128 heads, head dimension 128, BF16 storage—each token needs about 240 KB, and a single 1M-context request needs about 229 GiB. That one item alone would nearly consume all the VRAM, making deployment impossible to even discuss.
2.5 The Third Block: Runtime Workspace
Beyond weights and KV, you must reserve space for:
- Activations and intermediate tensors: state updates in linear attention, hidden states before MoE routing.
- Communication buffers: send/receive buffers for all-to-all dispatch and aggregation, and scratch space for collective communication.
- Framework overhead: the memory pool, fragmentation, and runtime metadata such as CUDA graphs / compilation caches.
As a rule of thumb, reserving 10%–20% of the weights is safe. For a trillion-scale MoE, that is about 150–300 GB.
2.6 Add the Three Parts Together
| VRAM component | Estimated magnitude | Notes |
|---|---|---|
| Weights (elements + scales) | ≈1.35 TiB | Physical floor, cannot be compressed |
| Non-4-bit tensors, metadata, alignment | ≈0.10–0.20 TiB | Varies by implementation |
| KV cache (example config) | ≈0.02–0.16 TiB | Depends on context length and concurrency |
| Runtime workspace | ≈0.15–0.25 TiB | Activations, communication buffers, fragmentation |
| Total | ≈1.6–1.9 TiB | Cluster-wide VRAM requirement |
A reusable estimation script:
def estimate_vram(total_params=2.8e12, # total parametersweight_bits=4.0, # effective weight bit widthblock=32, # microscaling block sizescale_bits=8, # shared scale factor bits per blockbytes_per_token=5300, # per-token KV cache bytes (incl. linear attention state)context_len=128_000, # context length per requestconcurrency=32, # number of concurrent sequencesoverhead_ratio=0.15, # reserve ratio for workspace, metadata and fragmentation):# 1) weights: elements + per-block scale factorsweights = total_params * (weight_bits + scale_bits / block) / 8# 2) KV cachekv_cache = bytes_per_token * context_len * concurrency# 3) reserve workspace as a ratio of weightstotal = (weights * (1 + overhead_ratio)) + kv_cachegib = 1024 ** 3print(f"Weights: {weights / gib:.0f} GiB")print(f"KV cache: {kv_cache / gib:.1f} GiB")print(f"Total: {total / gib:.0f} GiB ≈ {total / gib / 1024:.2f} TiB")return totalestimate_vram()# Weights: 1385 GiB# KV cache: 20.1 GiB# Total: 1613 GiB ≈ 1.58 TiB
3. Card Count and Parallel Topology: How to Split This 1.6 TB
3.1 Card Count Is Determined by “Total VRAM ÷ Per-Card VRAM”
- With 192 GB per card: 192 × 8 = 1536 GB, which leaves only about 48 GB above the 1487.5 GB weight floor—nowhere near enough once you add the workspace and KV cache. A single 8-card server therefore cannot carry it; you need at least two 8-card servers (16 cards). At 16 cards, each card holds about 93 GB of weights on average, leaving roughly 80 GB for KV and workspace.
- With 96 GB per card: you need about 20 cards just to fit the weights, usually rounded up to four 8-card servers (32 cards) in practice.
3.2 Confirm It Fits Before Asking Whether It Runs
VRAM determines “can it run,” bandwidth determines “how fast it runs,” and compute determines “how far it runs.” The three form a serial relationship: if it does not fit, performance is moot; if it fits but bandwidth is insufficient, decode throughput falls far below the theoretical value; if compute is insufficient, long-context prefill becomes the bottleneck.
3.3 Common Hybrid Parallelism Strategies
| Parallelism | What it splits | Main role in MoE inference |
|---|---|---|
| Tensor parallelism (TP) | A single weight matrix split by row/column | Splits attention and shared layers; introduces per-layer AllReduce |
| Expert parallelism (EP) | Different experts distributed across cards/nodes | Lowers per-card weight footprint; introduces all-to-all for token routing |
| Pipeline parallelism (PP) | Layers split across nodes | Cross-node scaling with relatively controllable communication |
| Data parallelism (DP) | Replicates the entire model | Increases concurrency throughput and availability |
| Context parallelism (CP) | Splits along the sequence dimension | Relieves KV and attention pressure under ultra-long context |
A common combination: use TP+EP within a node to confine the all-to-all for expert routing to the intra-machine high-speed interconnect domain; scale across nodes with PP or DP replicas.
3.4 The Expert Parallelism Degree Should Ideally Divide the Expert Count
896 = 128 × 7, with divisors 1, 2, 4, 7, 8, 14, 16, 28, 32, 56, 64, 112, 128, 224, 448, 896.
- Choosing a divisor such as EP=8, 16, or 32 gives each parallel group an integer number of experts (56 per group at EP=16), avoiding load skew from “remainder experts.”
- If EP=64 while each node has only 8 cards, routing must cross nodes and all-to-all traffic rises sharply.
3.5 Capacity Planning Also Needs Three Things
- Failure redundancy: at 16 cards, a single card failure interrupts service, so you need hot-standby cards or fast reconstruction.
- Adding replicas: as concurrency rises, horizontally adding whole model replicas is usually more cost-effective than further enlarging TP/EP on a single replica.
- Storage and network: the model files are about 1.5 TB; a fast local disk usually loads them in tens of seconds, while pulling from remote storage is bandwidth-bound. Cross-node deployment also needs an RDMA-class high-speed network and a sensible switching topology.
4. Bandwidth and Compute: The Real Cost of a Single Inference
4.1 The Decode Stage Is Bandwidth-Bound
During token-by-token decoding, every step reads the weights of the experts activated by that token from VRAM. The theoretical upper bound on single-card decode throughput is:
single-card ceiling ≈ HBM bandwidth ÷ bytes read per token per card
With the example parameters, a rough calculation gives about 2.8T × 90% × 1.79% × 0.53125 bytes (4.25-bit) ≈ 24 GB of expert weights read per token. Even if a single card reaches 8 TB/s of bandwidth, the theoretical ceiling is only about 320 tokens/s. The actual figure is further reduced by routing overhead, communication, and scheduling losses.
This also shows that, compared with a dense model of similar parameter count, MoE has a lower per-token cost; but the premise of “keeping all weights resident” means it remains a heavy-asset deployment.
4.2 The Prefill Stage Is Compute-Bound
When processing long-context inputs, the computation per token is about 2 × the actually-computed parameters. Estimating with the example’s ~325B, that is about 650 GFLOPs per token. A 1M-token input implies about 650 PFLOPs of computation—a demanding requirement on compute.
4.3 The Two Loads Have Different Characteristics
| Dimension | Prefill | Decode |
|---|---|---|
| Bottleneck | Compute | VRAM bandwidth |
| Parallelism | High, large matrices | Low, token-by-token serial |
| Optimization focus | Improve compute utilization, chunked processing | Increase batch size, reduce weight reads |
The two loads have different resource ratios, which motivates “prefill/decode disaggregated deployment”: let each type of instance configure resources around its own bottleneck to avoid mutual interference.
5. Hardware and Peripherals: Accelerators Are Only Part of the Bill
- Accelerators: the core metrics are not just peak compute but also per-card VRAM capacity, VRAM bandwidth, inter-card interconnect bandwidth, and native support for microscaling 4-bit weights and 8-bit activations. Without native support, dequantization overhead eats into part of the compute advantage.
- Servers and nodes: high-speed interconnect is usually concentrated within a single 8-card machine. Going beyond a single machine requires additional high-speed networking and the corresponding switches.
- Network: cross-node needs an RDMA-class high-speed network; if serving the public internet, it also needs load balancing and a gateway.
- Storage: about 1.5 TB of weight files, so evaluate both load time and redundancy (multi-replica or erasure coding).
- Power and cooling: a single node can draw on the order of 10 kW, so the data center needs matching power-distribution capacity and cooling, which directly affects site selection and retrofit costs.
- Software and manpower: inference frameworks, scheduling systems, monitoring and alerting, model evaluation, and the MLOps engineers who can maintain this system are all ongoing investments.
6. Cost Structure and Estimation Method
| Cost category | Main components | Billing/pricing characteristics |
|---|---|---|
| Upfront investment (CapEx) | Compute servers, high-speed network, storage, data center retrofit | Paid once by equipment and engineering, amortized over the depreciation period |
| Ongoing operations (OpEx) | Electricity, bandwidth, data center rental, hardware maintenance, ops manpower | Recurring by usage or subscription cycle |
| Software and licensing | Inference engines, scheduling platforms, monitoring tools | Open-source options carry no license fee but require manpower |
| Manpower | Deployment, tuning, ops, evaluation | Counted by team size and duration |
When estimating, use a “lower bound + headroom” approach:
- Lower bound: card count = total VRAM requirement ÷ per-card VRAM.
- Headroom: then add VRAM and compute margin for concurrency growth, failure redundancy, and model upgrades.
In one sentence: “can it run” is decided by VRAM, “does it run well” by bandwidth, and “can you afford it” by utilization and depreciation together.
A rough but practical rule: per-token cost ≈ (depreciation + electricity + manpower) ÷ effective token output, where effective output = peak throughput × actual utilization. Raising utilization from 30% to 60% nearly halves the per-token cost. So before scaling up, tightening scheduling and batching is often more worthwhile than adding hardware.
7. Cost-Reduction Paths: Six Actionable Levers
- Quantization and mixed precision: microscaling 4-bit weights with 8-bit activations, balancing accuracy and bandwidth.
- Parallel topology optimization: confine all-to-all to the intra-machine high-speed domain; scale across nodes with PP or DP.
- Prefill/decode disaggregation: configure resources independently for the two load types so long inputs do not slow down decode.
- Batching and cache management: increase batch size to amortize weight-reading cost; apply paging or tiered caching to the KV cache.
- Speculative decoding and similar techniques: use a draft model or small model to guess several tokens first, then have the large model verify them in parallel, reducing the large model’s forward passes.
- Scheduling and prioritization: schedule by SLA tier and run batch jobs during idle periods (e.g. offline evaluation, data synthesis) to raise overall utilization.
8. Self-Built or Cloud: How to Choose
Each path has its own fit:
| Dimension | Self-built data center | Cloud (e.g. Baidu AI Cloud’s GPU cloud instances/bare metal, elastic containers, object storage, and other general capabilities) |
|---|---|---|
| Upfront investment | High; one-time purchase | Low; pay as you go |
| Elasticity | Long scaling cycle | Fast scaling by load |
| Long-term stable high load | Unit cost may be better | Needs evaluation with a commercial plan |
| Ops complexity | Must build monitoring, scheduling, and fault-handling systems | Can leverage the platform’s monitoring/alerting, logging, storage, and other general capabilities |
| Compliance and data residency | Data fully self-held | Can choose a dedicated deployment option |
The decision rule can be simplified to:
- Long-term stable and large-scale load → lean toward self-built;
- Load with clear peaks and troughs, or in the validation phase → lean toward cloud;
- Strong constraints on data residency → prioritize private or dedicated deployment options.
Conclusion
Back to the original question: how much does it cost to deploy a 2.8-trillion-parameter MoE model? The answer is not a single number but a three-layer structure:
- The VRAM floor: weights of about 1487.5 GB, plus workspace and KV cache, give a cluster total of roughly 1.6–1.9 TiB, which directly determines the minimum card count.
- Performance and peripherals: bandwidth determines decode throughput, compute determines long-context prefill capability, and network, storage, power, and manpower complete the cost.
- Utilization and depreciation: these determine the actual per-token price and are the main battleground for cost reduction.
Only by accounting for and optimizing these three layers separately can you form a clear expectation of total cost of ownership (TCO) before procurement and avoid “buying the cards but not being able to afford the service.”
