How Much Does It Cost to Deploy a Trillion-Parameter LLM On-Premises?
A practical cost breakdown for self-hosting a sparse MoE model with roughly 3 trillion total parameters. It explains why VRAM demand follows total parameter count rather than activated parameters, why KV cache is a second cost center, and why MoE performance hinges on communication. It also gives a four-step method for estimating VRAM and server count, three budget tiers ranging from 2–3 eight-GPU servers for validation to 16 or more for high availability, and the networking, storage, power, software, and staffing costs that make up the full TCO. A deployment optimization checklist and common budgeting mistakes round out the guide.
TL;DR
Assuming 100 GB-class memory per GPU and 8 GPUs per server, with current market prices of roughly RMB 200,000–400,000 per card:
- Minimum viable: three 8-GPU servers, about RMB 5M–9M in total (~USD 700K–1.3M).
- Production-recommended: around eight 8-GPU servers (~64 GPUs), about RMB 15M–25M (~USD 2M–3.5M).
- High availability / high concurrency: 16 or more servers (~128 GPUs), starting at about RMB 30M (~USD 4M+).
These are server-only figures, and GPU prices swing with supply and demand, so get a live quote. They correspond to a sparse MoE model with roughly 3T total parameters, about 1.5 TB of 4-bit quantized weights, and a per-request context of up to a million tokens. The budget also grows with parallelism topology, networking, data center, and operations, as explained below.
1. Why Trillion-Parameter MoE Models Are Especially Costly to Self-Host
1.1 Total parameters set the weight footprint; activated parameters only set per-GPU compute
Sparse mixture-of-experts (MoE) models contain many expert subnetworks, and each token is routed to only a few of them. Inference therefore costs much less compute per token than a dense model of the same size. But every expert’s weights must stay resident in GPU memory so they can be dispatched on demand. Deployment cost is driven by the total parameter count, not the activated parameter count, which is the easiest thing to get wrong in on-prem planning.
1.1 (English: see above) — heading retained as in source structure
Recent trillion-parameter MoE models generally do two things at the attention layer:
- compress history into a smaller state with linear attention;
- cut attention compute with sparse attention.
They also use attention variants that fold several feature projections into a low-rank latent vector, often called MLA (multi-head latent attention). These designs sharply reduce the memory and compute cost of long context, which is what makes million-token contexts feasible. But the KV cache is still there:
- the higher the concurrency and the longer the context, the more GPU memory the KV cache needs;
- under high concurrency with long context, KV cache usage can approach the size of the weights themselves.
1.3 For MoE, the bottleneck is communication, not compute
When experts are sharded across devices, every token goes through a dispatch–expert compute–combine cycle that requires an all-to-all exchange.
- Inter-node network bandwidth is far lower than intra-node interconnect bandwidth.
- If the network jitters or congests, GPUs sit idle waiting for data, and compute utilization drops sharply.
So intra-node interconnect, network bandwidth, collective communication libraries, and kernel quality together determine actual throughput. Many teams spend half their budget on GPUs but never approve the other half meant for networking and engineering. The result is a cluster that fails to deliver the performance they paid for.
2. Sizing GPU Memory and Server Count: A Four-Step Method
Step 1: Estimate the weight footprint
- At 4-bit quantization, each parameter takes about 0.5 byte.
- A model with about 3T total parameters needs roughly 1.5 TB of weights.
- At 8-bit or higher precision, the footprint grows several-fold.
A single 8-GPU server has about 0.8 TB of GPU memory, so the weights alone have to span more than two servers.
Step 2: Estimate the KV cache
KV cache bytes per token ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
The factor of 2 accounts for one copy each of key and value. Models with low-rank latent compression can bring per-token usage down to tens of KB. A rough peak KV cache estimate:
# Illustrative only; use the parameters of the target model's actual architectureavg_bytes_per_token = 96 * 1024 # Assumed: ~96 KB/token with a hybrid-attention designpeak_tokens = max_concurrency * max_context_lenkv_cache_gib = peak_tokens * avg_bytes_per_token / 1024**3print(f"Peak KV cache: ~{kv_cache_gib:.0f} GiB")# Typical results (illustrative)# Concurrency 32 x 8K -> ~ 24 GiB# Concurrency 128 x 128K -> ~1536 GiB
Step 3: Add activation and runtime overhead
- Activation memory: depends on batch size and sequence length.
- Framework overhead: CUDA contexts, communication buffers, memory fragmentation, and the like. Allow 10–15% headroom.
Step 4: Convert to server count
Required GPU memory = weights + peak KV cache + activation and runtime overheadUsable memory per server = GPUs per server × GPU memory × utilization factor (~0.85)Servers per replica = required GPU memory / usable memory per server (round up)Production server count = servers per replica × number of replicas + hot standby node(s) (keep at least 1)
Example: at concurrency 32 and an 8K context, the requirement is about 0.8 (weights) + 0.02 (KV) + 0.12 (overhead) ≈ 0.95 TB. That means 2–3 servers per replica and about 5 for dual replicas. At concurrency 128 and a 128K context, the KV cache alone grows to about 1.5 TB and the total to about 2.4 TB. That means 4 servers per replica, and 9 in total for dual replicas plus one hot standby. Changing just these assumptions nearly doubles the server count. In practice, benchmark with your target production traffic first, then size the deployment from the results.
3. Three Configuration Tiers and Their Budgets
| Tier | Typical configuration | Use case | Main trade-off | Server budget (indicative) |
|---|---|---|---|---|
| Minimum validation | 2–3 eight-GPU servers | Proof of concept, prototyping | Concurrency in single digits; context must be tightly capped | ~RMB 5M–9M (~USD 700K–1.3M) |
| Production-recommended | 8 eight-GPU servers (~64 GPUs) | Dual replicas for availability; tens of concurrent users | Requires high-speed networking and dedicated operations staff | ~RMB 15M–25M (~USD 2M–3.5M) |
| High availability / high concurrency | 16+ eight-GPU servers (~128 GPUs) | 3 replicas; multi-rack or multi-site deployment | Procurement, facility, and operations costs all rise | Starting ~RMB 30M (~USD 4M+) |
The jump from minimum validation to production-recommended is usually not linear. Once you require multiple replicas, high concurrency, and long context, single-replica scale, network specifications, and operational complexity all rise together.
Beyond servers, a full TCO (total cost of ownership) should include at least:
- Networking: the parameter, application, and storage planes are usually built as separate networks. For a high-spec RDMA network, optics, switches, and cabling can account for 10–20% of the server budget.
- Storage: you need space for about 1.5 TB of weights, multiple versions and precisions, and local caching for fast loading.
- Power and cooling: a fully loaded 8-GPU server typically draws 8–10 kW, so 8 of them draw 64–80 kW. At RMB 0.5/kWh (~USD 0.07) with a PUE of 1.3, electricity costs roughly RMB 300K–450K (~USD 40K–65K) per year. If the facility cannot supply enough high-power racks, power and cooling upgrades need their own line item.
- Software and implementation: inference framework adaptation, quantization evaluation, monitoring and alerting. Allow 10–20% of the hardware budget.
- Staffing: 2–3 engineers experienced in distributed inference and cluster operations. Long-term staffing often costs more than the one-time hardware purchase.
4. Key Points for a Successful Deployment
Buying the hardware is only the beginning; results depend on tuning software and hardware together. These are industry-standard techniques:
- Low-bit quantization: run weights natively in a 4-bit-class floating-point format such as MXFP4 (a 4-bit float format with block-shared scaling factors). This can significantly shrink the weight footprint and ease GPU memory bandwidth pressure. Validate accuracy loss on your business data before going live.
- Attention kernels and KV cache management: efficient kernels for hybrid attention, plus paged KV cache management (allocate on demand to reduce fragmentation, similar to virtual memory paging in operating systems), improve effective memory utilization under high concurrency.
- MoE kernels and communication optimization: low-precision matmul optimizations, plus topology-aware expert parallelism and all-to-all optimization. The goal is to overlap communication with compute and cut tail-latency jitter, so network bandwidth turns into real throughput.
- Speculative decoding: use the model’s built-in multi-token prediction (MTP) module, which predicts several extra tokens in a single forward pass, to propose draft tokens and cut the number of serial decoding steps. The theoretical speedup is significant, but actual gains depend on the task and hardware.
- Layered parallelism: keep tensor parallelism within a single 8-GPU node to take advantage of fast intra-node interconnect and reduce communication overhead. When expert parallelism spans machines, prefer node counts that are powers of two so all-to-all communication partitions cleanly. Pipeline parallelism can split layers across machines, but it introduces pipeline bubbles, so weigh the trade-off.
On the framework side, mainstream open-source inference frameworks now support MoE expert parallelism, speculative decoding, and paged KV caches fairly well. That is one important reason to validate in the cloud first and then decide whether to self-host.
5. Decision Checklist and Common Mistakes
Before submitting the budget request, answer these questions:
- Does the use case really need self-hosting? For Q&A, copywriting, and lightweight coding assistance, a hosted model service is usually more cost-effective. Self-hosting has clear advantages mainly when data cannot leave the internal network, deep customization is needed, or call volume is very high.
- What are the real concurrency and context length distribution? Size against peak concurrency and P95/P99 input lengths, not averages.
- What are the latency targets? Define time to first token (TTFT) and time per output token (TPOT) separately.
- How much precision do you need? 4-bit, 8-bit, or higher directly determines hardware scale.
- Do you have the operations capability? Without a dedicated team, cluster availability is hard to guarantee.
Five common mistakes:
- Budgeting only for GPUs while ignoring networking, facilities, and electricity.
- Sizing with activated parameters and forgetting that all weights must be resident.
- Ignoring KV cache growth from long context.
- Launching with a single replica, so any node failure takes the service down.
- Relying on public benchmarks instead of business testing; good scores do not mean the service is usable.
Summary: The cost of self-hosting a trillion-parameter MoE model comes down to three parts: GPU memory for the weights, GPU memory for the KV cache, and the networking and engineering needed to use that memory efficiently. Use the four-step method to size a single replica, choose a tier based on your concurrency and availability requirements, and fold networking, facilities, power, and staffing into the TCO. That is how you build a budget that holds up to review.
