Deploying Trillion-Parameter MoE LLMs On-Premises: Hardware Requirements and Cost Estimation
This article explains how to size hardware and budget for running very large sparse Mixture-of-Experts (MoE) models on your own infrastructure. It covers why memory capacity rather than raw compute is the hard constraint, how low-precision quantization, million-token context, and KV cache drive memory demand, why high-bandwidth interconnects and "super node" architectures are essential for expert parallelism, and how to build a working CapEx/OpEx estimation model instead of relying on generic list prices. Software-side levers such as tensor, expert, and pipeline parallelism, kernel-level optimization, and quantization trade-offs are also covered, along with practical decision-making and cost-control guidance.
1. Why “can we run it locally, and what will it cost?” became a hot topic
Over the past two years, a wave of open-weight models at the trillion-parameter scale have been released, and enterprises are actively evaluating private (on-premises) deployment, driven by data compliance, operational autonomy, and deep customization needs. But these models are on a completely different scale from the models of a few hundred million or a few billion parameters we used to work with. The question is no longer “can it run at all” but rather a set of engineering and financial questions: how much GPU memory, how many cards, how many machines, and how much budget.
The various “starting prices” and “recommended configurations” circulating online use inconsistent assumptions, and the root cause is that many people fixate only on GPU unit price while ignoring the memory wall, interconnect bandwidth, long-context memory overhead, and total cost of ownership (TCO). Below we first clarify the model characteristics that actually drive cost, then provide a practical estimation method.
2. Start with the architecture: this is where cost differences come from
2.1 Sparse MoE: huge total parameters, limited activated parameters
These models generally adopt a sparse Mixture-of-Experts (Sparse MoE) architecture, whose defining trait is “an enormous total parameter count, with only a small fraction activated per forward pass.” One representative open model, for example, has on the order of 2.8T (about 2.8 trillion) total parameters, yet each token activates only a subset of those experts.
This is critical for deployment: compute (FLOPs) is determined by the activated parameters, while memory is determined by all parameters. In other words, MoE sparsity reduces inference compute but does nothing to reduce the memory occupied by weights — you must load every expert’s weights into memory, ready to be routed at any moment.
2.2 Low-precision quantization: the first lever on memory footprint
To compress memory usage, weights for these models are often published in low-precision formats. Common options include FP8, 4-bit floating point (such as the MXFP4 microscaling format), and INT4. Estimating 2.8T parameters at roughly 0.5 bytes per parameter (4-bit):
- Weight footprint ≈ 2.8×10¹² × 0.5 byte ≈ on the order of 1.4 TB (including shard and metadata overhead, roughly 1.5–1.6 TB in practice)
These figures set a hard floor for memory: the weights alone reach the 1.5 TB level, which no single card can hold, so multi-card, multi-node coordination is mandatory.
2.3 Long context and KV cache: the easily overlooked memory sink
Flagship models have extended their context windows to the million-token level. The longer the context, the larger the cached historical Key/Value state (KV cache) during inference, and its memory cost can grow linearly — or faster — with sequence length and concurrent batch size. In high-concurrency, long-document scenarios it can approach or even exceed the weights themselves.
Some models use hybrid attention designs such as linear attention, sparse attention, and multi-token prediction (MTP) to relieve long-context memory and throughput pressure, but engineering practice still requires reserving ample headroom for the KV cache.
2.4 Summary: memory requirement = weights + KV cache + runtime overhead
In one sentence: the hard constraint on deployment cost is not “how fast is compute” but “does it fit.” This determines all of the hardware selection logic that follows.
3. Sizing the hardware: memory is the hard constraint, interconnect is the weak link
3.1 Memory capacity sets a floor on machine count
Start with a rough calculation (all units in GB):
weights_mem = params(B) × bytes_per_param # weight memorykv_cache_mem = 2 × num_layers × num_kv_heads × head_dim × seq_len × batch × dtype_bytes / 1e9runtime_extra = (weights_mem + kv_cache_mem) × overhead # framework/activations/fragmentation, use 0.15~0.25total_mem = weights_mem + kv_cache_mem + runtime_extramin_gpus = ceil(total_mem / memory_per_gpu)
Plugging in numbers: a data-center GPU with around 140 GB per card, at 8 cards per node, gives roughly 1.12 TB per node.
- Weights only: 1.6 TB ÷ 1.12 TB ≈ 2 nodes required;
- After adding long-context KV cache, the alignment overhead of tensor and expert parallelism, and the headroom needed for stable operation, real deployments typically land in the 6–8 full nodes range.
This is why the combination of “trillion-parameter MoE plus million-token context” starts at several high-memory whole nodes rather than one or two servers.
3.2 Why a “super node” architecture is necessary
The bottleneck for running trillion-parameter MoE is often not compute but cross-card and cross-node communication. MoE expert parallelism generates heavy All-to-All data exchange, and long-context attention involves frequent distributed memory access. If interconnect bandwidth is insufficient, GPUs spend much of their time “waiting for data” rather than “computing data,” and utilization drops sharply.
Production-grade designs therefore typically adopt a “super node” architecture:
- Inside a node: high-bandwidth, low-latency card-to-card interconnect (such as a high-bandwidth bus) so that 8 cards behave like “one large card”;
- Between nodes: high-speed RDMA networking (InfiniBand- or RoCE-class solutions) to reduce the latency and jitter of cross-node All-to-All.
When interconnect capacity is inadequate, even with enough memory and enough cards, throughput may be only a small fraction of theoretical peak.
3.3 Storage and loading
Weights at the 1.5 TB level require a high-speed parallel file system or object storage to support fast loading and distribution across nodes; otherwise every startup or scale-out event means waiting tens of minutes to several hours for weight retrieval. Storage is not the most expensive component, but it is frequently underestimated.
4. A methodology for cost estimation
4.1 Breaking down the cost structure
The total investment in on-premises deployment splits into one-time CapEx and ongoing OpEx:
- CapEx (one-time)
- GPU servers: the dominant share, typically 60%–80% of the total;
- High-speed interconnect: switches, NICs, cabling (RDMA fabric);
- Storage: parallel file system or object storage plus media;
- Facility fit-out: racks, power distribution (PDU/UPS), cooling.
- OpEx (ongoing)
- Electricity: fully loaded high-memory GPU nodes can draw several kW per node, and 24/7 operation adds up;
- Operations staff: cluster scheduling, fault handling, model iteration;
- Facility, bandwidth, and depreciation.
4.2 A practical estimation model
CapEx ≈ N_nodes × price_per_node+ networking equipment and cabling+ storage system+ facility fit-out (racks / power / cooling)OpEx_year ≈ node_power_draw(kW) × 24 × 365 × electricity_price+ operations staff+ bandwidth and other
Estimating with a typical configuration of “1.5 TB weights, million-token context, requiring 6–8 high-memory whole nodes”: a single high-memory node is typically in the low-millions-of-yuan range, and once networking, storage, and facility fit-out are added, the full cluster lands in the millions to tens of millions of yuan range, varying up or down with card count and memory specification. This confirms a common conclusion: for local deployment of this class of model, the “starting price” is in the millions, and a “recommended configuration” can reach tens of millions. It is better to plug your own concurrency and context requirements into the model above and fill in each term, rather than applying generic numbers from the internet.
4.3 A recommended cost checklist
| Category | Key items | Notes |
|---|---|---|
| Compute | High-memory GPU nodes ×N | Main cost driver; select by memory and card count |
| Network | RDMA switches / NICs / cabling | Determines MoE All-to-All efficiency |
| Storage | Parallel file system / object storage | Supports loading and distribution of 1.5 TB weights |
| Facility | Racks / power / cooling | High power draw needs dedicated power and heat removal |
| Operations | Electricity / staff / depreciation | Ongoing OpEx; do not look at purchase price only |
5. Software stack optimization: squeezing every card
The same hardware can deliver several times the difference in effective throughput depending on the software stack. Key optimization directions:
- Combining parallelism strategies: tensor parallelism (TP) splits computation within a layer, expert parallelism (EP) distributes experts, and pipeline parallelism (PP) splits by layer depth. The three must be configured in coordination with the network topology to avoid communication bottlenecks.
- Kernel-level optimization: optimizing memory and memory access for attention, and accelerating low-precision matrix multiplication for MoE, can substantially reduce memory-access overhead under high concurrency.
- Communication topology optimization: topology-aware scheduling built around All-to-All latency and jitter is often the deciding factor in large-scale MoE deployments.
- Quantization vs. throughput/latency trade-offs: more aggressive quantization saves memory and raises throughput, but its marginal impact on output quality must be assessed. You can also choose between “low latency” and “high throughput” serving modes according to business needs.
When deploying, validate the full pipeline on a mainstream open-source inference framework first, then do targeted kernel and parallelism tuning.
6. Deployment decisions and cost-saving approaches
- Size for your load; do not blindly pile on cards: use the formulas above to derive the memory floor first, then determine node count from target concurrency and context length, avoiding idle waste from “buying for the future all at once.”
- Prioritize quantization: adopting lower-precision weights, provided quality is acceptable, is the most direct way to lower the hardware threshold.
- A hybrid on-premises vs. cloud strategy: keep data-sensitive, steady-load core workloads on-premises, and use general-purpose cloud GPU capacity for elastic peak demand. Standard cloud capabilities such as GPU instances, container services, object storage, and monitoring and alerting can supplement capacity elastically, balancing cost and compliance.
- Do capacity planning up front: include long-context KV cache and concurrency growth in planning, so you do not hit the ceiling on day one.
- Watch TCO, not purchase price: over a 3–5 year cycle, electricity, operations, and depreciation account for a meaningful share. Run the full accounting when selecting hardware.
7. Summary
For on-premises deployment of very large sparse MoE models, the core question is not “can it run” but “how many machines, how much money, and can we use it well.” The first principles of cost come down to three statements: weights determine the memory floor, memory determines the machine count, and interconnect determines actual utilization. Work out the memory accounting of “weights + KV cache + runtime overhead,” pair it with a high-speed interconnect super node, then use parallelism and kernel optimization to drive utilization up, and you get a budget that holds up to scrutiny. Plug your own parameters and concurrency targets into the formulas above, and you move from “I heard it costs tens of millions” to “I know exactly how many nodes I need and where every dollar goes.”
