# Cost Breakdown for On-Premises Deployment of Trillion-Parameter MoE Models
A practical guide to estimating the true cost of deploying ultra-large Mixture-of-Experts (MoE) models locally. It frames the problem as a systems-engineering exercise rather than a GPU-count question, splitting budgets into three service levels—minimum runnable, single-user usable, and production-grade concurrency—and walks through GPU memory sizing, the outsized role of KV cache, high-speed interconnect and parallelism strategies, storage and cold-start overhead, and a three-year cost model that compares an on-premises build-out against on-demand cloud. Readers will understand where the memory floor comes from, why MoE expert residency breaks naive per-parameter math, and which levers—quantization, KV-cache management, expert placement, and batching—keep budgets under control.
Cost Breakdown for On-Premises Deployment of Trillion-Parameter MoE Models
1. Reframing the Problem: On-Premises Deployment Is Systems Engineering, Not a Single-GPU Question
When teams evaluate “how much does it cost to deploy an ultra-large-parameter Mixture-of-Experts model locally,” they usually start by asking “how much VRAM” or “how many accelerators.” But these models are not ordinary mid-sized language models. They simultaneously involve very large parameter counts, sparse expert routing, long-context caching, multi-card communication, weight loading, inference scheduling, and high-concurrency serving.
As a result, the budget cannot be estimated as “number of accelerators × unit price.” It should be broken into three levels:
- Minimum runnable — the model can load and generate, but slowly. Suitable for internal validation, offline evaluation, and algorithm research.
- Single-user usable — for one user, output speed reaches an interactive level, e.g. roughly 25 tokens/s.
- Production-grade concurrency — supports a level of concurrent users, e.g. 10 concurrent users, while each still sustains 25 tokens/s or more.
These three levels correspond to completely different hardware scales. The first cares about “can it even fit,” the second about “whether the single-request pipeline is smooth,” and the third about “throughput, memory headroom, network bandwidth, and scheduling capability.”
2. Memory Sizing: Capacity Is the First Threshold
2.1 The weights themselves set the memory floor
Take an ultra-large model with roughly 2.8 trillion parameters as an example. Using 4-bit quantization, the theoretical lower bound of the bare weight data alone is:
2.8 × 10^12 params × 4 bit / 8 = 1.4 × 10^12 bytes ≈ 1400 GB
In real deployment, you cannot count bare parameters only. Quantization formats typically carry scale factors, zero points, block structures, metadata, and padding/alignment overhead. For instance, if every group of 32 4-bit weight elements carries an 8-bit scale, the effective bit width is close to 4.25 bit:
2.8 × 10^12 × 4.25 bit / 8 ≈ 1487.5 GB
That is already near 1.5 TB, and it still excludes:
- Non-quantized layers, such as some normalization layers, embedding layers, and routing layers;
- The inference runtime workspace;
- Activation tensors;
- KV cache;
- Communication buffers;
- The scheduler, request queues, and intermediate state;
- Memory fragmentation and alignment overhead.
Therefore, the full-precision-in-memory weight footprint should reserve at least a 1.6 TB to 2 TB-class memory capacity as the base. If you want to support long contexts or multiple concurrent requests, you need to add more on top.
2.2 KV cache significantly amplifies memory pressure
For autoregressive generative models, the KV cache is the key variable that determines the cost of long contexts and high concurrency. A simplified formula:
KV cache ≈ layers × 2 × KV heads × head dim × sequence length × concurrency × bytes per element
Even if a single request’s KV cache is not huge, when concurrency rises from 1 to 10 and context length grows from 2K to 32K or 128K, total cache volume can increase by tens of times.
So for on-premises deployment of an ultra-large MoE model, the memory budget must cover not only the weights, but also “weights + KV cache + activations + communication buffers + safety margin.” A common engineering practice is:
actual memory required = weight memory × 1.2 to 1.5 + peak KV cache at the target concurrency
For a production-grade 10-concurrent, long-context scenario, reserving 2.5 TB to 4 TB of memory is generally the safer direction.
2.3 MoE expert residency means you can’t count “active parameters only”
A common misconception about Mixture-of-Experts models is: since each token only activates a small number of experts, can memory also be sized by active parameters alone?
The answer is no.
The parameter-access pattern of an MoE model can be summarized as:
- All expert weights must first reside in some addressable storage;
- After each token is routed, it is dispatched to several experts;
- After the compute node holding those experts finishes, the results are returned to the main path.
So the expert activation ratio affects compute consumption and communication volume, but weight capacity must still be sized for all experts. If you want to avoid frequently swapping expert weights in and out from external storage, you must keep the vast majority of experts resident in high-speed memory.
This means the memory floor of an on-premises MoE deployment is inherently higher than that of a dense model with equivalent active parameters.
3. Hardware Configuration: From “Can Run” to “High-Concurrency Production”
3.1 Minimum runnable: prioritize capacity aggregation
If you only want to verify that the model can load and infer, you can aggregate memory across multiple servers. The core goal here is not speed, but loading all expert weights into a unified, addressable pool of accelerated resources.
Such a configuration typically has these characteristics:
- A relatively large number of accelerators, to aggregate memory;
- Inter-card interconnect bandwidth that is not necessarily top-tier, but must be stable;
- Tolerable, slower generation speed;
- Insensitivity to latency, suitable for offline testing;
- Basic weight replication and fault-restart mechanisms.
For a 2.8-trillion-parameter, 4-bit-quantized model, a few high-memory accelerators are often not enough. You typically need dozens of accelerators forming a unified resource domain. If each card has around 80 GB of memory, 64 cards provide about 5.12 TB of memory—enough to cover weights, runtime, and a certain scale of KV cache. If per-card memory is lower, you need more cards or a tiered storage design.
3.2 Single user at 25 tokens/s: the decode pipeline is the focus
Reaching 25 tokens/s for a single user means the model can’t just “run”—every decode step must complete within an acceptable time.
You then need to watch:
- Weight read bandwidth;
- Inter-card communication latency;
- Collective-communication efficiency after expert routing;
- Whether tensor parallelism, expert parallelism, and pipeline parallelism are partitioned sensibly;
- Whether KV cache reads and writes create additional bottlenecks.
In sparse-expert models, generating each token may require accessing multiple experts. If the experts are spread across many cards, the decode phase frequently triggers cross-card or cross-node communication. Once network latency is too high, even with ample compute, output speed is dragged down by communication.
Therefore, single-user low-latency deployment is not just “adding cards”—it is about partitioning the model within a high-speed interconnected compute domain.
3.3 10 concurrent users at 25 tokens/s: production deployment needs headroom
When concurrency rises to 10, the system shifts from “single-request optimization” to “multi-request throughput optimization.” Now every request must not only hit the target speed, but do so without noticeable jitter under batching, request queuing, KV cache allocation, and uneven expert-hotspot distribution.
Production-grade configurations usually need:
- More memory headroom to absorb KV cache growth;
- Higher inter-card communication bandwidth to reduce expert-routing congestion;
- A finer-grained scheduler that supports continuous batching;
- A more sensible expert-placement strategy to avoid overloaded hot experts;
- Monitoring and rate limiting to prevent long-context requests from monopolizing resources.
To reliably support 10 concurrent users while keeping each user’s output at no less than 25 tokens/s, you typically need to add 20% to 50% compute and memory headroom on top of the minimum-runnable configuration.
4. Networking and Parallelism: The Real Challenge of MoE Deployment
4.1 Large-scale MoE is extremely sensitive to communication
In MoE inference, after each token is routed, it must be sent to the compute device holding the target expert. Once processed, the result is returned to the original path. This process is essentially a collective-communication pattern.
If the model is small, communication overhead may be negligible. But when the number of experts reaches hundreds or even thousands, each token may trigger a cross-card transfer. At that point, network bandwidth and latency directly determine inference performance.
This communication can be summarized as:
input token → routing decision → activation dispatch → expert compute → result reduction → continue generation
If “activation dispatch” and “result reduction” rely on a low-bandwidth network, compute units spend a lot of time waiting. In practice this shows up as high GPU utilization but low actual throughput—or as a single card having ample compute while overall speed still can’t scale.
4.2 A high-speed interconnect domain matters more than just piling up cards
It is commonly recommended that large-scale MoE models run within a unified high-speed interconnected compute domain. “High-speed interconnect” doesn’t just mean “connected”—it means having:
- Low latency;
- High aggregate bandwidth;
- Stable topology;
- Support for large-scale collective communication;
- Cross-node communication that is not an obvious bottleneck.
If the accelerators can only rely on ordinary networking, the model may barely run, but performance and concurrency capability will drop noticeably. For production-grade deployment, network gear, switch fabric, topology optimization, and communication-library configuration are often as important as the accelerators themselves.
4.3 Parallelism strategies must be combined
Common parallelism methods include:
- Tensor parallelism: splitting a single layer’s compute across cards;
- Expert parallelism: placing different experts on different cards;
- Pipeline parallelism: splitting different layers across devices;
- Data parallelism: multiple replicas carrying requests.
For ultra-large MoE models, a single parallelism strategy is usually not enough. A more sensible approach is:
Tensor parallelism solves single-layer compute partitioning;Expert parallelism solves expert capacity and distribution;Pipeline parallelism balances cross-device dependencies;Data parallelism carries multi-request throughput.
If the parallelism strategy is designed poorly, you get communication hotspots, uneven expert load, or a few cards that are chronically overloaded.
5. Storage and Startup: Weight Loading Is Also a Cost Item
5.1 Weight file size drives storage design
A 2.8-trillion-parameter, 4-bit-quantized model can have weight files reaching 1.5 TB or more. Add multiple replicas, version management, logs, and checkpoints, and the actual storage demand rises further.
So on-premises deployment can’t consider compute nodes alone. It must also consider:
- High-speed local solid-state storage;
- Parallel file systems;
- Object-storage archiving;
- Model version management;
- Sharded weight loading;
- Fault recovery and integrity-check mechanisms.
If every restart requires reloading several TB of weights from slow storage, both operating cost and startup latency rise significantly.
5.2 Cold start and elastic scaling
The cold-start time of an ultra-large model is often much longer than that of small-to-mid models. In production, if the service restarts frequently or needs to scale out quickly, you must consider weight pre-warming, sharded loading, cache reuse, and replica management.
A common approach is:
persisted weight replicas + compute-node pre-warming + health checks + automatic bring-up
This lets you restore service quickly when some nodes fail, rather than waiting for all weights to reload.
6. Cost Estimation: How Much Does It Actually Cost?
The following estimates are illustrative. Actual costs vary with procurement channels, facility conditions, electricity prices, network architecture, operational maturity, and utilization.
6.1 Initial hardware cost
For a 2.8-trillion-parameter, 4-bit-quantized scenario needing 64+ high-speed accelerators, the cost mainly includes:
| Item | Notes | Illustrative budget (RMB) |
|---|---|---|
| Accelerators | 64 high-bandwidth-memory accelerators | ¥6.4M – ¥12.8M |
| Servers & system integration | Power, motherboards, backplanes, cooling, assembly | ¥3M – ¥8M |
| High-speed network | Switches, optical modules, cabling, topology tuning | ¥1M – ¥4M |
| Storage system | High-speed weight storage, backup, object storage | ¥0.5M – ¥3M |
| Facility retrofit | Power, cooling, racks, wiring | ¥1M – ¥5M |
Under a more aggressive high-performance configuration, the total initial budget can reach ¥12M – ¥30M. If you lower per-card specs and add more cards, or reduce the scale of high-speed interconnect, the initial cost may drop—but system complexity and the difficulty of communication optimization will rise.
6.2 Operating cost
On-premises deployment is not a one-time investment; it also includes ongoing operating expenses:
- Electricity;
- Cooling;
- Network and storage operations;
- Model upgrades and rollbacks;
- Security hardening;
- Monitoring and alerting;
- Dedicated or part-time operations staff;
- Spare parts for failures.
Taking 64 accelerators as an example, at roughly 500 W per card, the accelerators alone draw about 32 kW. Adding servers, network, storage, and cooling, total power draw can reach 50 kW to 100 kW. Monthly electricity cost varies significantly with regional power prices and facility efficiency.
So the on-paper cost of on-premises deployment is not just the server purchase price—it also depends on power, operations, and utilization over a three-year lifecycle. If model usage is very low, the unit cost of a heavy-asset on-premises deployment can exceed renting general-purpose compute on demand.
6.3 Comparison with on-demand cloud usage
For validation-oriented teams, the safer path is usually not to buy heavy assets immediately, but to first do small-scale validation with general-purpose cloud acceleration. The cloud approach offers:
- Low initial investment;
- The ability to trial by the hour or over short periods;
- Easy adjustment of hardware specs;
- Suitability for load testing and architecture validation;
- No need to carry long-term facility operations.
Only when the model is stable, business requests are sustained, and per-inference cost can be amortized does on-premises deployment start to show a long-term cost advantage.
7. Cost Optimization: Making the Budget More Controllable
7.1 Quantize sensibly, not blindly to the extreme
4-bit quantization significantly reduces weight size, but different layers have different sensitivities. Production deployments usually adopt a mixed strategy:
- Use low-bit quantization for most expert weights;
- Keep higher precision for critical layers, embedding layers, and routing layers;
- Optimize the KV cache separately;
- Calibrate activation distributions to avoid outliers degrading accuracy.
Quantization is not about being as aggressive as possible—it is about balancing capacity, accuracy, and stability.
7.2 Optimize KV cache management
In long-context and high-concurrency scenarios, the KV cache is a major source of memory consumption. You can apply:
- Paged cache management;
- Request-level cache reuse;
- Sliding window or context compression;
- Low-precision caching;
- Dynamically releasing idle request caches;
- Capping the concurrency of overly long requests.
These methods raise effective concurrency without significantly hurting experience.
7.3 Expert placement and hotspot governance
Expert load in MoE models isn’t always even. Some experts may be called frequently, forming hotspots. If hot experts concentrate on a few cards, those cards become heavily queued.
Strategies include:
- Replicating hot experts;
- Rebalancing expert shards;
- Routing constraints or capacity limits;
- Assigning cold experts to lower-priority resources;
- Monitoring expert call frequency and dynamically adjusting placement.
7.4 Batching and scheduling optimization
Production inference can’t process requests one at a time serially. Continuous batching can significantly raise throughput, but it also increases memory peaks and scheduling complexity. Tune against your business latency targets:
- Low-latency scenarios: smaller batches, prioritize per-request speed;
- High-throughput scenarios: larger batches, raise overall utilization;
- Mixed scenarios: separate queues for different request priorities.
8. Conclusion: Set the Budget by Service Level
The cost of deploying an ultra-large-parameter MoE model on-premises is not a fixed number. It is jointly determined by business goals, model size, quantization method, context length, concurrency, and operational capability.
If the goal is minimum runnable, the budget focuses on memory aggregation and weight loading, and slower speed is acceptable. If the goal is single-user at 25 tokens/s, the focus shifts to communication latency, the decode pipeline, and parallelism partitioning. If the goal is 10 concurrent while each user still sustains 25 tokens/s, then throughput headroom, network bandwidth, KV cache management, and scheduling capability must all be considered.
Overall, for trillion-parameter-class MoE models, a complete on-premises deployment typically requires a multi-card high-speed interconnected compute domain, with initial investment often in the tens-of-millions (RMB) range. For most teams, the more prudent path is:
- First validate model capability and performance bottlenecks using general-purpose cloud compute;
- Then load-test memory, network, and concurrency metrics on a small-scale cluster;
- Confirm business utilization and long-term cost before deciding whether to build heavy on-premises assets;
- Finally, keep costs within an acceptable range through quantization, cache management, expert scheduling, and parallelism optimization.
