Deploying Extremely Large MoE Models On-Premises: Three Deployment Tiers and Cost Estimates
This guide presents a three-tier framework for deploying a roughly 2.8-trillion-parameter Mixture-of-Experts model locally. It explains how quantization, memory capacity, compute bandwidth, KV cache, and expert-parallel communication affect feasibility and cost, from basic CPU–GPU offloading to a multi-accelerator supernode capable of serving 10 concurrent users at 25 tokens per second each. All performance and hardware figures are estimation frameworks that must be validated after the model weights and benchmarks are released.
Deploying Extremely Large MoE Models On-Premises: Three Deployment Tiers and Cost Estimates
The most common mistake when deploying an extremely large Mixture-of-Experts (MoE) model locally is to treat the problem as simply buying several accelerator cards. In practice, feasibility depends on three factors:
- Whether the weights can fit into addressable memory and storage
- Whether per-token compute and memory bandwidth can support the target speed
- Whether all-to-all communication under concurrency consumes the available compute capacity
Using a publicly described MoE model with roughly 2.8 trillion parameters, 896 experts, and 16 experts activated per token as an example, this article breaks deployment into three target tiers. The hardware and performance figures are estimation frameworks; every figure must be verified against the actual weights and benchmark results once they are available.
1. Start With the Baseline: Parameters, Quantization, and Minimum Storage
The first principle of MoE deployment is:
Memory usage is determined by total parameters, while compute consumption is determined by the activated experts.
Each token is routed to only 16 of the model’s 896 experts, giving an expert activation ratio of approximately:
16 / 896 ≈ 1.79%
However, this is not the same as the exact ratio of active parameters. More importantly, it does not mean that only 16 experts need to be loaded. During inference, all 896 experts must remain in addressable storage, after which the router selects and loads the experts required for each token.
Quantization determines the lower bound for storage. This model reportedly introduces quantization-aware training during supervised fine-tuning and uses 4-bit microscaled weights with 8-bit activations. For 2.8 trillion parameters, the theoretical raw-data requirement for 4-bit weights is:
2.8e12 × 4 bits ÷ 8 ≈ 1,400 GB
A microscaling format is not pure 4-bit storage. Every 32 4-bit values also carries an 8-bit scale. Based on common microscaling-format assumptions, the effective storage requirement is approximately 4.25 bits per weight:
2.8e12 × 4.25 bits ÷ 8 ≈ 1,487.5 GB
This excludes non-4-bit tensors, metadata, alignment padding, runtime workspace, and KV cache. The actual memory required for a usable deployment is therefore usually well above 1,487.5 GB.
As a point of comparison, a roughly 671-billion-parameter MoE model may require about 685 GB for its main parameters and multi-token prediction parameters in FP8. After 4-bit weight quantization, the weights may occupy about 335 GB, with approximately 37 GB for activations. The total is around 400 GB, enough to run on eight 80 GB accelerators. A 2.8-trillion-parameter model is more than four times larger, which establishes the lower bound for a full deployment.
KV cache must also be budgeted separately. A general estimate is:
KV ≈ 2 × number of layers × hidden dimension × bytes per element × sequence length × concurrency
A single user with a long context and ten concurrent users with long contexts represent completely different memory requirements. The deployment budget should therefore include:
weights + runtime workspace + KV cache + fragmentation
rather than focusing only on the weight files.
2. Three Targets, Three Architectures
Tier 1: Make It Run, Regardless of Speed
The goal at this tier is to place nearly 1,500 GB of weights into addressable storage and generate tokens reliably, even if throughput is only in the single-digit range.
The most direct approach is CPU–GPU heterogeneous offloading:
- Keep the compute-intensive shared attention components on the GPU.
- Place the 896 sparse experts, which account for most of the storage requirement, in large CPU memory.
- Use the CPU system to feed the GPU with the 16 experts selected for each token.
Public tutorials have demonstrated a dual-socket CPU server with a single consumer accelerator in the 24 GB class running a 671-billion-parameter 4-bit model. Under a restricted expert-selection strategy, such systems can reportedly achieve more than 200 tokens per second during prefill and approximately 14 tokens per second during decode.
For a 2.8-trillion-parameter model, the storage lower bound is approximately 1.5 TB. System memory should be in the 2 TB-plus range, with particular attention to NUMA layout, the number of memory channels, and PCIe bandwidth. The number of accelerators does not need to be large at this tier, but CPU memory capacity and bandwidth become the primary bottlenecks.
Cost is dominated by the large-memory server. A small number of accelerators, storage, power, and cooling make up the remaining cost. Initial hardware investment may range from tens of thousands to several hundred thousand in local currency, depending on memory capacity and accelerator selection. Stable long-context service and multi-user queuing also require additional workspace, monitoring, and alerting resources.
Tier 2: 25 Tokens per Second for a Single User
A target of 25 tokens per second means that every decode step must complete routing, expert-weight reads, attention computation, and communication. The bottleneck shifts from capacity to bandwidth and compute.
Simply placing the experts in CPU memory is unlikely to sustain 25 tokens per second because of system-memory bandwidth and PCIe limitations. More accelerators are needed to host popular experts or shard all experts, while a high-speed interconnect domain is required to reduce cross-device data movement.
The quantization level should be no lower than 4-bit microscaled weights with 8-bit activations. If lower precision is used to improve speed, quality must be validated on representative workloads.
A suitable baseline for this tier is:
- Multiple accelerators
- Large system memory
- High-speed interconnects
Before committing to a full-scale deployment, run a small-scale benchmark with fixed input and output lengths. Measure:
- Prefill throughput
- Decode throughput
- P50 and P95 latency
- Peak accelerator memory
- Expert-routing hit distribution
At this tier, the cost structure changes from “primarily memory” to “primarily accelerators and interconnects.” One-time investment typically moves into the range of several hundred thousand to several million in local currency, depending on local hardware prices and the selected architecture.
Tier 3: Ten Concurrent Users at 25 Tokens per Second or More Each
Ten concurrent users at 25 tokens per second require an aggregate throughput of:
10 × 25 tokens/s = 250 tokens/s
The system must simultaneously handle ten KV caches, ten routing streams, and substantial all-to-all communication.
When 896 experts are distributed across multiple accelerators, the activation values generated by routing must be transferred to the accelerators hosting the target experts. After computation, the results must be transferred back. Communication latency can directly consume the available compute capacity.
Public recommendations for this class of deployment include using a supernode with at least 64 accelerators and a high-speed interconnect that keeps the communication domain large. The goal is to keep expert movement within a single communication domain whenever possible.
In production planning, counting accelerators is not enough. The system must satisfy all of the following:
- Total storage must cover the weights, runtime workspace, and ten KV caches.
- Aggregate compute must support the required 250 tokens per second.
- Interconnect bandwidth and topology must support expert-parallel all-to-all traffic.
This tier typically costs several million to more than ten million in local currency. Interconnects, switches, power delivery, cooling, and operations become a much larger share of the total cost.
3. Cost Model: Do Not Count Only the Accelerators
A basic budget can be expressed with the following pseudocode:
Total cost = CAPEX + OPEXCAPEX = number of accelerators × unit price+ memory capacity × unit price+ storage capacity × unit price+ high-speed interconnects/switches × unit price+ power and cooling+ rack modificationsOPEX = system power × electricity price × operating time+ operations labor+ spare parts+ software platform costs
The differences between the three tiers are not determined by a single unit price, but by the cost structure:
- Tier 1: Sensitive primarily to memory capacity
- Tier 2: Sensitive primarily to accelerator count and interconnects
- Tier 3: Sensitive primarily to switch topology, cluster scheduling, and failure recovery
Cloud services can be used as elastic supplements. Relevant general-purpose capabilities may include high-performance computing instances, object storage, container platforms, monitoring, and alerting. These can support peak capacity, weight distribution, and observability. However, the core capacity requirement must still be calculated using the formulas above.
4. Validation Checklist After the Weights Are Released
- Storage validation: Measure the actual weight-file size, non-4-bit tensors, metadata, and total size after alignment. Record peak GPU and system-memory usage during loading.
- Throughput validation: Benchmark prefill and decode separately. Record input length, output length, batch size, and P50/P95 latency.
- Concurrency validation: Under ten-way concurrency, measure aggregate throughput, per-request tokens per second, KV-cache growth, and queueing latency.
- Communication validation: Measure the percentage of time spent on expert-parallel all-to-all communication, comparing intra-domain and cross-machine traffic.
- Quality validation: Compare quantized and unquantized accuracy on business task sets, long-context workloads, code tasks, and mathematics tasks.
- Stability validation: Test continuous operation, failure recovery, monitoring, alerting, and capacity thresholds.
Conclusion
The cost of deploying an extremely large MoE model locally is determined jointly by the total-parameter storage floor, target throughput, and communication requirements under concurrency.
- Tier 1 can use CPU–GPU heterogeneous offloading, trading speed for large system memory.
- Tier 2 requires multiple accelerators and high-speed interconnects to reach 25 tokens per second during decode.
- Tier 3 must be designed as a supernode; otherwise, communication will become the bottleneck before compute capacity does under ten-way concurrency.
The practical approach is to calculate the roughly 1.5 TB weight requirement and KV-cache budget first, then use benchmarks to calibrate accelerator count, memory capacity, and interconnect design. Only then can the deployment cost be estimated with confidence.
