Local Deployment of Trillion-Parameter Mixture-of-Experts Models: Memory, Hardware, and Cost Planning
This article examines the key cost drivers for locally deploying very large Mixture-of-Experts models. It estimates VRAM requirements for a roughly 2.8-trillion-parameter model under 4-bit quantization, explains why a low activated-parameter count does not remove the need to store all expert weights or provide high-speed interconnects, and compares three deployment targets: runnable-only, single-user 25 token/s, and 10 concurrent users at 25 token/s. It also covers KV cache planning, software-level optimizations, and a validation-first procurement process.
Local Deployment of Trillion-Parameter Mixture-of-Experts Models: Memory, Hardware, and Cost Planning
1. First, Clarify What Determines Local Deployment Cost
When discussing local deployment of very large models, it is not enough to ask how many parameters the model has. Deployment cost also depends on model format, concurrency, target generation speed, context length, hardware interconnect, and service stability.
Deployment cost can be divided into six parts:
- Model weight storage cost.
- VRAM usage during inference.
- Number of accelerator cards and VRAM per card.
- Card-to-card interconnect and network switching equipment.
- Local storage, hosts, and rack infrastructure.
- Software deployment, monitoring, failover, power, and cooling.
For ordinary small models, a single multi-GPU server is often sufficient. For trillion-parameter Mixture-of-Experts models, however, it is not safe to purchase hardware simply according to the number of activated parameters, even if each input token activates only a small number of experts. During inference, the complete expert set must remain accessible, and the weights must still be distributed across VRAM or high-speed memory. The activated-parameter ratio mainly affects compute volume; it does not directly determine weight capacity.
2. How Much VRAM Does a Trillion-Parameter Mixture-of-Experts Model Need?
Assume the target model has about 2.8 trillion total parameters, uses a Mixture-of-Experts architecture with hundreds of experts, and routes each token to only a few of them.
First, estimate weight capacity.
With idealized 4-bit weights:
2.8 trillion × 4 bit ÷ 8≈ 1.4 TB
This is only a theoretical lower bound. Practical quantization formats usually also need to store scaling factors, grouping information, tensor metadata, and memory alignment space. For a format that stores additional scaling information per group of weights, actual weight capacity may approach 1.5 TB or even more.
In addition, the following VRAM overheads must be considered:
- Non-quantized layers, such as embedding layers, output layers, and some normalization layers.
- Router and expert-dispatch related parameters.
- Activations.
- Temporary compute buffers.
- Communication buffers.
- KV cache.
- Runtime overhead of the inference framework.
Therefore, “1.4 TB of weights” does not mean “1.4 TB of VRAM is enough to run the model.” To leave a reasonable margin, actual VRAM requirements should usually be planned at 1.6 TB to 2 TB or higher.
3. Why a Low Activated-Parameter Count Still Requires Substantial Hardware
The defining characteristic of a Mixture-of-Experts model is that each token passes through only part of the experts. For example, a model may contain hundreds of experts but activate only a dozen or so per token. This can significantly reduce per-token compute, but it does not automatically eliminate weight storage requirements.
Inference roughly works as follows:
- Input tokens enter the shared trunk network.
- The router selects target experts for each token.
- Token representations are sent to the corresponding accelerator cards through high-speed interconnects.
- The target experts perform computation.
- Results return to the trunk network for further processing.
If experts are distributed across multiple machines or multiple accelerator cards, a large amount of cross-card data exchange occurs. At that point, the system bottleneck may no longer be matrix multiplication, but card-to-card communication, routing scheduling, and data movement.
As a result, large Mixture-of-Experts models are highly sensitive to the following:
- Bidirectional card-to-card bandwidth.
- Communication latency.
- Topology.
- Collective communication efficiency.
- Whether routing load is balanced.
- Whether intra-server interconnect is fast enough.
Connecting multiple accelerator cards with ordinary server networking usually makes it difficult to achieve ideal performance. Even if total VRAM is sufficient, communication latency can sharply reduce generation speed.
4. Cost Analysis for Three Deployment Targets
The following estimates are organized around three common targets. Because hardware prices, model implementations, and quantization methods vary significantly, the amounts should be treated only as budget ranges, not as substitutes for actual benchmarking.
Target 1: The Model Only Needs to Run, With No Focus on Speed
This is the minimum requirement. The main goal is for the model to load and return results.
In theory, a large number of high-VRAM accelerator cards can be used to split the complete quantized weights across multiple cards. If using devices with about 80 GB VRAM per card and calculating based on more than 1.6 TB of usable VRAM, at least around 20 cards are needed. Considering runtime margin, KV cache, and load imbalance, a practical configuration is closer to 24 to 32 cards.
For a single-server multi-card approach, the host must also provide:
- Enough PCIe lanes.
- Large system memory.
- Multiple power supplies.
- High-speed local solid-state storage.
- Good cooling and airflow.
- A software environment that supports large-scale tensor partitioning.
In terms of budget, accelerator cards alone may cost several million RMB. After including the full server, storage, power, and data center infrastructure, total investment may reach several million RMB or more. With rental, costs depend on the number of GPUs, rental duration, storage, and network traffic. Short-term testing is cheaper than building a system, but long-term operation is not necessarily more economical.
The practical experience at this tier usually involves long first-token latency and low continuous output speed. It is suitable for functional validation, offline experiments, and model compatibility testing, but not for user-facing interactive services.
Target 2: Single-User Output Speed of 25 token/s
Reaching 25 token/s for a single user is no longer just about “being able to load” the model. It requires joint optimization of compute throughput and communication efficiency.
At minimum, consider the following configuration:
- 32 or more high-VRAM accelerator cards.
- Total VRAM of no less than 2 TB.
- High-bandwidth card-to-card interconnect.
- Dedicated high-speed switching equipment.
- Enough memory and local high-speed storage per server.
- An inference engine optimized for the quantization format.
- Reasonable context length and KV cache capacity settings.
If the model uses higher-precision weights, VRAM requirements increase further. For example, the theoretical capacity of 8-bit weights is roughly twice that of 4-bit weights, and retaining higher precision for some layers adds additional overhead.
For 25 token/s, two concepts must also be distinguished:
- First-token latency: how long after a user sends a request before the first output appears.
- Decode speed: how many tokens can be generated per second after the first token.
Both matter for interactive experience. Improving decode speed alone does not fully solve long first-token waits. Long-context input increases computation during the prefill stage, so benchmarks should record short-input, long-input, short-output, and long-output scenarios separately.
In terms of cost, this tier usually requires hardware investment in the several-million-RMB range. If redundant nodes, backup power, and continuous monitoring are also required, the budget rises further.
Target 3: 10 Concurrent Users, Each Sustaining More Than 25 token/s
This is a production-service target and is much more difficult than a single-user test.
Ten concurrent requests do not simply mean scaling a single-user configuration by 10×. A real system can improve overall utilization through continuous batching, request scheduling, and KV cache reuse, but growing concurrency still creates the following pressures:
- VRAM usage continues to grow during the decode phase.
- KV cache usage increases with context length.
- Routing communication traffic rises noticeably.
- Different request lengths create load imbalance.
- Long requests may slow down short requests.
- Service quality requires timeout and rate-limiting policies.
If every user must reliably receive 25 token/s, total system throughput should be at least close to 250 token/s, while also leaving margin for scheduling, communication, and failures. Ideally, multiple inference nodes should be provisioned rather than stacking all accelerator cards into one logical instance.
A more reasonable production architecture is:
Request ingress↓Authentication and rate limiting↓Request scheduler↓Multiple model inference replicas↓Monitoring, logging, and failover
Each inference replica contains a group of high-speed interconnected accelerator cards, and the scheduler distributes requests across replicas. This prevents a single replica failure from taking down the entire service and makes capacity expansion easier.
Hardware investment at this tier may range from several million RMB to over ten million RMB, depending on accelerator card specifications, node count, network equipment, data center conditions, and redundancy level. If around-the-clock operation is required, power and cooling costs cannot be ignored.
5. Do Not Overlook KV Cache and Context Length
Many deployment estimates calculate only model weights and ignore KV cache. For long-context and high-concurrency services, KV cache can become the VRAM bottleneck.
At a high level, KV cache size depends on:
- Number of layers.
- Number of attention heads.
- Context length per request.
- KV precision.
- Number of concurrent requests.
- Whether paged cache management is used.
For example, with the same 10 concurrent requests, if each request contains only a few thousand tokens, VRAM pressure may be acceptable. If each request contains hundreds of thousands of tokens, cache requirements rise rapidly. Before deployment, maximum context length and average context length must be defined, and capacity planning must be based on actual request distribution.
Production systems should also set:
- Maximum input length per request.
- Maximum output length per request.
- Maximum concurrency.
- Maximum cache usage.
- Timeout and cancellation mechanisms.
- Isolation policies for long and short requests.
6. Key Software-Level Optimizations
The number of hardware units is not the only determinant of performance. Software configuration is equally important.
The first consideration is quantization format. 4-bit weights can significantly reduce VRAM requirements, but different quantization schemes differ clearly in accuracy, decode speed, and hardware compatibility. Prefer formats that match the model training method, accelerator card instruction set, and inference framework.
The second consideration is batching strategy. Static batching is suitable for offline tasks with similar request lengths. Dynamic batching and continuous batching are better for online services because they can adjust batches dynamically as requests arrive and complete.
The third consideration is expert parallelism. Model experts should be distributed as evenly as possible to avoid hot spots on a small number of accelerator cards. If token traffic generated by the router remains concentrated on certain experts, local congestion occurs. Even when overall VRAM and compute resources are sufficient, actual speed can still decline.
Finally, monitoring is essential. At minimum, monitor:
- First-token latency.
- Output tokens per second.
- Input processing speed.
- GPU utilization.
- VRAM usage.
- KV cache usage.
- Card-to-card communication bandwidth.
- Expert load distribution.
- Request queue time.
- Error rate and timeout rate.
7. Recommended Procurement and Validation Process
Do not purchase the entire equipment set first and then attempt deployment. A safer process is:
- Confirm the model’s final weight format, context length, and officially recommended deployment method.
- Use a small number of accelerator cards for compatibility validation, focusing on whether the model loads correctly, whether quantized weights match, and whether parallel strategies work properly.
- Establish a small-scale performance baseline, testing single-request, short-context, long-context, and multi-concurrency scenarios separately.
- Based on measured weight usage, cache usage, and communication bandwidth, estimate the full cluster scale.
- Add at least 20% to 30% capacity headroom for version upgrades, traffic spikes, and node failures.
- Then decide whether to build a data center, rent compute capacity, or use a hybrid deployment model.
8. Conclusion
For a Mixture-of-Experts model with about 2.8 trillion parameters, the theoretical capacity of 4-bit weights is about 1.4 TB, but practical deployment should usually be planned around 1.6 TB to 2 TB or more of VRAM. A low number of activated experts reduces compute volume only. It cannot replace storage for the complete weights, and it cannot eliminate the need for high-speed communication among experts.
If the goal is only to validate whether the model can run, the budget is roughly in the several-million-RMB range. If the requirement is a stable 25 token/s for a single user, a larger high-bandwidth interconnected cluster is needed. If 10 concurrent users must also maintain the same speed, the system should be planned as a production-grade multi-replica architecture, with total investment potentially reaching several million to over ten million RMB.
Ultimately, cost cannot be judged only by accelerator card prices. VRAM headroom, network interconnect, KV cache, storage, power, cooling, software optimization, and operations must all be included. The most reliable approach is to complete small-scale benchmarking first, then scale capacity based on real throughput, latency, and concurrency data rather than paper estimates based only on parameter count or activation ratio.
