Local Deployment Costs and Strategies for Ultra-Large MoE Models
Deploying trillion-parameter Mixture of Experts (MoE) models locally requires careful planning beyond simply fitting weights into a GPU. This article breaks down the hardware and software requirements for ultra-large MoE model deployment across three tiers: basic validation, single-user high-speed inference, and multi-user concurrent production. We explore weight capacity estimation, the impact of sparse activation on memory, KV Cache bottlenecks, and parallelism strategies, providing a clear framework for calculating your infrastructure budget and deployment architecture.
I. First, Clarify: What Are You Actually Deploying?
For models with total parameters reaching the trillion level that utilize a Mixture of Experts (MoE) architecture, local deployment cannot be simplified to “downloading weights onto a GPU.” These models typically contain a massive number of expert networks. Although only a subset of experts is activated per token during inference, the complete model must still store all expert weights in accessible storage or memory.
Therefore, deployment costs are primarily determined by the following factors:
- Weight Capacity: Determines the required VRAM, system memory, or local high-speed storage.
- Active Computation: Determines the required compute power per second.
- Inter-Expert Communication: Determines the network and interconnect requirements for multi-GPU and multi-node deployments.
- KV Cache Capacity: Determines long-context and concurrency capabilities.
- Target Throughput: Determines the required number of compute devices and bandwidth.
- Software Adaptation: Determines whether the hardware can translate theoretical performance into actual speed.
This article categorizes deployment targets into three tiers:
- Tier 1: Just getting it to run, regardless of speed.
- Tier 2: Single-user generation speed reaching approximately 25 token/s.
- Tier 3: Supporting 10 concurrent users, each maintaining around 25 token/s or higher.
It should be noted that if the complete model weights, official inference implementations, and final quantization schemes are not yet public, all hardware and speed data can only serve as estimates, not guaranteed values.
II. Why Trillion-Level MoE Models Still Require Massive Hardware
Assume a model with approximately 2.8 trillion total parameters using an MoE architecture, routing each token to only a few experts. Even if each token activates only about 1% to 2% of the experts, you still cannot store only those activated experts during deployment.
The reason is that routing results change based on input content. The current token might call Expert A and Expert B, while the next token might call Expert C and Expert D. To ensure the model’s complete capabilities, the system must be able to access all expert weights.
1. Weight Capacity Estimation
If using 4-bit weights, the required capacity under ideal conditions is approximately:
2.8 trillion × 4 bit ÷ 8≈ 1.4 TB
However, the low-bit formats used in practice usually include scaling factors, grouping information, metadata, alignment space, and some non-quantized tensors. Therefore, the actual weight capacity may approach 1.5 TB or even higher.
Additionally, extra space is required at runtime for:
- Model metadata;
- Routing tables;
- Temporary activations;
- Communication buffers;
- KV Cache;
- Inference framework workspace;
- Operating system and service process overhead.
Therefore, you cannot simply purchase memory based on the “1.5 TB weights equals 1.5 TB memory” logic. A more robust approach is to reserve at least 20% to 40% additional runtime space.
2. Fewer Active Parameters Does Not Mean Lower VRAM Demand
MoE’s sparse activation primarily reduces the computation required per token, but it does not automatically reduce the storage requirement for the complete weights.
This can be simply understood as:
- Total Parameters: Determine “how much data the model must store.”
- Active Parameters: Determine “how much computation each token requires.”
- Expert Distribution: Determines “how much data must be transferred between multiple GPUs.”
This is why ultra-large MoE models, even if their theoretical computation does not equal that of dense models of similar size, still require large memory systems or multi-device clusters.
III. Tier 1: Just Run It, Speed Can Be Slow
1. Recommended Architecture: CPU Memory Hosts Experts, GPU Handles Shared Computation
If the goal is merely to verify whether the model can load and complete simple Q&A, the most realistic approach is heterogeneous inference:
- GPU stores shared attention modules and some high-frequency compute modules.
- CPU memory stores the majority of expert weights.
- Local high-speed storage is used for weight loading and caching.
- During inference, corresponding experts are read based on routing results.
The core advantage of this approach is relatively controllable cost; the disadvantage is typically slow speed. Because for every token generated, the following process may occur:
- The GPU completes preliminary computation.
- The router selects target experts.
- Corresponding expert weights or activation data are read from CPU memory.
- The CPU or GPU completes the expert computation.
- Results are transferred back to the main compute device.
- Subsequent layer computation continues.
As long as CPU memory capacity is sufficient, the model can run; however, if memory bandwidth, PCIe lanes, or inter-device communication speeds are insufficient, the generation speed may only be a few tokens per second, or even lower.
2. Suggested Hardware Configuration
An experimental-grade system might consider:
- 1 to 2 compute cards with large VRAM.
- 1 TB or more system memory (1.5 TB to 2 TB highly recommended).
- Multiple high-speed NVMe SSDs configured as a large local cache.
- Multi-socket processor platform.
- Sufficient PCIe lanes.
- Stable power supply and cooling system.
The key here is not simply increasing CPU core count, but ensuring:
- Sufficient memory capacity.
- Sufficient memory channel bandwidth.
- Data transfer between GPU and CPU does not become an absolute bottleneck.
- Storage can withstand sustained large-scale read operations.
3. Tier 1 Cost Range
Estimating based on server-grade hardware, large memory, large VRAM, and high-speed storage:
- Compute cards: Several thousand to tens of thousands of USD.
- Large-capacity memory: Several thousand to tens of thousands of USD.
- High-speed storage: A few hundred to several thousand USD.
- Motherboard, power supply, chassis, and cooling: Several thousand to tens of thousands of USD.
- Software adaptation and debugging: Depends on team capability.
The total cost may range from $15,000 to $45,000 USD. If purchasing a complete pre-built server, costs may be higher.
Note that the “minimum cost” tier is about “being able to run,” not “being suitable for production.” It is best suited for model compatibility verification, quantization testing, and functional experiments.
IV. Tier 2: Single-User Reaching ~25 token/s
If a single user needs to continuously achieve around 25 token/s, the problem shifts from “can it load” to “is the compute and communication fast enough.”
1. You Cannot Look Only at VRAM Capacity
Even if a single GPU’s VRAM can accommodate a portion of the weights, it does not mean the model will reach the target speed. For sparse expert models, performance is typically affected by the following factors combined:
- Attention module compute efficiency.
- Expert weight loading speed.
- Expert routing efficiency.
- Multi-GPU communication latency.
- Quantization operator implementation quality.
- KV Cache read/write efficiency.
- Input and output lengths.
- Whether continuous batching is enabled.
If experts are distributed across multiple GPUs, every token requires transmitting activation data between devices. Interconnect bandwidth directly impacts generation speed. Low-speed networks can easily leave compute cards waiting.
2. Recommended Architecture: Multi-GPU High-Speed Interconnect
To achieve high single-user speed, you usually need:
- Multiple large-VRAM compute cards.
- High-speed inter-GPU interconnect.
- High-bandwidth host memory.
- Inference engine adapted for low-bit matrix operations.
- Expert parallelism and tensor parallelism strategies.
- Minimizing cross-node communication.
If all experts are placed within the same high-speed interconnect domain, post-routing activation transfer will be faster. If experts span multiple nodes, you must rely on high-bandwidth, low-latency networks; otherwise, communication time may exceed expert computation time.
3. Rough Hardware Scale
For ~1.5 TB of low-bit weights, if you want a single user to reach ~25 token/s, you cannot calculate purely based on “weight capacity.” You must also reserve headroom for runtime space and compute throughput.
From an engineering perspective, evaluate based on the following:
- 8 to 16 large-VRAM compute cards.
- Total VRAM of at least 1.5 TB or more.
- Ideal total VRAM is 1.8 TB to 2.5 TB.
- High-speed inter-GPU interconnect.
- Ample host memory for overflow and caching.
- Unified multi-GPU scheduling and monitoring system.
The hardware cost for such a solution may reach hundreds of thousands of USD, depending on compute card specifications, VRAM size, interconnect methods, and whether a complete cluster is used.
4. Speed Tests Must Have Standardized Conditions
“25 token/s” cannot be discussed outside of testing conditions. At minimum, you must specify:
- Input prompt length.
- Output length.
- Whether first-token latency is counted.
- Whether sampling time is included.
- Whether quantization is enabled.
- Whether a single request is used.
- Whether caching is enabled.
- Whether it is average speed or minimum speed.
- Whether it is generation speed or end-to-end response speed.
For example, the average generation speed under short input and short output conditions is usually significantly higher than in long-context scenarios. Actual service must also account for queueing, network transmission, request scheduling, and result return times.
V. Tier 3: 10 Concurrent Users, Each at ~25 token/s
This is the most difficult of the three targets. 10 concurrent users at 25 token/s each means the system needs to maintain a sustained generation throughput of about 250 token/s, while also handling fluctuations caused by varying request lengths.
1. Concurrency Capacity is Not Simple Multiplication
Theoretically, a single user at 25 token/s means 10 users is 250 token/s. However, actual systems are typically affected by the following factors:
- Different context lengths for different requests.
- KV Cache usage grows with sequence length.
- Expert routing experiences dynamic load imbalance.
- Certain experts may become hotspots.
- Batching increases scheduling complexity.
- Long requests slow down short requests.
- Communication overhead increases with concurrency.
Therefore, you cannot simply purchase a system that achieves 25 token/s for a single user and expect it to naturally support 10 concurrent users.
2. Focus on Building KV Cache
During concurrent inference, KV Cache often becomes a VRAM bottleneck faster than weights. Assuming each request has a long context and 10 requests are generating simultaneously, the cache space will grow rapidly.
System design should consider:
- Whether KV Cache uses low-precision storage.
- Whether it supports paged management.
- Whether it can offload part of the cache to host memory.
- Whether dynamic cache eviction is allowed.
- Whether there is a maximum context limit per request.
- Whether prefix caching is used to reduce redundant computation.
If you only focus on weight capacity and ignore KV Cache, the model may perform normally in single-user tests but experience OOM (out of memory) or sudden speed drops once concurrency increases.
3. Recommended Cluster Scale
To achieve 10 concurrent users at ~25 token/s each, you typically need:
- A larger multi-GPU cluster.
- Higher total VRAM.
- Higher inter-GPU and inter-node bandwidth.
- An inference service supporting continuous batching.
- A combination of expert parallelism, tensor parallelism, and pipeline parallelism.
- Comprehensive rate limiting, queueing, and failover mechanisms.
Rough estimates suggest needing 16 to 64 high-end compute cards, with total costs ranging from hundreds of thousands to millions of USD. If standard Ethernet is used to connect multiple nodes, actual throughput may fall significantly below theoretical values. If dedicated high-speed interconnects are used, equipment and network costs will rise further.
Deployment at this level is closer to a small compute cluster rather than a standard workstation.
VI. Key Aspects of Software Deployment
1. Weight Format Conversion
You need to verify whether the model supports the target quantization format and whether the inference engine has the corresponding operators. Common issues include:
- Weight files can be read, but lack high-performance operators.
- Some tensors are still stored at higher precision.
- Different devices have inconsistent support for low-bit formats.
- Group size mismatches cause additional conversions.
- Unstable accuracy and speed post-quantization.
Therefore, do not just look at file size. You should simultaneously check weight formats, operator implementations, and device compatibility.
2. Choosing Parallelism Strategies
Common parallelism methods include:
- Tensor Parallelism: Splitting single-layer computation across multiple GPUs.
- Expert Parallelism: Distributing different experts across different devices.
- Pipeline Parallelism: Placing different layers on different devices.
- Data Parallelism: Replicating compute paths for multiple requests.
- Hybrid Parallelism: Combining the above methods.
For ultra-large MoE models, expert parallelism is usually the core strategy, but it places high demands on communication. Higher parallelism does not necessarily mean faster speeds; testing must be conducted based on communication bandwidth, expert load, and batch size.
3. Service Layer Capabilities
Production environments also require:
- Request queues.
- Concurrency limits.
- Timeout controls.
- Streaming output.
- Task cancellation.
- Access authentication.
- Logging.
- Metric monitoring.
- VRAM and cache eviction.
- Node fault handling.
If it’s just a local experiment, these components can be simplified. If it’s for multi-user access, they must be factored into the overall cost.
VII. How to Calculate Budget More Accurately
It is recommended to use the following formula for a preliminary estimate:
Total Cost =Compute Device Cost+ Host & Memory Cost+ High-Speed Storage Cost+ Interconnect Network Cost+ Rack, Power & Cooling Cost+ Software Adaptation Cost+ Operations & Power Cost
Simultaneously, establish three budget tiers:
Budget A: Validation Deployment
The goal is to confirm the model can load and generate text. The budget is roughly tens of thousands of USD. Speed may be low, suitable for R&D verification.
Budget B: Single-User Experience Deployment
The goal is to achieve ~25 token/s for a single user. The budget is roughly hundreds of thousands of USD, requiring strong multi-GPU interconnect and specialized software optimization.
Budget C: Production Service Deployment
The goal is to support 10 concurrent users with stable throughput. The budget may reach hundreds of thousands to millions of USD, with additional investments in networking, monitoring, disaster recovery, power, and operations.
VIII. Conclusion: Define Goals Before Deciding on Hardware
The most common mistake in local deployment of ultra-large MoE models is focusing solely on parameter count or VRAM capacity. The correct evaluation order should be:
- Define context length and concurrency scale.
- Determine target generation speed.
- Calculate the actual capacity of low-bit weights.
- Reserve runtime space and KV Cache.
- Evaluate the communication volume brought by expert routing.
- Choose between CPU heterogeneous, multi-GPU single-node, or multi-node cluster.
- Calibrate costs through real benchmark testing.
If you just want to verify if the model can run, you can use a heterogeneous approach with a large-memory server and a few compute cards. If you require high single-user speed, you need multi-GPU high-speed interconnects. If you also need to support 10 concurrent users, the final form will essentially be a small high-performance inference cluster.
Therefore, there is no one-size-fits-all “minimum configuration” for such models. The truly reasonable answer is: Just running it may start at tens of thousands of USD; achieving high single-user speed usually requires hundreds of thousands of USD; and for multi-user production services, it may enter the hundreds of thousands to millions of USD range.
