Local Deployment of Trillion-Parameter MoE Models: Hardware Requirements and Cost Analysis
This article examines the local deployment requirements for a representative 2.8-trillion-parameter MoE model with roughly 104 billion activated parameters per token. It breaks down VRAM demand, explains why about 1.5TB of GPU memory is the practical baseline, compares three deployment tiers, and provides hardware and cost considerations for private deployments.
Large-scale mixture-of-experts (MoE) architectures have become a major focus in the evolution of large language models. Through sparse activation, these models retain the strong generalization ability of massive parameter counts while reducing the compute required for each inference pass. For developers and enterprises planning local private deployment, however, the challenge is not merely the capital cost of hardware. Deploying a trillion-parameter-class MoE model is far beyond a desktop task that can be solved by buying a few flagship consumer GPUs.
This article uses a representative MoE model with 2.8 trillion total parameters and roughly 104 billion activated parameters per token as an example. It breaks down the hardware requirements for local deployment and provides configuration references and cost estimates for different business scenarios.
1. Core Technical Breakdown: The Underlying Logic of VRAM Requirements
To estimate the hardware needed for local deployment, the first task is to determine how much VRAM the model requires to run. This model natively uses MXFP4 (4-bit floating point) precision. This is a critical premise: 4-bit weights are the model’s official native precision, not a forced quantization from FP16 or BF16. Running at native 4-bit therefore maximizes memory savings while preserving the model’s baseline inference quality.
1. Model weight VRAM footprint
At 4-bit precision, each parameter occupies 0.5 bytes (4 bits / 8 bits per byte). The model has 2.8 trillion total parameters, so the pure weight footprint is calculated as follows:
2.8 trillion × 0.5 bytes = 1.4 TB
This means that merely loading the full model into GPU memory requires at least 1.4TB of physical VRAM capacity.
2. Runtime and state VRAM footprint
Weight memory is only the baseline. During actual inference, the system also needs additional VRAM for activations, KV cache (or state-space cache), and runtime context management. Although the number of parameters activated per token is relatively small compared with the total parameter count, the system still needs to maintain large routing tables and expert states during scheduling. Based on a combined estimate, a single-user, single-request scenario requires approximately 1.5TB of actual GPU memory.
This 1.5TB hard threshold immediately rules out conventional workstations and consumer GPUs, pushing local deployment to standard server or even cluster-level infrastructure.
2. Hardware Configuration Tiers for Different Performance Requirements
Based on the 1.5TB VRAM baseline, local deployment targets can be divided into three typical performance tiers, each corresponding to different business scenarios and hardware investment levels.
Tier 1: Basic Usability (Runs, but Speed Is Not Critical)
In some internal testing or very low-frequency usage scenarios, the only requirement is that the model can load and produce results. There is no strict requirement for generation speed.
1. The dilemma of previous-generation mainstream enterprise GPUs
From a theoretical capacity perspective, deploying this model requires approximately 19 previous-generation mainstream enterprise GPUs with 80GB of memory (19 × 80GB = 1520GB), or 11 same-generation GPUs with 141GB of memory (11 × 141GB = 1551GB). However, there is a critical technical conflict: previous-generation GPU architectures cannot natively execute MXFP4 precision computation. If the model is forced to run on these cards, the inference framework must dequantize the 4-bit weights to a higher precision at runtime. This introduces significant instruction overhead and register pressure. The memory may be large enough to “fit” the model, but actual inference efficiency will be seriously degraded and may even create a severe compute bottleneck.
2. Realistic minimum configuration
A more reasonable approach is to use new-generation enterprise GPUs that natively support low-precision computation. When mainstream open-source inference frameworks publish deployment guides for this type of model, their recommended starting configuration is also 8-way parallelism. Using 8 new-generation large-memory GPUs (approximately 192GB or more per card) can just meet the 1.5TB VRAM requirement while natively supporting 4-bit acceleration.
Conclusion: even if the goal is only to “make it run,” this is already a genuine server-class project. The procurement cost of a single high-spec 8-GPU server starts at a seven-figure level.
Tier 2: Smooth Single-User Reading (25 tokens/s)
25 tokens/s is widely regarded as the comfortable lower bound for a human reading native-language text. If a single user needs to reach this speed, can the hardware starting point be lowered?
1. The memory wall prevents reducing the GPU count
Many developers have a misconception: since only one person is using the model and the compute load is small, an 8-GPU server can be reduced to 2 or 3 cards. The biggest challenge of MoE models, however, is the “memory wall,” not the “compute wall.” The 1.4TB of weights is an indivisible whole. Regardless of how many concurrent requests there are, the full 1.4TB must reside in VRAM. Therefore, even for single-user use, at least 8 new-generation enterprise GPUs are still required to cover the VRAM capacity.
2. Speed assessment
On an 8-GPU server with native low-precision support, the model activates only about 104 billion parameters per pass. Compute for single-request LLM inference is usually not the main bottleneck, and reaching “tens of tokens per second” is a reasonable expectation. Therefore, 25 tokens/s is a relatively achievable target. It is important to note, however, that this speed will decline as context length increases. Before launch, rigorous stress testing must be performed with the context lengths used in actual business scenarios, such as 32K, 128K, or longer.
Tier 3: High-Concurrency Production (10 Concurrent Requests at 25 tokens/s Each)
When the model enters a production environment, it often needs to support multiple simultaneous users. Suppose the business requires support for 10 concurrent requests, with each user maintaining 25 tokens/s. This means the inference system must stably provide 250 tokens/s of aggregate output throughput.
1. VRAM configuration may not need to increase
An interesting technical phenomenon is that moving from Tier 2 to Tier 3 does not necessarily require doubling the hardware investment. Under the MoE architecture, multiple concurrent requests can share computation graphs and model weights through continuous batching. According to industry testing estimates, a reasonably configured 8-GPU node can theoretically hold around 40 concurrent requests under normal context lengths (not extreme million-token lengths). Therefore, 10 concurrent requests are fully within the capacity of a single node.
2. Compute scheduling and network requirements
Although no additional physical nodes are required, extracting 8-GPU performance to support high concurrency places extremely high demands on the underlying software stack. Enterprises need to rely on efficient inference engines for request scheduling and VRAM reuse. For deployment form, enterprises can consider mature cloud-native compute platforms such as Baidu AI Cloud. By using their high-performance container services, compute network, and monitoring and alerting systems, enterprises can let the cloud platform manage complex underlying hardware scheduling and focus on upper-layer business logic development.
3. Summary and Deployment Recommendations
Based on the analysis above, deploying a 2.8-trillion-parameter MoE model is fundamentally a heavy-asset investment in enterprise-grade compute infrastructure:
- Very high hardware barrier: The 1.5TB VRAM requirement directly rules out consumer hardware. An 8-GPU new-generation enterprise server is the minimum starting point.
- Precision matching is critical: New-generation hardware with native 4-bit compute support must be selected. Otherwise, the performance loss caused by dequantization will waste the expensive hardware investment.
- Concurrency advantage is clear: The MoE architecture is highly cost-effective for moderate concurrency. A single node can support dozens of concurrent requests.
For enterprises with private deployment needs, directly purchasing physical servers has high upfront costs and carries device depreciation risk as technology iterates. In the initial stage, it is advisable to use dedicated compute leasing services or managed LLM platforms from leading cloud providers such as Baidu AI Cloud to validate the business model in a more flexible way. After concurrency scale and the commercial loop mature, enterprises can then consider customized private cluster deployment.
