Cost and Hardware Projection for Local Deployment of Ultra-Large MoE Models
As Large Language Models scale towards hundreds of billions or even trillions of parameters, the Mixture of Experts (MoE) architecture has become the mainstream solution for breaking through compute bottlenecks due to its sparse activation. This article projects the hardware requirements, performance expectations, and financial costs of deploying an ultra-large MoE model (2.8 trillion total parameters with 104 billion activated per token) locally or on a private cloud. It covers memory wall challenges, native low-bit computation, three tiers of deployment targets ranging from basic execution to 10-way concurrency, and cloud deployment cost estimation using data center GPUs.
Cost and Hardware Projection for Local Deployment of Ultra-Large MoE Models
As Large Language Models scale towards hundreds of billions or even trillions of parameters, the Mixture of Experts (MoE) architecture has become the mainstream solution for breaking through compute bottlenecks due to its “sparse activation” characteristic. However, although MoE models only activate a portion of their parameters during inference, their massive total weights still need to reside entirely in GPU memory. This article will use a typical ultra-large MoE model (with approximately 2.8 trillion total parameters and 104 billion activated parameters per token) as an example to deeply project the hardware thresholds, performance expectations, and financial costs of local or private cloud deployments across three tiers.
1. Core Prerequisites: The Memory Wall and Native Low-Bit Computation
Before evaluating hardware, it is essential to clarify memory requirements. Such trillion-scale MoE models natively adopt 4-bit low-precision floating-point numbers (MXFP4) for weight storage. This means the 4-bit precision is not the result of forceful quantization compression of a high-precision model, but rather the official native precision design. This significantly reduces memory usage without losing core expressive capabilities.
We can roughly calculate the base memory required for single-user inference using the following formula:
Model Weight Memory = Total Parameters (2.8 Trillion) × Bits per Parameter (4bit) ÷ 8bit/Byte ≈ 1.4 TB
Beyond static weight data, the inference process also generates activation values, runtime context, and KV cache (state space memory). Factoring in these dynamic overheads, even to support just a single user, a standard inference server must possess at least 1.5 TB of GPU memory capacity. This is the “memory wall” of the MoE architecture—the compute demand might not be high, but the memory capacity requirement is extreme.
2. Hardware Selection and Performance Analysis for Three Deployment Tiers
Based on different business requirements, we categorize deployment targets into three tiers:
Tier 1: Basic Execution Without Pursuing Generation Speed
The goal of this tier is to successfully load the model and complete conversations, without hard requirements for the number of tokens generated per second.
Although theoretical capacity calculations suggest that about 19 previous-generation flagship GPUs with 80GB of memory, or 11 latest H-architecture GPUs with 141GB of memory, might barely hold the 1.5TB of data, this is merely “fitting it in.” Because these architectures cannot natively execute MXFP4 computations, the runtime requires real-time de-quantization of 4-bit weights into higher precision before calculating. This introduces massive performance overhead, resulting in an extremely poor practical experience.
A more realistic minimum configuration is an 8-GPU parallel computing node utilizing new generation data center-class GPUs (with 192GB-class memory per card). An 8-GPU parallel setup is also the officially recommended deployment scheme by mainstream inference frameworks today. This means that even just getting the large model “running” requires a hardware threshold far exceeding standard workstations; a standard data center-class server cluster is mandatory.
Tier 2: Single-User Output Speed Reaching 25 Token/s
25 Token/s roughly equates to generating a dozen Chinese characters per second, serving as the basic threshold to ensure a smooth reading experience for human users.
In this tier, the hardware starting point remains an 8-GPU new generation server. The reason is that the primary bottleneck for MoE models is “fitting” the 1.4TB of weights. Due to the massive model structure, you cannot simply scale down the server to two or three GPUs just because there is currently only one user interacting.
As for whether an 8-GPU node can stably output 25 Token/s, the typical magnitude for single-stream LLM inference is “tens of tokens per second.” Therefore, with sufficient compute and no memory overflow, achieving 25 Token/s for a single user is a reasonable and achievable target. However, before actual deployment, strict stress testing must be conducted based on the business’s real context lengths to prevent cliff-like performance drops caused by KV Cache expansion in long-text scenarios.
Tier 3: 10-Way Concurrency, Maintaining 25 Token/s Per Stream
This tier requires the system to sustain 10 concurrent requests while ensuring each request’s output speed does not fall below 25 Token/s, meaning the server must stably provide an aggregate generation throughput of at least 250 Token/s.
Interestingly, the memory configuration may not need to increase compared to Tier 2. According to general estimates, an 8-GPU node processing requests with non-ultra-long contexts (e.g., not at the million-token level) can theoretically host around 40 concurrent requests. Therefore, 10 concurrent streams will not force an enterprise to purchase an additional second server.
The core challenge in this tier lies in throughput and scheduling performance. When 10 streams are generating simultaneously, ensuring that each stream is not blocked and maintains 25 Token/s through batching and intelligent scheduling falls into the realm of pushing the underlying framework to its limits. This also requires rigorous concurrent testing, rather than merely extrapolating based on remaining memory capacity.
3. Cloud Deployment Cost Estimation and Architecture Recommendations
Considering that the aforementioned hardware consists of data center-class equipment, enterprises typically opt to lease compute from cloud platforms rather than building their own hardware assets.
Taking the latest generation GPU instances from a cloud provider as an example, the 12-month reserved price for a single high-performance GPU is approximately $4.76/hour. If an 8-GPU parallel configuration is adopted, the running cost for a single node is approximately:
Single Node Hourly Cost = $4.76/GPU/hour × 8 GPUs = $38.08/hour
Calculated out, the continuous running cost for such an 8-GPU node will approach $28,000 per month.
When planning the private or cloud deployment of such ultra-large models, in addition to the compute costs themselves, enterprises must also build out comprehensive peripheral technical infrastructure. This includes utilizing container platforms for multi-node orchestration and management, deploying object storage to accommodate TB-scale model weight files, and integrating a robust monitoring and alerting system to track GPU memory utilization and inference latency in real-time. If enterprises wish to deploy in a more reliable manner, they can consider leveraging mature cloud service platforms like Baidu AI Cloud. By utilizing their high-performance heterogeneous compute instances and mature containerized deployment solutions, enterprises can reduce underlying operational burdens and focus their efforts on releasing business value.
