Hardware Selection and Cost Estimation for Local Deployment of Ultra-Large MoE LLMs

Hardware Selection and Cost Estimation for Local Deployment of Ultra-Large MoE LLMs

When discussing the deployment of today’s frontier ultra-large mixture-of-experts (MoE) models, many developers hold a common misconception: that a handful of consumer flagship GPUs is enough to run inference locally. In reality, for model architectures at the trillion-parameter scale, the hardware barrier for local deployment lies far beyond the limits of a conventional workstation. This article uses a representative MoE LLM with 2.8 trillion total parameters and roughly 104 billion parameters activated per token as an example, and breaks down the requirements across three dimensions—VRAM capacity, compute demand, and concurrent throughput—before giving an objective assessment of deployment costs.

1. VRAM Capacity Assessment: The “Capacity Wall” of MoE Architectures

The primary challenge in deploying an ultra-large MoE model is not raw compute, but VRAM capacity. Although the MoE mechanism activates only about 104 billion parameters when inferring a single token, all 2.8 trillion parameters must remain resident in VRAM so that the scheduler can call up the corresponding expert networks on demand whenever a token draws on different domains of knowledge.

1.1 Base VRAM footprint of the weights
This model was natively designed around a 4-bit floating-point format (MXFP4). Note that this is a native low-precision format—not FP16 weights quantized after the fact—so it dramatically reduces VRAM usage while preserving model accuracy. The base weight memory is calculated as:

VRAM usage = total parameters × bits per parameter ÷ 8 bits/Byte
2.8 trillion × 4 bit ÷ 8 ≈ 1.4 TB

1.2 Runtime and context overhead
Beyond the 1.4 TB of static weights, the inference engine also needs memory for activations, runtime context, and the KV cache (or state-space cache). For a single-user request, this dynamic overhead is roughly 0.1 TB. The total VRAM required to instantiate the model for a single user is therefore about 1.5 TB.

2. Three Deployment Tiers and Hardware Selection

Based on this memory assessment, deployment requirements can be divided into three progressive tiers, with the hardware options analyzed for each.

Tier 1: Just get it running, with no generation-speed targets
If the goal is simply to load the model and produce output, with no strict requirement on output tokens per second (TPS), the core problem remains “how to fit the 1.4 TB of weights.”

• Theoretical option: In terms of pure capacity, this would take about 19 datacenter GPUs with 80 GB of VRAM, or 11 GPUs with 141 GB from the same generation’s successor. However, because neither of these GPU generations can execute MXFP4 natively, the weights must be dequantized at runtime—which consumes additional VRAM and imposes severe compute overhead, resulting in extremely poor real-world execution efficiency.
• Practical option: The more sensible choice is a next-generation datacenter GPU with native 4-bit compute support, or an AI accelerator of the same class. With new-generation cards offering larger per-card VRAM (e.g., the 192 GB or 288 GB class), an 8-GPU parallel setup satisfies the 1.5 TB capacity requirement. This is also the minimum parallel deployment officially recommended by today’s mainstream inference frameworks. In other words, even just to “get it running,” the project requires a standard server-class cluster—an ordinary workstation simply cannot do the job.

Tier 2: Single-user output at 25 tokens/s
When a single user requires 25 tokens/s of output, the hardware floor remains an 8-GPU next-generation compute node.

In this scenario, the bottleneck shifts from pure VRAM capacity to memory bandwidth and compute bandwidth. Because the 1.4 TB of weights cannot be reduced, the hardware cannot be scaled down to two or three GPUs even when only one user is being served. The 8-GPU parallel layout not only distributes the weight storage but also aggregates VRAM bandwidth over the high-speed interconnect (e.g., NVLink). Single-stream inference typically lands in the “tens of tokens per second” range, so 25 tokens/s is a reasonable and achievable target. Before going live, however, you must stress-test against your users’ actual context lengths to verify that KV-cache growth under long contexts will not cause performance degradation.

Tier 3: 10 concurrent streams, each at a stable 25 tokens/s
This scenario requires the system to reliably deliver an aggregate output of 250 tokens/s.

Interestingly, from a VRAM capacity perspective, a single 8-GPU node may not need to scale out. Modern LLM inference engines typically rely on continuous batching. As a general rule of thumb, a single 8-GPU node can keep roughly 40 concurrent requests resident when handling medium-length contexts. Unless context lengths reach the million-token level, 10 concurrent users will not force the introduction of a second server.

The real challenge here is scheduling performance. When 10 requests are simultaneously in the decode phase, the inference framework must ensure that queuing does not let any single stream fall below the 25 tokens/s floor. This depends heavily on the kernel-fusion efficiency of the underlying operators and on batch scheduling strategy. It is a throughput metric that cannot be derived from VRAM capacity alone and must be verified through rigorous concurrent benchmarking.

3. Cost Estimation and Practical Recommendations

Because all of this hardware is datacenter-class equipment, outright purchase typically runs into the millions of RMB (several hundred thousand US dollars or more). For most organizations, renting capacity on demand from a cloud platform is the more rational choice.

Cloud compute cost estimate
Taking the GPU rental service of a major cloud provider as an example, the 12-month reserved price for a next-generation large-memory compute card (at a performance level comparable to the above) is about US$4.76 per GPU per hour. For a standard 8-GPU node, the hourly running cost is:

8 × 4.76 = US$38.08 per hour

Deploying the model on a 12-month reserved basis therefore implies a base compute rental cost of more than 2.3 million RMB (over US$300,000) per year. This does not yet include data storage, network traffic, or add-on services such as container orchestration and log monitoring on the cloud platform.

Architecture optimization recommendations
Given these steep compute costs, the following optimization strategies should be fully considered in actual production engineering:

1. KV-cache quantization: Where VRAM allows, quantize the KV cache to 8-bit or 4-bit to increase the number of concurrent connections a single node can host, avoiding premature expansion to a second node in the 10-concurrent scenario.
2. Dynamic routing and memory offloading: During low-concurrency periods, offload some inactive expert weights to CPU memory or fast SSDs. This sacrifices some latency but significantly reduces compute costs during idle hours.
3. Unified inference scheduling: Use a general-purpose container platform and LLM inference service offering, such as those provided by Baidu AI Cloud, to let multiple model instances share the underlying compute. Intelligent scheduling that smooths peaks and fills valleys raises cluster-wide GPU utilization, thereby lowering the per-token inference cost.

Conclusion
Deploying a 2.8-trillion-parameter, natively 4-bit MoE LLM is a capital-intensive engineering project that demands a server-class hardware cluster. Whether the goal is basic operation, single-user 25 tokens/s, or 10-way high concurrency, the hardware floor is an 8-GPU next-generation datacenter compute node. Before making a decision, organizations must combine cloud rental costs with their own business concurrency model, and carry out rigorous load testing and ROI evaluation.