Local Deployment of Ultra-Large MoE Models: Cost and Hardware Configuration Analysis

Local Deployment of Ultra-Large MoE Models: Cost and Hardware Configuration Analysis

As large language models continue to scale, the exponential growth in parameter counts and the widespread adoption of Mixture-of-Experts (MoE) architectures have pushed the barrier to local deployment ever higher. For models that routinely reach trillions of parameters, the first question developers and enterprises face is often: Can this even run locally? What hardware does it actually require? And what will it ultimately cost?

1. The Core Bottleneck: VRAM Capacity, Not Raw Compute

Before discussing specific configurations, one conclusion must be made clear: for ultra-large MoE models, the primary deployment bottleneck is typically VRAM capacity rather than computational throughput.

The total parameter count of such models can reach 2.8 trillion. Although only about 10% of parameters (e.g., 104 billion) are activated when generating a single token during inference, all inactive expert network weights must still reside permanently in VRAM so they can be called upon whenever the front-end routing mechanism triggers them.

Fortunately, newer ultra-large models are often designed natively in low-precision formats (such as 4-bit floating point). This means 4-bit weights are the officially designed standard operating precision, not the result of forcibly quantizing and compressing higher-precision weights. This significantly reduces the baseline VRAM footprint.

Even so, we can still make a rough estimate using the following formula:

• VRAM required for model weights alone = total parameters × bits per parameter ÷ 8
• Calculation: 2.8 trillion × 4 bit ÷ 8 ≈ 1.4 TB

Adding model activations, runtime context, and KV cache (or state-space cache), supporting actual single-user inference requires at least about 1.5 TB of GPU VRAM. This capacity far exceeds what ordinary workstations or even a single standard server can accommodate, fundamentally making such deployments data-center-level projects.

2. Hardware Configuration Breakdown for Three Usage Tiers

Based on different business requirements, we can divide deployment needs into three typical tiers to derive reasonable hardware configurations.

Tier 1: Basic Usability — It Runs, Speed Not Critical

If the business only requires the model to complete inference without strict demands on generation speed, the focus remains on “filling up 1.5 TB of VRAM.”

From theoretical capacity estimates, roughly 19 data-center-grade compute cards with 80 GB VRAM each, or 11 newer-generation compute cards with 141 GB each, would be needed. However, there is a technical pitfall here: some previous-generation compute cards cannot natively execute 4-bit floating-point instructions. If deployed forcibly, runtime dequantization is required, which not only consumes additional VRAM but also severely slows inference efficiency.

Therefore, a more realistic minimum configuration uses 8 newer-generation data-center compute cards (with per-card VRAM in the 192 GB to 288 GB range). Mainstream open-source inference frameworks also generally recommend an 8-card parallel architecture as the starting point in their reference solutions for such large models. This signals that even the most basic “it runs” requirement has completely left the realm of individual developers.

Tier 2: Smooth Single-User Experience — Targeting 25 Tokens/s

When single-user generation speed is required to reach 25 tokens/s (the threshold for comfortable human reading), the hardware starting point should still remain at the level of 8 newer-generation compute cards.

The reason is that ultra-large models cannot reduce card count just because only one user is active. The 1.4 TB of weights must be fully loaded, and two or three cards simply cannot provide such a large VRAM space.

As for whether an 8-card configuration can stably output 25 tokens/s in single-stream inference, this depends on the compute cards’ VRAM bandwidth and interconnect topology. Generally, the upper limit of single-stream large model inference speed is often constrained by “VRAM bandwidth / model parameter count.” When activated parameters are in the hundred-billion range, the high bandwidth of newer-generation compute cards is sufficient to support output of tens of tokens per second. So 25 tokens/s is a reasonable target, but before actual deployment, rigorous stress testing must be conducted with real context lengths.

Tier 3: Multi-Concurrency Throughput — 10 Concurrent Streams at 25 Tokens/s Each

This means the system needs to stably provide 250 tokens/s of aggregate output capacity.

Interestingly, for 10 concurrent streams, the VRAM configuration may not necessarily need to increase compared to Tier 2. Because modern inference frameworks typically employ continuous batching techniques. An 8-card node with 1.5 TB of VRAM can theoretically hold more than 40 concurrent requests simultaneously, provided user context lengths are not in the million-token range. Therefore, 10 concurrent streams alone do not force an upgrade to a multi-node cluster.

The real challenge lies in scheduling performance and throughput bottlenecks. When 10 requests are generating tokens simultaneously, the overhead of compute unit contention and context switching rises significantly. Maintaining 25 tokens/s per stream depends not only on VRAM capacity but also on the efficient scheduling strategies of the underlying inference engine. This must be verified through actual testing and cannot be estimated from paper specifications alone.

3. Cost Estimation and Cloud Deployment in Practice

Since the hardware barrier is locked at data-center-grade compute cards, physical procurement locally is extremely difficult and time-consuming (involving not just capital but also supply chain channels). Therefore, evaluating deployment costs for such models typically uses cloud platform compute rental pricing as the primary reference.

Taking a common high-end compute card instance on cloud platforms as an example, if using reserved instance billing (typically reserved for 12 months to obtain a lower unit price), the hourly unit price per high-end compute card is approximately $4.7. For an 8-card node, the hourly operating cost per node is about $38.

Converted:

• Daily cost: approximately $912
• Monthly cost: approximately $27,000

This is only the bare-metal compute cost. In actual enterprise deployments, the following hidden costs must also be considered:

1. Storage costs: 1.4 TB of model weight files requires cloud disks or parallel file systems with extremely high IOPS to support second-level loading.
2. Networking and load balancing: If business volume surges and expansion to multiple nodes is needed, the cost of high-speed inter-node networking rises sharply.
3. Operations and monitoring: A complete logging service and monitoring/alerting system must be established to track GPU utilization, VRAM fragmentation, and inference latency fluctuations in real time.

4. Summary and Architecture Recommendations

Overall, deploying trillion-parameter MoE models is by no means a problem that can be solved by purchasing a few consumer-grade GPUs. It is fundamentally constrained by the 1.5 TB-level VRAM barrier.

For enterprises with private deployment requirements, the following technical paths are recommended:

• Validate business necessity first: Assess whether general-purpose LLM APIs can meet requirements. If private deployment is not strictly necessary, API usage costs are far lower than self-built compute.
• Adopt cloud-native containerized deployment: Leverage container platforms and elastic compute scheduling provided by cloud vendors to release some instances during low-demand periods to reduce costs.
• Deeply optimize the inference engine: Introduce VRAM optimization techniques such as PagedAttention to reduce KV cache fragmentation and improve concurrency capacity without adding hardware.

Only by finding a balance among compute scale, scheduling optimization, and budget can the local deployment of ultra-large models have practical engineering value.