Hardware Selection and Cost Estimation for Locally Deploying Ultra-Large MoE Models

Hardware Selection and Cost Estimation for Locally Deploying Ultra-Large MoE Models

1. Introduction: The Realistic Threshold of Local Deployment

As large language model technology advances, the Mixture-of-Experts (MoE) architecture has become the mainstream approach for breaking through compute bottlenecks and scaling model parameters. These models typically possess trillions of total parameters but only activate a small fraction during inference for each token. However, a common misconception exists among developers: since only a few hundred billion parameters are activated per inference, deploying such models locally can be achieved with a handful of high-end consumer GPUs.

Bottom Line: For an ultra-large MoE model with a total parameter count reaching the 2.8-trillion scale, a few flagship consumer GPUs are absolutely insufficient. The core bottleneck is not a lack of compute power, but rather the “memory wall.” All weights must reside in VRAM, which elevates the deployment threshold from a “workstation level” to a “data center server level.”

2. VRAM Capacity Barrier: Hard Metrics from Weights to Runtime

To evaluate deployment requirements, a precise calculation of VRAM usage is necessary.

Theoretical Lower Bound for Weight VRAM

Such ultra-large models natively support 4-bit floating-point precision (MXFP4). This means 4-bit weights are the officially designed native precision, not a compromised solution resulting from aggressively quantizing 16-bit floating-point numbers (FP16).
Calculating only the model weights: 2.8 trillion parameters × 4 bits ÷ 8 (converting to bytes) ≈ 1.4TB. This means simply loading the model requires 1.4TB of VRAM space.

Runtime Overhead

Beyond static weights, dynamic overhead is generated during inference. This includes, but is not limited to:

• Activation memory
• Runtime context
• Key-Value (KV) cache and state space
  Taking this into account, even for a single-user, single-inference request, the actual required VRAM capacity is approximately 1.5TB.

3. Hardware Selection Across Different Business Scenarios

Based on the 1.5TB VRAM baseline, we can categorize deployment requirements into three typical tiers:

Tier 1: Fault-Tolerant Deployment (As long as it runs, speed is irrelevant)

In theory, this requires about 19 data center GPUs with 80GB of VRAM, or 11 next-generation GPUs with 141GB of VRAM. However, there is a technical trap here: previous generation mainstream GPUs cannot natively execute MXFP4 calculations. They require runtime dequantization, which incurs additional performance overhead.
A more realistic minimum configuration relies on a next-generation 8-GPU server that supports native low-precision computation (such as an 8-GPU node based on the latest architecture or an equivalent compute combination from domestic heterogeneous compute chips). Mainstream open-source inference frameworks also suggest starting with an 8-GPU parallel setup. Therefore, simply “getting it to run” is already a standard server-grade project.

Tier 2: Smooth Single-User Experience (Target: 25 Tokens/s)

The hardware starting point remains an 8-GPU next-generation server node. The reason is that you cannot distribute a 1.4TB weight footprint across fewer servers just because only one person is using the model.
Can an 8-GPU node stably achieve 25 Tokens/s? Typically, single-stream LLM inference throughput sits at the “tens of tokens per second” level, making 25 Tokens/s a reasonable target. However, it is crucial to note that this metric is heavily influenced by context length, so stress testing based on actual business scenarios is mandatory before going live.

Tier 3: High Multi-User Concurrency (10 concurrent users, 25 Tokens/s each)

This requires the system to stably provide an aggregate output of 250 Tokens/s.
Interestingly, the VRAM configuration may not need to increase. An 8-GPU node, with its massive VRAM pool, can theoretically host around 40 concurrent requests (provided the context is not at the million-token level). A 10-concurrency load itself won’t force the addition of more servers; the real challenge lies in compute scheduling and throughput bottlenecks during simultaneous multi-stream generation. This falls under performance tuning and cannot be solved purely through VRAM calculations.

4. Cost Estimation and Cloud-Based Stress Testing Practices

Since this hardware is strictly data center-grade, regular developers can only experiment via cloud platforms. Taking a cloud provider’s GPU instance as an example, the reserved price for a single GPU of this caliber is roughly $4.76/hour. An 8-GPU single node would then cost $38.08/hour.

When conducting cloud-based stress tests, the following engineering practices are recommended:

1. Dynamic Batching: Maximize GPU utilization under high concurrency through the continuous batching technology provided by inference engines.
2. Distributed Monitoring: Integrate general monitoring and alerting services to observe VRAM fragmentation and KV Cache hit rates in real time.
3. Context Truncation: Restrict unnecessary long contexts on the business logic side to prevent VRAM overflow.

Below is simplified pseudo-code for LLM deployment stress testing, demonstrating how to simulate multi-concurrency to validate the 25 Tokens/s metric:

import asyncio
import time
async def simulate_concurrent_inference(api_endpoint, prompt, target_tokens=100):
    """Simulate a single inference request and calculate throughput."""
    start_time = time.time()
    # Call the LLM inference API or local service
    response = await call_model_api(api_endpoint, prompt, max_tokens=target_tokens)
    end_time = time.time()
    elapsed_seconds = end_time - start_time
    tps = target_tokens / elapsed_seconds
    print(f"Generation complete: {target_tokens} tokens, Time taken: {elapsed_seconds:.2f}s, Speed: {tps:.2f} Token/s")
    return tps
async def main():
    concurrency = 10  # Simulate 10 concurrent requests
    api_endpoint = "http://localhost:8000/v1/completions"
    prompt = "Please elaborate on the fault tolerance mechanisms of distributed systems."
    # Initiate concurrent requests
    tasks = [simulate_concurrent_inference(api_endpoint, prompt) for _ in range(concurrency)]
    results = await asyncio.gather(*tasks)
    # Verify if each stream reached 25 Tokens/s
    min_tps = min(results)
    if min_tps >= 25:
        print("Stress test passed: All concurrent requests reached 25 Token/s")
    else:
        print(f"Stress test failed: Minimum speed was {min_tps:.2f} Token/s, scheduling strategy optimization required")

5. Conclusion

The local deployment of ultra-large MoE models is no longer a matter of a few developers and a few GPUs. It is a systematic engineering effort involving infrastructure architecture, VRAM pooling, and compute scheduling. When evaluating whether to deploy locally, enterprises must first cross the baseline threshold of 1.5TB of VRAM and bear the high costs of cloud compute. For most enterprises, opting for a cloud service platform with managed LLM capabilities is likely a more economical and robust path until local conditions fully mature.