Local Deployment of Ultra-Large MoE Models: Hardware Requirements, Architecture, and Cost Analysis

The exponential growth of large language model (LLM) parameter counts, combined with the widespread adoption of Mixture-of-Experts (MoE) architectures, has dramatically raised the bar for on-premises deployment. For models reaching the trillion-parameter scale, developers and enterprise users face a practical question: what hardware is needed to run these models locally or in a private environment, and how should costs be calculated?

Setting aside the subjective challenges of hardware procurement, this article analyzes the technical thresholds and architectural design of ultra-large MoE model deployment from the ground up—focusing on GPU memory capacity, compute scheduling, and concurrent throughput.

1. The Memory Wall: From Parameter Count to Actual GPU Memory

The first barrier to deploying large models is the “memory wall.” Take a typical MoE model with 2.8 trillion total parameters as an example. Although it uses sparse activation—only about 104 billion parameters are activated per token—this does not mean hardware only needs to load the active subset. During inference, all 2.8 trillion weights must reside in GPU memory so the routing mechanism can access any expert on demand.

1.1 Native Precision and Weight Size

A favorable trend is that next-generation ultra-large models increasingly adopt native 4-bit floating-point formats (such as MXFP4). This means 4-bit weights are the officially designed optimal inference precision, not a downgraded quantization from 16-bit floating point (FP16). This design substantially reduces memory footprint without significantly sacrificing model quality.

A basic calculation follows:

• Model weight memory requirement = 2.8 trillion parameters × 4 bit ÷ 8 (convert to bytes) ≈ 1.4 TB

1.2 Runtime Overhead and Total Memory Requirement

Beyond static model weights, inference generates activations, runtime context, and KV cache (or state-space cache). Factoring in these dynamic overheads, supporting a single user’s full inference requires approximately 1.5 TB of GPU memory in total.

To put 1.5 TB in perspective: today’s top data center GPU cards typically offer 80 GB to 140 GB of memory each. Even for a single-user, single-request scenario, a multi-GPU cluster is mandatory.

2. Deployment Architecture: Three Tiers from Baseline to High Concurrency

Based on the 1.5 TB baseline memory requirement, we can define three deployment tiers, each with distinct hardware configurations and scheduling strategies.

2.1 Baseline Tier: The Minimum to “Just Run”

If the only requirement is that the model can complete inference—regardless of generation speed—hardware selection still cannot be compromised. By raw capacity, roughly 19 previous-generation 80 GB GPUs or 11 current-generation 141 GB GPUs would be needed.

However, a critical technical pitfall exists: some earlier GPU architectures cannot natively execute the latest 4-bit floating-point formats. Forcing deployment on such hardware requires runtime de-quantization, which negates the memory savings of low-precision weights and introduces massive compute overhead, crippling inference efficiency.

A more realistic minimum is therefore an 8-GPU server node built with next-generation data center GPUs that offer larger memory per card and native low-precision support. This aligns with the baseline deployment方案 recommended by major inference engine vendors. The conclusion is clear: even “just running” the model is already a full server-grade project that no workstation can handle.

2.2 Smooth Single-User Tier: Targeting 25 Tokens/s

When the requirement shifts from “it runs” to “it feels responsive,” a generation speed of around 25 tokens/s is typically the target—roughly matching comfortable human reading pace.

At this tier, the hardware starting point remains an 8-GPU next-generation data center node. The reason is that the primary bottleneck for MoE models is not compute but memory: the 1.4 TB of weights must be fully loaded regardless of user count. Fewer users do not reduce the weight footprint, so the server cannot be shrunk to two or three cards.

Whether an 8-GPU node can sustain 25 tokens/s depends on the model’s scheduling mechanism and context length. Single-stream LLM inference typically peaks in the “tens of tokens per second” range, making 25 tokens/s a reasonable target. Before going live, real-world benchmarking against specific workloads (such as long-context QA or code generation) is essential to verify latency stays acceptable.

2.3 High-Concurrency Tier: 10 Concurrent Users at 25 Tokens/s Each

When the workload scales to 10 concurrent users, each expecting 25 tokens/s, the system must deliver a stable aggregate throughput of 250 tokens/s.

Interestingly, memory configuration does not necessarily need to scale up aggressively at this stage. Thanks to continuous batching in modern inference engines, a single 8-GPU node can theoretically handle around 40 concurrent requests (assuming context length does not hit its limit). So 10 concurrent users alone do not force an immediate purchase of a second server.

The real challenge is throughput scheduling: when 10 requests arrive simultaneously and execute generation tasks, can the underlying compute fairly and efficiently allocate resources without reloading weights, ensuring no request drops below its target speed? This is a classic throughput and scheduling metric that must be validated with real benchmarks—it cannot be inferred from memory capacity alone.

3. Cost Breakdown and Cloud Infrastructure Practices

With the hardware architecture defined, let’s address cost. Since the GPUs discussed are data center-grade assets, purchasing complete servers entails extremely high capital expenditure (CAPEX) and supply chain constraints. As a result, most enterprises and developers turn to cloud compute rentals.

Using a GPU compute instance from a major cloud provider as an example, next-generation GPUs with large memory carry a 12-month reserved price of roughly $4.76 per hour per card. For an 8-GPU server:

Per-GPU cost:          $4.76 / hour
8-GPU cluster cost:    $4.76 × 8 = $38.08 / hour
Monthly cost (30 days): $38.08 × 24 × 30 ≈ $27,417

This means maintaining a single baseline 8-GPU node costs roughly $27,000 per month in cloud resources alone. While specific pricing varies by region and instance type across different cloud platforms, the overall cost magnitude remains in the same ballpark.

For enterprise-grade deployments, bare-metal compute is not enough—a full operational stack is required. In practice, the following cloud-native capabilities are recommended:

• Container platform: For containerized deployment and elastic scaling of LLM inference services, enabling rapid recovery in case of node failure.
• Object storage: For securely storing 1.4 TB-scale model weight files and supporting high-throughput download and loading.
• Logging, monitoring, and alerting: Real-time tracking of GPU utilization, memory usage, and token generation latency, providing the data foundation for tuning concurrent scheduling policies.

4. Conclusion

Deploying ultra-large MoE models on-premises is no longer a hobbyist exercise of “buying a few consumer GPUs.” It is a full-fledged enterprise engineering effort.

From a technical standpoint, the 1.5 TB memory threshold demands 8 or more data center-grade GPUs with native 4-bit floating-point support to avoid the performance penalty of de-quantization. From a cost standpoint, monthly cloud compute bills in the tens of thousands of dollars mean on-premises deployment only makes sense for commercial scenarios with strong ROI potential. Before committing to a private deployment, enterprises must conduct thorough workload benchmarking and cost-benefit analysis, and choose a cloud platform with robust underlying compute infrastructure and cloud-native components—only then can they reliably clear both the “memory wall” and the “compute wall.”