Local Deployment of Trillion-Parameter MoE Models: Hardware Requirements and Cost Breakdown

1. Correcting a Common Misconception

Many developers considering local deployment of large language models instinctively ask whether buying a few high-end GPUs is enough. That assumption may hold for models with hundreds of millions or a few billion parameters. It does not hold for trillion-parameter sparse Mixture-of-Experts models.

This article analyzes a typical trillion-parameter sparse model with roughly 2.8 trillion total parameters. Each token activates about 104 billion parameters. Even so, inference still requires loading the full set of weights, not just the active portion.

This leads to the first core constraint: weight capacity requirements far exceed what a single-machine GPU setup can provide.

2. How Large Is the Weight Footprint?

The model stores weights in a native 4-bit floating-point format. This is not aggressive post-training quantization. The 4-bit precision is part of the original model design and should be treated as the official precision, not an approximation.

With 2.8 trillion parameters at 4 bits per parameter:

2.8 trillion × 4 bits ÷ 8 = 1.4 TB

In other words, the weights alone require about 1.4 TB of storage.

During real inference, you also need to account for:

• Activations
• Runtime overhead
• KV cache or state space
• Communication buffers

For a single-user request, total GPU memory demand typically rises to around 1.5 TB. This figure forms the basis for the rest of the analysis.

3. Minimum Hardware for “It Runs”

Since roughly 1.5 TB of GPU memory is required, a single GPU cannot meet the need. Common high-end data center GPUs generally offer between 80 GB and 192 GB of memory per card, so multi-GPU parallelism is mandatory.

A rough capacity-based estimate is:

Total required GPU memory ÷ per-GPU memory = minimum number of GPUs

But there is a commonly overlooked issue: some previous-generation or current-generation GPUs cannot natively execute this 4-bit floating-point format. They must dequantize weights at runtime, temporarily converting them to higher precision before computation.

What does that mean in practice? The model may fit in memory, but actual inference efficiency drops significantly. The model might load successfully, yet generation speed could be too slow to be practically usable.

A more realistic minimum configuration is usually a single node with 8 data center GPUs that natively support low-precision computation. This is not a recommendation from any specific cloud vendor. It is the mainstream approach used by common inference frameworks when deploying trillion-parameter models of this class.

The conclusion is straightforward: “It runs” is already a server-level project, not something achievable on an ordinary workstation or a single high-performance tower machine.

4. What Smooth Single-User Performance Requires

If the goal is to move from “it produces output” to “around 25 tokens per second for a single user,” the hardware floor does not drop. The starting point remains a single node with 8 current-generation data center GPUs.

The reason is direct: this is first and foremost a capacity problem, not a compute problem. The 1.4 TB of weights must remain resident in GPU memory. Having only one user does not allow you to shrink the deployment to two or three cards.

Does 8 GPUs guarantee a stable 25 tokens per second?

That depends on several variables:

• Input context length
• Output length
• Whether long-context state caching is enabled
• Communication topology and inter-GPU bandwidth
• Scheduling implementation in the inference framework

Without publicly available benchmarks, it is impossible to say that a given setup is guaranteed to hit that target. However, based on common performance ranges for single-stream large-model inference, tens of tokens per second is a reasonable objective. Before going live, you must run load tests under realistic context lengths rather than relying on theoretical numbers.

5. Multi-User Concurrency: Memory and Throughput Are Different Problems

Assume the goal is 10 concurrent users, each expecting around 25 tokens per second. The aggregate output rate would be:

10 × 25 tokens/s = 250 tokens/s

A key point here is that increased concurrency does not necessarily require more GPU memory.

Why? Because a single 8-GPU node often has enough memory capacity to hold multiple inference requests simultaneously. How many depends on each request’s context length. If contexts are not at the million-token level, a single node can theoretically handle far more than 10 simultaneous requests.

Therefore, 10 concurrent users should not automatically force you to provision a second server. The real issue is whether the system can sustain the target rate for every stream when all 10 requests are generating at the same time.

This is a throughput and scheduling problem, not a memory capacity problem. Factors that affect it include:

• Batching strategy
• Token-level scheduling granularity
• GPU memory bandwidth utilization
• Operator fusion quality
• Overlap between communication and computation

These must be measured through actual load testing. They cannot be inferred from memory capacity alone.

6. Cost Estimation: Cloud Hourly Pricing as the Baseline

Hardware at this level is intended for data centers. It is nearly impossible for individuals to purchase directly. The practical route is renting by the hour through a cloud platform.

Using publicly listed GPU instance pricing from one cloud provider as an example, a 12-month reserved price for a current-generation data center GPU is approximately:

USD 4.76 per GPU per hour

An 8-GPU node would then be:

4.76 × 8 = USD 38.08 per hour

This price covers only the bare GPU instance. It does not include:

• Storage
• Network egress
• Images and snapshots
• Technical support
• Bandwidth charges

On a long-term or annual reservation basis, a single node can reach hundreds of thousands of dollars per year. This explains why very few teams choose to build fully self-hosted environments for models of this class. Most either rent on demand in the cloud or call a hosted model service directly.

7. Practical Recommendations for Developers

If your team genuinely needs to build its own inference environment, proceed in the following order:

1. Confirm precision and format compatibility first
   Verify whether the target GPUs natively support the model’s weight format. If dequantization is required, assess the performance penalty before committing.

2. Use capacity as the first filtering criterion
   Calculate total weight size first, then work backward to determine the number of GPUs and per-GPU memory. No matter how powerful the compute is, if the weights do not fit, nothing else matters.

3. Distinguish between “it loads” and “it serves”
   Loading successfully is not the same as being production-ready. Define clear service-level targets, such as P50 and P95 latency, before running load tests.

4. Test throughput separately for concurrent scenarios
   Sufficient GPU memory does not guarantee sufficient throughput. Multi-stream token scheduling efficiency must be validated with realistic workloads.

5. Use cloud hourly pricing as the budget baseline
   Before deciding to self-host, use the hourly price of an 8-GPU cloud node as the reference point. Only then evaluate whether long-term self-hosting has a cost advantage.

8. Conclusion

Local deployment of trillion-parameter sparse models has fundamentally shifted from “choosing a GPU” to “planning server nodes.” Weight capacity is the first hard constraint, followed by format compatibility, and only then by compute and throughput.

If the goal is merely to validate feasibility, the minimum is still a single 8-GPU data center node. If the goal is smooth single-user performance, the hardware starting point is the same, but strict performance testing becomes essential. If you scale to multiple concurrent users, the focus shifts from GPU memory capacity to scheduling and throughput tuning.

Before taking action, establish a cost baseline using cloud hourly pricing. Only by evaluating capacity, performance, concurrency, and budget in the same framework can you avoid the common pitfall of buying hardware that loads the model but cannot serve it.