Compute and Hardware Budgeting for Local Deployment of Trillion-Parameter MoE Models

Over the past year, the parameter scale of open-source large language models has steadily progressed from the hundreds of billions into the trillion-parameter sparse Mixture-of-Experts (MoE) era. For developers, the open-sourcing of a model is merely the starting point. The truly棘手 problem lies in: How many machines and how much budget are actually required to run such a behemoth? This article does not review any specific product. Instead, it takes a generic trillion-parameter sparse MoE model as an example to provide a reusable evaluation methodology for local deployment.

1. Three Fundamental Properties of the Deployment Target

Before discussing the budget, we must clarify the resource characteristics of the model itself. Any hardware estimation made without considering the architecture is fundamentally flawed. For a typical trillion-level sparse MoE model, three dimensions require attention:

1. Total Parameters vs. Activated Parameters

A trillion-level MoE model typically has between 2 trillion and 3 trillion total parameters. However, the parameters actually activated during a single inference pass might only range from tens of billions to one or two hundred billion. This dictates a massive disparity between compute requirements (driven by activated parameters) and memory requirements (driven by total parameters).

2. Weight Precision and Storage Footprint

If weights are released in a native low-precision format (such as a 4-bit floating-point quantization format), the storage requirement is approximately:

Storage Footprint ≈ Total Parameters × Bytes per Parameter
2.8T Parameters × 0.5 Byte ≈ 1.4 TB

Adding overhead for shard metadata, quantization scaling factors, and other artifacts, the actual on-disk size is typically about 10% higher than the theoretical value. This represents the minimum footprint for the weights alone and does not include the KV cache or runtime buffers.

3. Context Window and Attention Structure

A context window supporting millions of tokens means the KV cache will expand rapidly with concurrency and sequence length. If the model employs a hybrid architecture utilizing both linear and sparse attention, the memory growth curve under long contexts will be significantly better than standard full attention, though it still cannot be ignored. The introduction of technologies like Multi-Token Prediction (MTP) will also impact the trade-off between throughput and memory usage.

2. Memory Calculation: Determining the Minimum Number of Machines

Memory is a hard threshold, while compute is a soft constraint. We must calculate memory first.

Assuming a single accelerator card has an effective memory capacity of 80GB (a common specification for current high-bandwidth memory cards), accommodating 1.4TB of weights requires at least:

1.4 TB / 80 GB ≈ 18 Cards

However, this is merely fitting the weights in. Actual runtime operation also requires:

• KV Cache: Dynamically occupied based on concurrent requests and context length, typically reserving 20%~30% of memory;
• Activations and Temporary Buffers: MoE routing, All-to-All communication buffers, etc.;
• Framework Overhead: The inference engine’s own memory management, graph capture, etc.

Taking these into account, a safety factor of 1.5x or more is recommended for weight memory, meaning at least 27 80GB cards are required. Considering that a single machine typically holds 8 cards, this implies a minimum of 4 8-card nodes just to barely load the model. But “able to load” does not equal “able to serve”—without headroom for concurrency, the system will crash under load.

If higher memory capacity accelerator cards are used (e.g., 141GB per card), a single 8-card node can provide over 1.1TB of memory. Combined with weight sharding and CPU offloading, this can reduce the node count, though offloading will significantly sacrifice throughput.

3. Compute Calculation: Determining the Speed

The compute requirement for a sparse MoE is determined by the activated parameters. Assuming a single token activates approximately 50B parameters, calculated using BF16, each token requires roughly:

50B × 2 FLOPs/Parameter × 2 (Forward + Partial Backward, inference is forward only) ≈ 100 GFLOPs/Token

To achieve a throughput of 1000 tokens per second on a single node, 100 TFLOPS of effective compute is needed. Current mainstream accelerator cards offer single-card BF16 peak compute in the hundreds of TFLOPS, which appears sufficient. However, effective compute is typically only 30%~50% of the peak, limited by memory bandwidth, communication latency, and operator efficiency.

Therefore, compute estimation cannot rely solely on peak values. It must be reverse-engineered from the target throughput (Tokens/s) and concurrency. A practical empirical formula is:

Required Cards ≈ (Target Throughput × FLOPs per Token) / (Single Card Effective Compute × Utilization Rate)

4. Communication Bottleneck: The True Threshold for Trillion-Parameter MoE

Expert Parallelism (EP) in trillion-parameter MoE models means that an All-to-All communication must occur during every forward pass. The latency and bandwidth of cross-node communication directly determine whether the system can scale linearly.

Key optimization directions include:

• Topology-aware Expert Placement: Placing frequently communicating experts within the same node or supracell to reduce cross-node traffic;
• All-to-All Latency Jitter Suppression: Fusing dedicated communication operators to merge multiple small messages into larger ones;
• Compute-Communication Overlap: Continuing to process other layers or micro-batches while waiting for expert results.

If communication optimization is neglected, adding more machines may not improve throughput and could actually cause performance degradation due to overhead.

5. Tiered Configuration Recommendations

Synthesizing the above analysis, we can outline three tiers of configuration strategies:

Tier	Target	Approx. Card Count	Description
Verification	Single-node run, functional verification	8~16 cards	Low concurrency, tolerates high latency, can use CPU offloading
Production	Medium concurrency online service	32~64 cards	Requires EP+TP hybrid parallelism, robust communication optimization
High-Performance	High concurrency, low latency	128+ cards	Supracell architecture, full-link tuning

Based on current market prices for high-bandwidth accelerator cards, the hardware cost for a verification-tier configuration falls in the single-digit millions (in equivalent USD), while the production tier crosses into the tens of millions. If higher-spec supracell solutions are adopted, the overall budget will increase further. Specific numbers fluctuate with market conditions, so actual quotes should be used as the standard.

6. The Software Stack is Equally Important

Having the right hardware is only the first step. The inference framework must support:

• Hybrid scheduling of Expert Parallelism and Tensor Parallelism;
• Native loading and operator acceleration for low-precision weights;
• Dynamic batching and PagedAttention-style memory management;
• Tiered KV cache storage for long-context scenarios.

These capabilities dictate how much effective compute can be extracted from the same hardware.

Conclusion

The local deployment of a trillion-parameter MoE model is essentially a tri-constraint problem involving memory, compute, and communication. Memory determines the minimum number of nodes, compute determines the throughput ceiling, and communication dictates scaling efficiency. Before applying for a budget, teams should first define their business targets for throughput and latency, and then calculate iteratively using the methods above to avoid the awkward situation of “buying too much and wasting money, or buying too little and failing to run the model.”