Hardware Costs and Tiered Deployment Strategies for Trillion-Parameter MoE Models

1. Activated Parameters vs. Deployment Parameters in MoE

When encountering specifications like “16 experts activated per token,” a common misconception is that memory only needs to be provisioned for those 16 experts. This is a fundamental error.

In a Mixture of Experts (MoE) architecture, the routing module selects a subset of experts to participate in the computation based on the input token’s features. However, during inference deployment, all experts must reside in memory or high-speed storage, ready to be called upon by the router. Therefore:

• Activated expert count dictates the computational power required for a single forward pass.
• Total expert count determines the lower bound for VRAM and storage requirements.

Consider a model with 2.8 trillion total parameters, 896 experts, and 16 activated experts per token. While the activation ratio is roughly 1.79%, this metric is only relevant to compute power and offers no relief for memory planning. Regardless of how many experts are activated during a single routing event, all 896 experts must be loaded into VRAM or high-speed accessible storage for deployment.

2. How Much VRAM Do 2.8 Trillion Parameters Actually Require?

2.1 From Parameter Count to Raw Weight Size

Assuming each parameter is stored at 2 bytes (FP16/BF16 precision):

2.8 Trillion × 2 bytes = 5.6 TB

If 8-bit quantization is applied:

2.8 Trillion × 1 byte = 2.8 TB

If using 4-bit quantization, the theoretical lower bound for raw data is:

2.8 Trillion × 0.5 bytes = 1.4 TB

However, real-world deployment requires significantly more than this.

2.2 Actual Overhead of Quantization Formats

4-bit quantization does not strictly mean each parameter consumes exactly 0.5 bytes. Take MXFP4, a quantization format that includes scale factors: for every 32 4-bit elements, an 8-bit scale value is attached. Based on standard industry specifications, the equivalent bit width per weight element is approximately 4.25 bits.

Recalculating:

2.8T × 4.25 bits ÷ 8 = 1487.5 GB

This accounts only for the weights. It does not include:

• Non-4-bit tensors, such as precision-sensitive layers;
• Metadata and routing tables;
• Memory alignment fragmentation;
• Runtime workspace;
• KV cache, which scales directly with concurrency and sequence length;
• Communication buffers.

Therefore, a seemingly 1.4 TB model realistically requires over 1.8 TB of VRAM to be safely provisioned. When factoring in the KV cache for a production environment, exceeding 2 TB is entirely expected.

2.3 Reference: Deploying Known-Scale Models

To contextualize these numbers, we can look at a widely deployed 671B parameter MoE model:

• Under FP8 precision, main parameters plus multi-token prediction parameters are roughly 685 GB;
• With 4-bit quantization, weights take about 335 GB, activations about 37 GB, totaling around 400 GB;
• This can run on 8 x 80GB accelerator cards (640 GB total VRAM).

The model discussed in this article is over four times that size. Scaling proportionally, the weights alone require 1.4 TB to 1.5 TB of VRAM. A full deployment requires a super-node of at least 64 or more high-performance accelerators. Official deployment scale recommendations corroborate this assessment.

3. Communication Bottlenecks: The Hidden Cost

Even with sufficient VRAM, the model won’t necessarily run fast. The true challenge of large-scale MoE inference lies in all-to-all communication.

After a token enters the model and passes through the router, it must be dispatched to the physical computing device hosting the target expert. With 896 experts distributed across multiple cards, every forward pass involves:

1. Routing decisions to determine activated experts;
2. Sending token activations to the target experts’ cards;
3. Computation by the target experts;
4. Aggregating the results back from the cards.

Each step involves cross-card communication. If the communication domain is small, data requires multiple hops, causing latency to accumulate rapidly. This ultimately leaves compute units idling while waiting for data, resulting in extremely low utilization.

There are generally two solutions to this:

• Expand the single communication domain: Utilize NVLink or equivalent high-speed interconnects to include as many accelerators as possible within a single low-latency communication domain, minimizing cross-domain data transfers.
• Optimize expert placement strategies: Distribute high-frequency experts based on routing frequency to alleviate communication pressure on hotspot cards.

This is why production-grade deployments demand super-nodes of 64+ cards, rather than patching together numerous low-to-mid-range cards with low-bandwidth interconnects.

4. Three Deployment Tiers and Cost Estimations

Tier 1: Minimum Viable Run (Performance Agnostic)

The goal here is to verify weight integrity, execute the inference pipeline, and conduct small-scale functional tests.

The minimum requirement is the storage lower bound for weights. Using MXFP4 quantization, weights plus scales require about 1487.5 GB. Temporary memory during model loading, a minimal runtime KV cache, and communication buffers must also be accounted for.

Hardware Estimation:

• Requires at least 1.7 TB to 1.8 TB of total VRAM;
• Using 80GB cards, theoretically requires 22+ cards;
• Using 141GB or higher capacity devices can proportionally reduce the card count;
• CPU memory must significantly exceed the raw model size for pre-processing and weight staging;
• For extreme cost-cutting, weights can be partially offloaded to CPU memory or NVMe SSDs, but speed will degrade to practically unusable levels.

Expected Experience: Output speeds might be measured in seconds or tens of seconds per token. Suitable only for pipeline validation, not for interactive scenarios.

Tier 2: Single-User Interaction at 25 tokens/s

A speed of 25 tokens/s provides a reasonably smooth reading experience for humans. This tier represents the threshold for “usable” interactive AI.

Achieving this speed requires two simultaneous conditions:

First, ample VRAM with weights fully residing on the GPU.
CPU offloading or SSD swapping is not viable, as communication latency will instantly breach the speed target.

Second, sufficiently low communication latency.
Inference latency consists of compute time and communication time. At a scale of 2.8 trillion parameters, even with only 16 experts activated, the compute volume per token is massive. If all-to-all communication frequently crosses nodes, communication time can exceed the compute time itself.

Reverse-engineering from 25 tokens/s, the total latency budget per token is 40ms. Compute and communication must heavily overlap, demanding a high sustained effective compute utilization rate.

Hardware Estimation:

• Recommend provisioning 1.3 to 1.5 times the full weight VRAM to reserve space for KV cache and communication buffers;
• Using 80GB cards, 24 to 32 cards are recommended;
• Alternatively, use higher VRAM density devices to reduce card count, provided interconnect bandwidth doesn’t become a bottleneck;
• All cards should be within the same high-speed interconnect domain to avoid cross-node communication.

Cost Range: This represents a small enterprise-level investment. Hardware procurement alone sits in the range of hundreds of thousands of dollars, excluding power, cooling, datacenter space, and operations.

Tier 3: 10 Concurrent Users at 25 tokens/s Each

This tier requires 10 simultaneous users, each receiving 25 tokens/s. The system’s total output throughput must reach 250 tokens/s.

This is not a simple 10x scale-up of Tier 2, as concurrency introduces three additional pressures:

First, linear KV cache expansion.
Each concurrent request occupies its own KV cache. If the sequence length per request is 8K, 10 concurrent requests mean KV cache usage expands roughly 10x. VRAM provisioning must reserve a substantial proportion for KV cache, potentially reaching hundreds of gigabytes.

Second, latency fluctuations from batch scheduling.
Concurrent requests won’t arrive or finish simultaneously. The scheduler must dynamically batch requests, balancing throughput and latency. Larger batches improve compute utilization but increase queueing latency for individual requests. Maintaining a 25 tokens/s experience requires end-to-end latency per token to stay under 40ms, placing heavy demands on scheduling strategies.

Third, intensified communication contention.
Multiple requests triggering all-to-all communication simultaneously cause link contention. If experts are unevenly distributed, certain cards may become hotspots, further amplifying latency.

Hardware Estimation:

• Based on Tier 2, increase total VRAM by at least 30% to 50%, primarily for KV cache;
• Recommend 32 to 48 x 80GB accelerator cards, or an equivalent higher VRAM configuration;
• If per-card VRAM density doubles, card count can be reduced, provided the interconnect topology maintains low latency;
• Requires a specialized inference serving framework supporting continuous batching, prefix caching, and similar features;
• Requires accompanying monitoring and elastic scaling capabilities to handle concurrency fluctuations.

Cost Range: The hardware investment for this tier enters the budget territory of large enterprises or specialized AI service teams. Organizations must also weigh building their own datacenter against using elastic cloud GPU instances.

5. Pragmatic Alternatives: On-Premises vs. Cloud

For most developers and small-to-medium businesses, building a full inference environment for a 2.8 trillion parameter model is not economically viable. More pragmatic paths typically include:

5.1 Cloud Inference Services

Major cloud providers usually offer model APIs billed per token. Users don’t need to worry about VRAM, card counts, or communication topologies—focusing only on two metrics: price per token and time to first token.

For teams with low call frequencies or those needing rapid validation of business scenarios, this is often the most cost-effective method. Pay-as-you-go models prevent idle resource waste.

5.2 Deploying Distilled or Quantized Versions

If the business scenario doesn’t require the absolute peak capabilities of the model, consider:

• Selecting smaller parameter models from the same series;
• Applying more aggressive quantization schemes, such as 3-bit or mixed-precision;
• Reducing KV cache pressure through context truncation.

These methods can drastically lower hardware barriers within a tolerable margin of accuracy loss.

5.3 Hybrid Architecture: Cloud Primary, Local Secondary

For sensitive data or high-frequency scenarios, a hybrid approach can be adopted:

• Route high-frequency, low-latency requests through cloud APIs;
• Deploy a compressed, medium-sized model locally for data-sensitive requests requiring on-premises processing;
• Use a unified gateway for routing and distribution.

This strategy balances cost, privacy, and performance.

6. Conclusion

Deploying a 2.8 trillion parameter MoE model is fundamentally a multi-constraint optimization problem involving VRAM, communication bandwidth, compute utilization, and cost.

Three key takeaways:

• VRAM lower bounds are dictated by total parameters, not activated ones. All 896 experts must remain resident; under MXFP4 quantization, the weights alone approach 1.5 TB.
• Communication topology is the lifeline of inference speed. All-to-all communication latency directly dictates performance. High-speed super-node interconnects are mandatory, not optional.
• The three demand tiers correspond to vastly different investment levels. From “minimum viable run” to “single-user smooth interaction” to “10-concurrent production,” hardware costs span an order of magnitude.

Prior to the official release of weights and technical reports, all figures in this article are estimates based on publicly available information. Actual deployments require verification through real-world benchmarking. For most teams, directly using cloud inference services or deploying compressed, mid-sized versions is a far more rational technical decision than building a full-scale on-premises environment.