Local Deployment of a Trillion-Parameter MoE Model: Cost and Architecture Estimates

1. Background: Why Trillion-Parameter MoE Deployment Is Not Just About Parameter Count

As of July 2026, the full weights and technical report for a certain trillion-parameter MoE model have not been officially released. The official information available so far covers architecture parameters, API pricing, and a release window. Any current discussion of VRAM usage, inference speed, or hardware selection must be based on publicly disclosed architecture details and industry-standard estimation methods.

The model has approximately 2.8 trillion total parameters, uses a Mixture-of-Experts architecture with 896 experts, activates 16 experts per token, and has an expert activation ratio of roughly 1.79%. It is important to note: the expert activation ratio is not the same as the parameter activation ratio. VRAM usage during MoE inference is primarily determined by the total parameter count, because the inference system must be able to access all expert weights at any time. The activation ratio mainly affects per-token compute cost, not storage requirements. In other words, all 896 experts must be loaded into addressable storage, and at inference time only 16 of them are fetched for computation based on routing results.

This distinction matters because it directly determines the cost estimation logic used throughout this article: VRAM requirements depend on total parameters; compute requirements depend on activated parameters.

2. Three Deployment Tiers: From “It Runs” to “It Serves”

To produce meaningful deployment plans, we first need to break the vague term “local deployment” into three measurable tiers:

Tier 1: It Runs, Speed Doesn’t Matter

The goal is to load all weights into a storage system and have the model generate tokens normally, even at single-digit tokens per second. The focus at this tier is addressable storage capacity and minimal compute capability.

Tier 2: Single-User, 25 tokens/s Output

The goal is to achieve approximately 25 tokens/s output in a single-user scenario. This tier places clear requirements on compute density, memory bandwidth, and inter-GPU communication latency, requiring GPUs and high-speed interconnects working together.

Tier 3: 10 Concurrent Requests, Each at 25 tokens/s

The goal is to sustain at least 25 tokens/s per request when 10 concurrent requests arrive simultaneously. This tier requires linearly scalable batch processing capability and significantly higher aggregate compute, aggregate memory bandwidth, and all-to-all communication bandwidth.

The hardware cost difference across these three tiers can span an order of magnitude. Business requirements must be clarified before any hardware discussion makes sense.

3. VRAM Lower-Bound Estimation: From Quantization Format to Real Footprint

This model introduces quantization-aware training during the SFT stage. The officially recommended inference format is MXFP4 weights with MXFP8 activations. At 2.8 trillion parameters, a naive 4-bit quantization would put the raw data floor at approximately 1,400 GB — but the real footprint is significantly higher.

MXFP4 is not pure 4-bit storage. Per the relevant OCP specification, every 32 4-bit elements carry an 8-bit scale factor. This works out to roughly 4.25 bits of effective storage overhead per weight element. For 2.8 trillion parameters, weights plus scales alone total approximately 1,487.5 GB. This does not yet account for the following additional overhead:

• Non-4-bit tensors (e.g., certain normalization layers, embedding layers, output heads)
• Metadata and routing tables
• Fragmentation from memory alignment
• Runtime workspace and intermediate activations
• KV cache usage

For reference: a 671B-parameter MoE model in FP8 format requires approximately 685 GB for main parameters plus multi-token prediction parameters combined. With 4-bit weights and FP8 activations, weights take about 335 GB and activations about 37 GB — roughly 400 GB total, which is what allows it to run on 8 × 80 GB accelerators. The target model is more than 4× that size, so the VRAM floor for a full deployment is naturally pushed into the ~1.5 TB range.

This means single-card or single-node VRAM capacity is essentially impossible to meet the requirement. Multi-GPU distributed deployment is the only option.

4. Architectural Constraint: Why Supernode Interconnect Is Not Optional

Large-scale MoE inference differs fundamentally from dense model inference in one key way: the all-to-all communication pattern.

In a dense model, the forward computation of each layer can be completed within a single GPU, and inter-GPU communication is concentrated at pipeline-parallel or tensor-parallel split points. In a large-scale MoE model, however, 896 experts are distributed across multiple accelerator cards, and every token at every layer goes through the following process:

1. Routing computation determines which experts are activated
2. The token’s hidden state is sent to the card hosting the corresponding expert
3. The expert forward computation completes on the target card
4. Results are sent back to the original card for aggregation

Since the 16 experts activated per token are distributed across different cards, this send–compute–return cycle repeats at every layer. If inter-GPU interconnect bandwidth is insufficient or latency is too high, communication time completely overwhelms compute time, causing effective compute utilization to drop sharply.

The official recommendation is to use 64 or more accelerators to form a supernode. The core purpose is to enlarge the communication domain through NVLink or equivalent high-speed interconnect, keeping expert activation transfers from crossing slow network boundaries. The supernode here is not a performance optimization — it is a precondition for making inference feasible at all.

5. Viable Deployment Paths

5.1 GPU-CPU Heterogeneous Inference

For Tier 1 (“it runs”), GPU-CPU heterogeneous inference is currently one of the most viable approaches. The core idea exploits the differing computational characteristics of the two component types in an MoE architecture:

• Shared attention portion: Every token passes through it. It is compute-intensive but its VRAM requirements are relatively manageable — well suited to remain on GPU.
• Sparse expert portion: This accounts for the overwhelming majority of storage, but each token only activates a small fraction of it. It can be offloaded entirely to CPU memory.

A large-memory server hosts all expert weights, while the GPU handles only the dense attention computation and high-speed inference for currently activated experts. This approach significantly reduces total hardware cost, at the price of output speed dropping to single-digit tokens per second or even lower. That is entirely acceptable for offline batch processing, model evaluation, and weight verification scenarios.

5.2 Full-GPU Supernode Deployment

For Tier 2 and Tier 3 requirements, a full-GPU supernode is unavoidable. Several key dimensions need to be considered:

VRAM capacity planning: Based on the MXFP4 estimate, weights alone require approximately 1.5 TB of VRAM. At 80 GB per card, the theoretical minimum is 19 cards. But accounting for KV cache, activation workspace, and communication buffers, a practical recommendation is no fewer than 32 cards. The official guidance of 64+ cards provides more generous headroom.

Communication topology design: Distributing 896 experts across 64 cards requires careful planning. Ideally, frequently co-activated experts should reside within the same communication domain to reduce cross-domain traffic. This is a classic graph partitioning problem that can be optimized using an expert co-activation matrix.

Batching strategy: Tier 3 requires 10 concurrent users each at 25 tokens/s, meaning aggregate system output must exceed 250 tokens/s. As batch size grows, KV cache usage increases linearly, so dedicated VRAM space must be reserved for the cache.

6. Cost Estimation Framework

Before weights are officially released, giving precise dollar figures is not rigorous. But an estimation framework can be established so readers can quickly plug in updated numbers as information becomes available.

Tier 1 (Heterogeneous Inference):

• 1 large-memory server: no less than 1.6 TB RAM, 2 TB recommended
• 1–2 × 80 GB accelerators for attention and hot expert computation
• Estimated cost in the hundreds of thousands of dollars range
• Output speed: single-digit tokens/s
• Use cases: offline evaluation, weight verification, academic research

Tier 2 (Single-User, 25 tokens/s):

• A supernode of 32–64 × 80 GB accelerators
• Requires NVLink or equivalent high-speed interconnect
• Estimated cost in the low millions of dollars range
• Output speed: approximately 25 tokens/s
• Use cases: individual research, prototype validation, low-concurrency internal tools

Tier 3 (10 Concurrent × 25 tokens/s):

• 64+ accelerators with refined communication topology design
• Comprehensive upgrades required for aggregate memory bandwidth, all-to-all communication bandwidth, and batch scheduling capability
• Estimated cost in the range of several million to tens of millions of dollars
• Use cases: enterprise-grade internal services, large-scale evaluation and tuning

7. Conclusions and Actionable Recommendations

Local deployment of a trillion-parameter MoE model is not simply a question of “how many cards to buy.” It is a systems engineering challenge spanning storage architecture, communication topology, batch scheduling, and quantization strategy. Before weight files are officially released, developers and enterprises should prepare as follows:

1. Define your business requirement tier clearly. Do not blindly pursue a full deployment. Most validation scenarios can start with heterogeneous inference.
2. Plan for hardware procurement lead times. Delivery cycles for high-end accelerators and supernode-class interconnect equipment are typically long.
3. Watch for community benchmarks after weight release. Use real data to verify the estimation assumptions in this article.
4. Use cloud pay-as-you-go pricing as a baseline. When local deployment cost significantly exceeds cloud API cost, prioritize a hybrid approach.

The core value of local deployment is not “saving money” — it is data controllability, deterministic latency, and deep customization capability. Understanding this is what enables rational technical decisions in the era of trillion-parameter MoE models.