Local Deployment of Trillion-Parameter MoE Models: VRAM Accounting, Hardware Sizing, and Cost Estimation

For Mixture-of-Experts (MoE) models in the trillion-parameter range, the first question you face when deploying locally is not “how do I get it running?” but rather “how much VRAM do I need and how much will it cost?” Many teams default to estimating resources based on activated parameters—reasonable for dense models but seriously misleading for MoE architectures. Using a reference model with 2.8 trillion total parameters and fine-grained expert routing, this article walks through VRAM accounting, hardware configuration, the software stack, and budget estimation, and provides a reusable derivation method so you can verify each item once weights become publicly available.

1. Get the Definitions Right: Total Parameters Govern VRAM, Activation Ratio Governs Compute

Two metrics that are easy to conflate in MoE resource estimation:

• Total Parameters: The sum of all expert weights across the model. This determines how much storage and VRAM the weights occupy.
• Activated Parameters: The number of parameters a single token actually traverses during a forward pass. This drives compute cost and inference latency.

Take the reference configuration: 2.8T total parameters, 896 experts, 16 experts activated per token. The expert activation ratio is roughly 16 ÷ 896 ≈ 1.79%. Important caveat: the expert activation ratio is not the same as the overall parameter activation ratio. Non-expert components—attention backbone, embedding layers, the routing network—participate in every token’s computation, so the true activated-parameter ratio is somewhat higher.

The deployment takeaway is straightforward: all 896 experts must reside in memory at all times, while the routing network dynamically dispatches 16 per token. VRAM footprint is based on total parameters; activation ratio only affects per-token compute. Keeping these two calculations separate is the prerequisite for everything that follows.

2. Quantization Format and the VRAM Lower Bound

The reference model uses quantization-aware training from the supervised fine-tuning stage onward: MXFP4 for weights and MXFP8 for activations. MXFP4 is a microscaling format defined in the OCP specification: every 32 four-bit elements share a single 8-bit scale factor. The average bit-width per weight element is therefore:

Bits per 32 elements = 32 × 4 bit + 8 bit = 136 bit
Average bits per element = 136 ÷ 32 = 4.25 bit

From this we derive the bare weight data lower bound:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1,487.5 GB

A naive 4-bit calculation yields roughly 1,400 GB—just the theoretical floor. Accounting for scale factors pushes it to about 1,487.5 GB. This number does not yet include:

1. Non-4-bit tensors (embedding layers, normalization layers, the routing network, typically kept at higher precision);
2. Tensor metadata and alignment padding;
3. Runtime workspace (temporary buffers, communication buffers);
4. KV Cache, which grows linearly with concurrency and context length and becomes non-trivial in long-context scenarios.

For perspective: a ~671B-parameter MoE model under FP8 occupies roughly 685 GB for main and multi-token prediction weights; even at 4-bit quantized weights it still needs about 335 GB plus ~37 GB for activations—roughly 400 GB total—to fit on 8 × 80 GB accelerator cards (640 GB aggregate) with headroom. Scaling total parameters by more than 4× raises the minimum VRAM threshold accordingly.

3. Hardware Configuration: Why a Supernode Is Non-Negotiable

Having enough VRAM is only step one. The performance bottleneck for large-scale MoE inference is usually communication, not compute.

With 896 experts distributed across multiple accelerator cards, each token’s activated values must be shipped to the card hosting the target expert after routing, then shipped back after computation. This is a classic all-to-all communication pattern. The finer the expert sharding and the more cards involved, the greater the communication volume and latency—directly eroding compute utilization.

The recommended deployment topology is therefore a supernode of 64 or more accelerators connected via high-bandwidth, low-latency inter-GPU links (e.g., NVLink or equivalent) so that inter-expert activation movement stays within a single communication domain. Using standard Ethernet for cross-node connections typically makes all-to-all bandwidth and latency the primary bottleneck, requiring additional optimizations (hierarchical all-to-all, expert replication, topology-aware routing) to approach usable throughput.

A rough capacity-planning reference:

Item	Estimation Basis
Weight VRAM	Total parameters × 4.25 bit ÷ 8, plus non-quantized tensors
KV Cache	Concurrency × context length × number of layers × hidden dimension × bytes per precision
Activations & Workspace	Peak-batch activation tensors, typically a fraction of weight size
Communication Buffer	All-to-all peak message volume
Headroom	15%–25% recommended for fragmentation and spikes

4. Software Stack and Deployment Path

On the software side, four things must be addressed: quantization format support, expert parallelism sharding, communication scheduling, and serving infrastructure.

# Pseudocode: expert parallelism sharding configuration
config = {
    "model_path": "/models/moe-2.8t-mxfp4",
    "quantization": "mxfp4_weight_mxfp8_act",
    "parallel": {
        "expert_parallel": 64,     # experts distributed across 64 ranks
        "tensor_parallel": 8,      # intra-expert tensor sharding
        "pipeline_parallel": 2,
    },
    "kv_cache": {
        "max_batch_size": 32,
        "max_seq_len": 32768,
        "dtype": "fp8",
    },
    "comm": {
        "all_to_all_backend": "high_speed_interconnect",
        "overlap_with_compute": True,
    },
}

Key practical points:

• Validate quantization kernels first: MXFP4’s microscaling requires runtime kernel support. Before deployment, confirm that your inference framework has complete load and compute paths for this format—avoid the “loads but won’t run” trap.
• Align expert parallelism with the interconnect domain: Keep expert parallelism within a single high-speed domain wherever possible; use cross-domain aggregation only where necessary.
• Overlap communication with compute: Pipelining all-to-all transfers with expert computation is a primary lever for throughput.
• Plan KV Cache separately: Long contexts and high concurrency drive VRAM usage sharply. Account for KV Cache independently from weight VRAM and consider paged management if needed.

5. Cost Estimation: Separate One-Time and Recurring Spend

Total cost of local deployment has four components, all of which matter:

1. Accelerator procurement: The largest line item. Starting at 64+ cards, per-card pricing varies widely by interconnect spec and memory capacity—quote based on actual selection. Higher throughput or longer contexts may require more cards.
2. Server and supernode infrastructure: High-speed interconnect switches, racks, power delivery, and cooling. High-density racks often have special power and liquid-cooling requirements that are commonly underestimated.
3. Facility and operations: Power capacity, cooling, network bandwidth, and ongoing SRE staffing. Trillion-parameter inference services demand high reliability and dedicated operational investment.
4. Software and adaptation costs: Inference framework adaptation, quantization kernel validation, and performance tuning—typically measured in person-months.

A practical decision framework: calculate the VRAM floor first, then the communication constraints, then the budget. If the VRAM floor already exceeds what you can procure, or the communication domain cannot accommodate single-domain all-to-all, full local deployment is not economically viable. At that point, on-demand cloud compute or a managed API becomes the more realistic path.

6. Verification Checklist

All VRAM, hardware, and performance figures in this article are derived from publicly available architecture information and are estimates. Once weights and technical reports are officially released, verify each item against this checklist:

• Actual weight file sizes and quantization format—confirm the 4.25-bit average bit-width assumption.
• Precision and proportion of non-quantized tensors (embeddings, normalization, routing).
• Per-card VRAM capacity and the actual expert-parallel sharding plan.
• Measured all-to-all bandwidth and latency within the target interconnect domain.
• KV Cache usage curves across different batch sizes and context lengths.
• End-to-end throughput and time-to-first-token benchmark results.

Once these six items are validated, your estimates become measured values and the deployment plan moves from “theoretically feasible” to “engineering-ready.”