Trillion-Parameter MoE LLM Deployment: How to Estimate VRAM for Local Inference

    This guide explains how to estimate the VRAM needed to deploy trillion-parameter MoE (Mixture-of-Experts) LLMs locally. It shows why VRAM is determined by total parameters rather than activated parameters. It then walks through the memory math for a 2.8T-parameter model trained with QAT (4-bit microscaling weights plus 8-bit activations), which reaches roughly 1487.5 GB for weights and scale factors. A 670B-parameter comparison puts that figure in context. You will also find capacity-planning guidance and practical strategies for quantization, mixed precision, parallelism, KV cache management, and cost evaluation.

    1. Total vs. Activated Parameters: Which One Determines VRAM?

    Take a MoE (Mixture-of-Experts) model with roughly 2.8 trillion total parameters. It follows a sparse activation approach: its expert layers are split into 896 independent experts, and each token is routed to and activates only 16 of them at inference. The expert activation ratio is therefore about 16/896 ≈ 1.79%.

    A common misconception needs clearing up here. The 1.79% figure is an activation ratio at the level of expert count. It is not a strict activation ratio at the parameter level. There are two reasons:

    • Each activated expert carries a full set of parameters. Activation happens at the granularity of entire experts, not scattered individual parameters.
    • Beyond the expert layers, non-expert parameters such as attention, embeddings, and normalization are fully involved in computation for every token.

    For these reasons, you cannot multiply 1.79% by the total parameter count to estimate actual compute.

    The more important conclusion is that MoE VRAM usage should be calculated from total parameters, not activated parameters. Your storage must hold the weights of all 896 experts. During inference, the router then dynamically selects 16 of them for each token’s computation. The activation ratio affects compute consumption (FLOPs) and inference throughput, but it does not reduce the storage requirement, which sets the VRAM floor. This is the essence of sparse models: they save compute, not memory.

    1. Backbone Architecture: How Attention Design Affects Deployment

    The model’s backbone combines several components:

    • A novel differential attention mechanism
    • An attention residual structure
    • Gated multi-head latent attention

    These designs aim to improve training stability and inference efficiency at very long context lengths. Examples include compressing the KV cache through low-rank latent representations and improving information flow in deep layers through residual connections.

    It is worth stressing that the backbone design affects compute graph complexity, the shape of the KV cache, and precision layout. It does not change the storage floor for the weights themselves. No matter how the attention mechanism is optimized, as long as all expert weights must stay resident in memory, the first-principles formula for VRAM estimation still comes down to total parameters × bytes per parameter.

    1. QAT and Calculating the VRAM Floor

    The model used quantization-aware training (QAT) starting at the supervised fine-tuning (SFT) stage:

    • Weights: 4-bit microscaling floating-point format
    • Activations: 8-bit floating-point format

    This brings two benefits. Weight size is compressed to near the theoretical lower bound without significant accuracy loss, while activations remain stable at higher precision.

    Start with the raw data floor. Converting 2.8T parameters to 4-bit gives 2.8T × 4 bit ÷ 8 ≈ 1400 GB. This is the absolute lower bound, with no additional overhead included.

    However, in microscaling floating-point formats, weights are not pure 4-bit. Every 32 4-bit weight elements share one 8-bit scale factor. Per parameter, that works out to 4 bit + 8 bit/32 = 4.25 bit. The total for weight elements plus scales is therefore:

    1. 2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

    This 1487.5 GB figure still does not include the following:

    • Non-4-bit tensors: Embedding layers, normalization parameters, and routing/gating networks are typically kept at higher precision.
    • Metadata and alignment padding: Aligning tensors to boundaries adds extra fragmentation.
    • Runtime workspace: Temporary memory for attention computation, expert routing, communication buffers, and so on.
    • KV cache: Grows linearly with context length and concurrent batch size, making it a major memory consumer in long-context scenarios.

    So 1487.5 GB is only the theoretical total for weights plus scales. In real deployments, you must add a substantial margin on top of it.

    1. Comparison: The VRAM Bill for Models of Different Sizes

    For an intuitive sense of scale, consider a MoE model with about 670B parameters:

    • FP8: About 670B main parameters plus roughly 14B parameters in a multi-token prediction module, for about 684B parameters in total. At 1 byte per parameter, that is about 685 GB.
    • 4-bit: Using another common 4-bit quantization scheme, weights take about 335 GB. Add roughly 37 GB for activations and workspace, for about 400 GB in total.

    A 400 GB footprint requires data-center-class accelerators with around 80 GB of VRAM per card. In practice, you would typically need 8 cards, about 640 GB of VRAM in total, to leave a safe margin.

    The comparison is telling. Moving from about 685 GB (8-bit floating point) to about 400 GB (4-bit) nearly halves VRAM usage, yet it still far exceeds the capacity of a single card. This is exactly why very large models require multi-GPU setups.

    1. Deployment Practices and Capacity Planning

    For a model at the 2.8T scale, weights plus scale factors already reach about 1487.5 GB. Once KV cache and runtime overhead are added, VRAM requirements generally need to be planned at around 1.6–2 TB or more. Consider the following when designing your deployment:

    1. Quantization choice: Prefer schemes that completed quantization-aware training beforehand, such as 4-bit weights plus 8-bit activations. Avoid ad-hoc quantization at inference time, which causes accuracy and performance losses.

    2. Mixed precision: Keep sensitive modules such as routing/gating networks, embeddings, and normalization at higher precision. Apply low-bit compression only to expert weights, balancing memory usage against accuracy.

    3. Parallelism strategy: Combine tensor parallelism (splitting the weights of individual layers) with expert parallelism (distributing the 896 experts across devices) to ease per-GPU load. Where possible, place frequently activated experts close together to reduce communication overhead.

    4. KV cache management: Use context paging, KV quantization, and request-level reuse to keep KV growth under control. For long-context scenarios, reserve ample VRAM in particular.

    5. Tiered loading and offloading: When memory is tight, move less frequently used expert weights to host memory or storage and load them on demand. This trades memory for latency, since it introduces data-transfer delays.

    6. End-to-end cost accounting: VRAM is only one part of the cost. Interconnect bandwidth, host memory capacity, network topology, and tokens-per-second throughput should also be evaluated together to get a realistic picture of deployment return on investment.

    7. Summary

    The core challenge in deploying very large MoE models locally is the mismatch between sparse compute and dense storage. A low activation ratio saves compute, but the need to hold all expert weights determines the VRAM floor.

    Using a model of about 2.8T parameters with 4-bit weights and 8-bit activations as an example:

    • The theoretical floor for weights plus scale factors is about 1487.5 GB.
    • Once KV cache and runtime overhead are added, plan for around 2 TB of VRAM.

    Grasping the first principle that total parameters determine VRAM while the activation ratio determines compute is the starting point. Combining it with engineering practices such as quantization choices, mixed precision, expert parallelism, and KV cache management is what allows you to estimate and plan hardware investment accurately.