MoE Model VRAM Estimation: Why Fewer Activated Parameters Do Not Mean Less Memory

    This article explains how to estimate VRAM requirements for large Mixture-of-Experts (MoE) models. It distinguishes activated parameters from total weights, identifies often-overlooked model components, breaks inference memory into weights, KV cache, and runtime overhead, compares weight sizes across numerical precisions, and outlines deployment options and pre-deployment checks for cases where a single GPU cannot hold the full model.

    1. Start With the Conclusion

    A common mistake when running large Mixture-of-Experts (MoE) models on consumer GPUs is treating “parameters activated per token” as “VRAM usage.” These are different concepts. Activated parameters determine the compute workload for each step, while total weight size determines how much data must be loaded into VRAM or kept in quickly accessible memory. For MoE models, total weights are usually much larger than activated parameters, so capacity planning must be based on total weights.

    2. Why VRAM Can Still Be Tight When Few Activated Parameters Are Used

    MoE architectures split the feed-forward network into multiple experts, and each token is routed to only a few of them. This significantly reduces compute per step and can improve inference speed.

    However, routing is dynamic. Different tokens may hit different experts, and the inference process cannot predict in advance which experts will be used in a given layer. Therefore, the weights of all experts must remain available at all times. Even if only a small subset of parameters is used in a single step, the full set of weights still needs to reside in VRAM or in memory with acceptable access latency.

    Beyond expert networks, several other components are easy to overlook:

    • Embedding tables: Larger vocabularies and higher hidden dimensions increase memory usage. Some models also use additional embedding structures, and these count toward the weights as well.
    • Multi-Token Prediction (MTP) modules: Used for training assistance or speculative decoding. If this module is retained during deployment, its parameters must also be loaded.
    • Vision encoders: Parameters that exist independently in multimodal models. Even if only text requests are processed, they still occupy space as long as the full model is loaded.

    Therefore, estimation should sum the parameters of all components to obtain the complete weight footprint.

    3. VRAM Usage Has Three Main Parts

    Inference VRAM usage can be broken down into the following three items.

    1. Model weights: Directly related to total parameter count and numerical precision. This is a fixed cost.
    2. KV cache: Attention keys and values cached during inference. It grows linearly with context length and the number of concurrent requests. The estimation formula is:

      KV cache bytes = 2 × layers × KV heads × head dimension × sequence length × batch size × bytes per element

      The factor 2 means both Key and Value are cached. When context is very long or concurrency is high, KV cache can become the main bottleneck. It must be calculated separately rather than focusing only on weights.

    3. Runtime overhead: Includes activation tensors, framework-internal buffers, communication buffers, and similar items. In engineering practice, it is common to reserve a certain margin to avoid out-of-memory errors at peak load.

    4. Numerical Precision Determines Weight Size

    For the same model, changing the numerical precision can multiply or divide weight memory usage. Common precisions use the following bytes per parameter:

    Precision Bytes per Parameter Notes
    BF16 / FP16 2 Common baseline precision for training and inference
    FP8 / INT8 1 Requires hardware and inference framework support; precision loss must be evaluated
    INT4 About 0.5 Quantization scaling factors and other metadata overhead must also be added

    The following script estimates weight usage at different precisions. The parameter scale in the example is an assumed value for calculation and is not measured data from a specific model.

    1. def weight_size_gib(total_params_billion: float, bytes_per_param: float) -> float:
    2. """Calculate weight usage in GiB from total parameters in billions and bytes per parameter."""
    3. total_bytes = total_params_billion * 1e9 * bytes_per_param
    4. return total_bytes / (1024 ** 3)
    5. if __name__ == "__main__":
    6. # Assume total parameters are 125B; for demonstration only
    7. for name, nbytes in [("BF16", 2), ("FP8", 1), ("INT4", 0.5)]:
    8. print(f"{name}: {weight_size_gib(125, nbytes):.1f} GiB")

    The results are approximately: BF16 232.8 GiB, FP8 116.4 GiB, and INT4 58.2 GiB. Even with FP8, the weights alone exceed 100 GiB. Adding KV cache and runtime margin, the total far exceeds the VRAM capacity of a single consumer GPU.

    Quantization can therefore compress weights by half or more, but it only changes the numerical representation, not the total parameter count. Lower precision may also degrade model output quality, so it must be validated on your own business evaluation set.

    5. Deployment Strategies When a Single GPU Cannot Hold the Model

    After identifying the capacity gap, common approaches include the following.

    1. Choose a smaller model: When quality requirements allow, this is the lowest-cost and lowest-risk option.
    2. Quantization: Convert weights to lower-precision formats such as FP8 or INT4. The benefit is clear, but accuracy, format-following rate, and other metrics on key tasks should be compared before and after quantization.
    3. Multi-GPU parallelism: Split weights across multiple GPUs using tensor parallelism or expert parallelism. If inter-GPU interconnect bandwidth is insufficient, communication can become a latency bottleneck, so testing should be done in advance.
    4. Memory offloading or tiered loading: Place some weights in host memory or on disk and load them into VRAM on demand. This can alleviate capacity issues, but throughput and latency will degrade noticeably. It is more suitable for offline batch processing.
    5. Use cloud instances with large VRAM on demand: Suitable for occasional tasks or stress testing. Pay attention to billing methods, data transfer, and compliance requirements.

    When choosing a solution, first clarify latency targets, throughput requirements, and quality baselines, then compare options item by item. Simply pursuing “it can run” often leads to latency or stability problems after launch.

    6. License and Compliance Checks

    The license terms for model weights may differ from the license of the code repository. Commercial use, redistribution, and modification followed by republication must each be confirmed separately. Read the license text on the model description page and focus on the following:

    • Whether commercial use is permitted;
    • Whether redistribution is permitted and what attribution requirements apply;
    • Whether there are restrictions on use cases or regions;
    • Whether the original license text must be retained.

    It is recommended to archive the license file version and retrieval date for later audits.

    7. Pre-Deployment Checklist

    • Count the total parameters of all components, including embedding layers, MTP modules, and vision encoders.
    • Determine the target precision and calculate weight usage.
    • Estimate the KV cache upper bound based on maximum context length and target concurrency.
    • Reserve runtime overhead margin and measure VRAM under peak load.
    • Test accuracy, latency, and throughput with real business samples to confirm the quality baseline.
    • Review the license terms for model weights and confirm commercial use and distribution conditions.