Very Large MoE Models: Local Deployment, VRAM, Precision, and Evaluation Essentials

    Very large MoE models are often described by total parameter count, but local deployment depends on the gap between total parameters and activated parameters. This article explains how to estimate weight and KV cache memory, assess low-bit quantization, and account for sparse attention in long-context deployments. It also outlines precision regression testing and performance checks for hardware and inference frameworks.

    1. Separate Total Parameters from Actual Compute

    The advertised numbers for very large models usually refer to total parameter count. For a mixture-of-experts (MoE) model, total parameters include a large number of expert networks, but each token passes through only a few of them. Total parameters therefore determine the lower bound for VRAM, while activated parameters determine per-step compute.

    Consider a hypothetical model with 2800B total parameters and about 100B activated per token. It has hundreds of experts, each routing step selects a dozen or so experts, and a small number of shared experts are retained that every token passes through. Under this structure, inference compute cost is close to that of a dense model in the tens-of-billions class, while storage cost is close to that of a trillion-parameter model. The gap between the two is central to deployment decisions.

    This gap leads to a direct conclusion: routing decides which experts to use only at runtime, so the experts cannot be known in advance. VRAM must hold all experts, not just the activated portion. Many teams estimate hardware from activated parameters and then find that VRAM is far from sufficient when they go live.

    2. Capacity Estimation: From Parameters to VRAM

    The basic formula for weight memory is as follows.

    Weight bytes = parameter count × bits per parameter ÷ 8

    The script below estimates weight size at different bit widths:

    1. def weight_memory_gb(params_b: float, bits: float) -> float:
    2. # params_b: parameter count in billions; bits: bits per parameter
    3. return params_b * 1e9 * bits / 8 / 1e9
    4. for bits in (16, 8, 4):
    5. print(f"{bits}-bit: {weight_memory_gb(2800, bits):.0f} GB")
    6. # 16-bit: 5600 GB
    7. # 8-bit: 2800 GB
    8. # 4-bit: 1400 GB

    The theoretical value is only a lower bound. Actual usage also includes the following:

    • Quantization scale factors: low-bit formats usually share one scale value per small block, and this adds extra memory.
    • Unquantized layers: embedding layers, output layers, normalization layers, and similar components often remain at higher precision.
    • Multimodal components: if the model includes a vision encoder, its weights must also be counted.
    • Runtime overhead: framework memory pools, communication buffers, and fragmentation all consume VRAM.

    Weight estimates should therefore be based on actual VRAM readings after loading, not only on the theoretical formula.

    In addition to weights, inference also requires storage for the key-value cache (KV cache). Its usage is proportional to context length and the number of concurrent requests:

    1. def kv_cache_gb(layers, kv_heads, head_dim, seq_len, batch,
    2. bytes_per_elem=2):
    3. # Factor 2 accounts for separate K and V caches
    4. total = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    5. return total / 1e9

    This formula applies to full-attention layers. For layers that use sparse attention, the caching strategy depends on the specific design and must be corrected according to the actual implementation. In long-context scenarios, the KV cache often exceeds the weights and becomes the main VRAM consumer, so capacity planning must consider both the longest context and the maximum concurrency.

    3. Low-Bit Quantization: Save VRAM While Protecting Accuracy

    The goal of quantization is to represent weights and activations with fewer bits. A common approach today uses a microscaling floating-point format of around 4 bits for weights, where each small block shares a scale factor, and an 8-bit format for activations. Such schemes can compress weight size to about one quarter of 16-bit, but whether the accuracy loss is acceptable depends on the quantization method.

    Quantization methods fall mainly into two categories:

    • Post-training quantization (PTQ): converts an already trained model directly. It is low cost, but accuracy degradation is more noticeable at extremely low bit widths.
    • Quantization-aware training (QAT): simulates quantization error during training so the model adapts to low-precision representations. Accuracy is usually better, but additional training compute is required.

    When choosing a method, check three things. First, whether the target hardware natively supports the format; otherwise it may fall back to software emulation, reducing both speed and memory benefits. Second, whether the inference framework’s operators cover all compute paths for that format. Third, whether the quantized model passes regression tests on your own tasks.

    Low bit width does not equal linear speedup. The decoding stage is usually limited by memory bandwidth, and smaller weights reduce the amount read, but if operator implementations are poor, the gains may be very limited. Performance conclusions should be based on measurements.

    The basic precision regression testing flow is as follows:

    1. results = {}
    2. for variant in ["reference", "target_quant"]:
    3. model = load_model(model_path, precision=variant) # implement separately
    4. results[variant] = {
    5. task: run_eval(model, task) # implement separately
    6. for task in EVAL_SUITE
    7. }
    8. drop = {
    9. task: results["reference"][task] - results["target_quant"][task]
    10. for task in EVAL_SUITE
    11. }
    12. flagged = [task for task, d in drop.items() if d > ACCEPTABLE_DROP]

    If only a single precision version is officially released, teams cannot directly obtain a reference baseline. They need to establish a baseline with their own evaluation set first, then observe per-task changes in the deployed version.

    4. Sparse Attention and Very Long Context

    Standard full attention grows quadratically in compute with sequence length, while the KV cache grows linearly with length. When context reaches hundreds of thousands or even a million tokens, both compute and memory pressure rise sharply.

    The idea behind sparse attention is to limit each query to a subset of key-value pairs, reducing compute on long sequences. The cost is that it may miss critical information at long distances. A common compromise is to use sparse attention only in some layers and keep full attention in the rest, balancing efficiency and long-range dependencies.

    A nominal context window length does not equal effective utilization. Model performance on long text usually varies by position, so evaluation should be conducted across lengths and positions rather than relying only on the nominal window size.