Running Trillion-Parameter MoE Models Locally: How to Calculate GPU Memory Requirements

    This article explains what open weights do and do not guarantee for local deployment. It covers how Mixture of Experts (MoE) models split storage and compute costs, how long context windows add memory pressure, and how to estimate weight size under quantization. It closes with a checklist for building a multi-GPU memory budget.

    Open Weights Do Not Mean Ready to Run

    Open weights refer to model parameter files that can be publicly downloaded after training. They answer whether a model can be obtained. They do not answer whether the model can run on your own machine. Three factors are independent of one another: the size of the parameter files, the amount of data that must stay in GPU memory during inference, and the license terms that restrict commercial use. When you evaluate an open model, check all three separately.

    The weights of very large models are usually split into hundreds of shards, and each shard is often tens of GB. Before downloading, confirm your available disk space, network stability, and support for resumable downloads. After the download finishes, run a full integrity check.

    MoE Architecture: Look at Total and Active Parameters Separately

    Mixture of Experts (MoE) models split the feed-forward layers into many expert sub-networks. For each token, a router selects a small number of experts to compute. As a result, these models have two size metrics:

    • Total parameters: the sum of all experts and shared components. This determines the size of the weight files and the storage requirements.
    • Active parameters: the parameters that actually take part in computing each token. This determines the compute per step and the inference throughput.

    The storage cost of an MoE model is close to that of a dense model with the same total parameter count. Its compute cost, however, is close to that of a dense model with the active parameter count. This asymmetry is the easiest thing to misjudge in local deployment. A model with a large total parameter count must be loaded entirely into GPU memory or system memory, even if its active parameter count is modest. Otherwise, the experts selected by the router cannot be read in time.

    Routing also creates two engineering problems. First, expert load can be unbalanced. Some experts are called frequently, and the devices that host them can become bottlenecks. Second, when experts are spread across multiple GPUs, cross-GPU communication is required, and communication bandwidth limits actual throughput. Expert parallelism and tensor parallelism are common ways to split a model. When the two are combined, their communication patterns differ greatly, so each must be measured separately.

    Long Context: The Compute and Cache Cost of Attention

    Standard attention compute grows quadratically with sequence length. The key-value (KV) cache grows linearly with sequence length. When the context window reaches the million-token scale, the cache alone can occupy a substantial amount of GPU memory.

    Sparse attention lets each query attend to only a subset of keys and values, which reduces compute. Some models use sparse attention in only some layers and keep full attention in the others, trading off quality against cost. Two points matter when you evaluate this. First, sparse attention mainly reduces compute. Cache capacity may not shrink by the same proportion, depending on whether the implementation compresses the cache. Second, the context length listed in model documentation is a theoretical upper bound. The usable length is constrained by your GPU memory budget.

    Quantization: From Bit Width to Byte Count

    Quantization compresses parameters from 16-bit floating point to lower precision. It is the main way to reduce GPU memory usage. The weight size is calculated as follows:

    Weight bytes = number of parameters × average bits per parameter / 8

    The “average bits per parameter” is not the same as the bit width of the data format. Take microscaling formats as an example. Every 32 elements share one 8-bit scaling factor, so the average overhead per parameter is 4 + 8/32 = 4.25 bits. If some layers keep higher precision, calculate those layers separately.

    The following code demonstrates this estimation:

    1. def weight_bytes(total_params, elem_bits, block_size=None, scale_bits=None):
    2. """Estimate the size of the weights in bytes. Excludes KV cache and runtime overhead."""
    3. bits = elem_bits
    4. if block_size and scale_bits:
    5. bits += scale_bits / block_size # Amortized overhead of block-level scaling factors
    6. return total_params * bits / 8
    7. # Example calculation for a hypothetical 2.8-trillion-parameter model. The values are for illustration only.
    8. size = weight_bytes(2.8e12, elem_bits=4, block_size=32, scale_bits=8)
    9. print(f"{size / 1e12:.2f} TB") # Output: about 1.49 TB

    The result shows that even with a 4-bit format, the weights of a very large model are still on the terabyte scale. Quantization can considerably lower the barrier to entry, but it does not make trillion-parameter models lightweight.

    GPU Memory Budget for Multi-GPU Deployment

    To determine whether a hardware setup can run a model, list the complete GPU memory budget:

    1. Weights: calculate with the formula above, then divide by the share each GPU carries after partitioning.
    2. KV cache: depends on batch size, sequence length, and attention structure.
    3. Activations and temporary buffers: depend on batch size and operator implementation.
    4. Communication and runtime overhead: communication buffers and framework usage in multi-GPU setups.
    5. Safety margin: reserved for memory fragmentation and peak fluctuations.