Deploying Large-Scale MoE Models: A Practical Guide to GPU Memory Planning

    This guide explains how to plan GPU memory and deployment capacity for large-scale Mixture-of-Experts (MoE) models. It clarifies why total parameters, activated parameters, KV Cache, runtime overhead, and concurrency all affect memory usage; provides a simple KV Cache estimation example; and outlines practical deployment strategies for POC, standard production, and long-context production. It also covers request-length tiering, separating model capability from the default serving window, and tuning key scheduling parameters such as max-model-len and max-num-seqs based on real workloads.

    1. Small Activated Parameters Do Not Mean Small Memory Usage

    In large-scale Mixture-of-Experts (MoE) architectures, one commonly overlooked fact is that total parameter count and activated parameter count are two different dimensions, and they affect GPU memory differently.

    For a model with roughly 744B total parameters and about 40B activated parameters per token, many teams ask: “If only 40B parameters are activated per token, shouldn’t memory usage also be around the 40B scale?” This is a typical misunderstanding.

    The reasons are:

    • All expert weights must remain resident in memory. MoE routing only determines which experts a token passes through. The weights of all experts must be loaded into memory in advance, ready to be activated. Activation chooses a computation path; it does not reduce storage requirements. Therefore, the memory baseline is determined by the full weight set, not by activated parameters.
    • Activated parameters affect compute and throughput, not the baseline memory footprint. A 40B activated parameter model means that the matrix operations for a single forward pass correspond to the scale of 40B parameters. It affects per-token compute cost, but it does not change the memory required to keep all weights resident.
    • KV Cache and concurrency are the second major source of memory pressure. This part grows with context length and concurrency, sometimes even faster, and is independent of parameter count. It is often the first thing that causes a service to fail.

    The conclusion is clear: to estimate GPU memory, you need to account for full model weights, KV Cache, runtime framework overhead, and concurrent requests together. Activated parameters only affect the compute side.

    2. Four Memory Accounts: Do Not Look at Weights Only

    A rigorous memory assessment should calculate at least four parts:

    1. Weight usage: total parameters × bytes per parameter. FP16 is about 2 bytes per parameter, while FP8/INT8 is about 1 byte per parameter. This is the fixed memory floor.
    2. KV Cache: strongly related to the number of layers, KV heads, head dimension, sequence length, batch size, and precision. It is a dynamically expandable part.
    3. Runtime overhead: inference framework memory pools, communication buffers, temporary activation tensors, and other runtime allocations. It is common to reserve 10%–20% extra headroom.
    4. Concurrency headroom: continuous batching dynamically combines requests. The peak memory from KV Cache under high concurrency should not be underestimated.

    Below is a simplified but practical KV Cache estimation script. In production, use the actual output from your serving framework.

    1. def estimate_kv_cache_gb(num_layers, num_kv_heads, head_dim,
    2. seq_len, batch_size, bytes_per_elem=1):
    3. # bytes_per_elem: FP16=2, FP8/INT8=1
    4. total = (2 * num_layers * num_kv_heads * head_dim
    5. * seq_len * batch_size * bytes_per_elem)
    6. return total / (1024 ** 3)
    7. # Example: 128K context length, batch size 8, INT8 KV
    8. print(estimate_kv_cache_gb(60, 8, 128, 131072, 8, 1))

    A credible memory lower bound comes from adding weights, KV Cache, and runtime overhead, then applying a safety factor. Estimating only weights will almost certainly overflow during production peaks.

    3. Deployment Strategies by Stage: From Proof of Concept to Production

    Different stages should have different requirements for hardware and precision. It is not necessary to start with the highest configuration.

    • Proof-of-Concept (POC) stage: use low-bit quantization such as INT4/Q4 to reduce the memory barrier. A small number of high-end accelerators can validate the pipeline, test business feasibility, and check model quality.
    • Standard production stage: FP8 is commonly used. On an 8-GPU single-node configuration with accelerators that provide around 141GB of memory per card, this can serve as a reference single-node deployment configuration, balancing accuracy and throughput.
    • Ultra-long-context production stage: fully supporting 1M context requires more memory and bandwidth. It is often better to evaluate higher-end accelerators with around 180GB of memory per card first, leaving enough room for KV Cache.

    These are resource-tier references, not a parameter table to copy directly. The final choice depends on your workload profile and SLA constraints.

    4. 1M Context Is Not a Default Serving Window

    When a model supports 1M context, this describes a capability limit, not the default serving window. Confusing the two is a common cause of stability problems in long-context services.

    The reason goes back to the four memory accounts above: longer context means larger KV Cache. When a request approaches 1M tokens, one request can consume the KV Cache equivalent of hundreds of thousands of smaller requests. If the full window is enabled by default without limits, a single long request can exhaust the memory of an entire GPU, causing other requests to queue or fail with out-of-memory errors.

    The correct approach is to decouple the model capability limit from the default serving window. Set the default serving window to the length actually needed by the business, such as 32K or 64K, and enable longer context only through a separate path when explicitly needed.

    5. Request Tiering: Differentiate Service by Request Length Distribution

    Instead of opening the full context window by default, first measure the distribution of request lengths and then design tiered service strategies. This is often the dividing line between a stable and unstable long-context service.

    A suggested tiering approach:

    • Within 32K: use the standard shared service pool. With RAG and retrieval augmentation, most business requests fall into this range and become the main throughput workload.
    • 32K to 128K: set quotas and queues to limit the number of long requests processed within a given time window, preventing them from consuming resources needed by shorter requests.
    • Above 128K: physically isolate long-context service instances from short-request services.
    • Approaching 1M: use a dedicated queue with low concurrency, sacrificing concurrency for stronger per-request resource guarantees.

    A key insight: one request close to 1M tokens can put roughly the same pressure on memory and queues as multiple 128K requests. The purpose of tiering is not to limit capability, but to route different request lengths to matching resource pools and prevent interference.

    6. Understanding Key Scheduling Parameters

    max-model-len and max-num-seqs are two deployment parameters that are easily misunderstood. Many capacity planning mistakes come from misreading them.

    • max-model-len: controls the maximum context window allowed for a single request. It defines how long one request can be, not how many users are online.
    • max-num-seqs: controls the number of active sequences, meaning requests currently being decoded, in a single scheduling round. It defines how many sequences are being generated at the same time, not the number of business users online.

    Neither parameter represents online user count. Real business concurrency also depends on request length, output length, queue policies, rate-limiting thresholds, and the results of continuous batching scheduling.

    For example, with the same max-num-seqs=256:

    • If most requests are short and memory has spare capacity, 256 may be conservative and can be increased gradually.
    • If long requests make up a large share, each sequence can require a huge KV Cache. In that case, 256 can exhaust GPU memory immediately and must be reduced significantly.

    Parameter tuning example. Field names may vary by framework:

    1. # Short-request-heavy workloads with ample GPU memory
    2. config_short = {
    3. "max_model_len": 32768,
    4. "max_num_seqs": 256,
    5. "gpu_memory_utilization": 0.90,
    6. }
    7. # Long-context workloads that need conservative scheduling
    8. config_long = {
    9. "max_model_len": 131072,
    10. "max_num_seqs": 32, # Significantly lower to prevent KV Cache buildup from exhausting memory
    11. "gpu_memory_utilization": 0.92,
    12. }

    7. Tuning Methodology: Let Real Workloads Drive Decisions

    In summary, tuning the deployment of ultra-large-scale MoE models follows one main principle: be data-driven, not default-driven.

    1. Measure first, plan second: before deployment, collect real request-length distribution, output-length distribution, and peak concurrency to build a workload profile.
    2. Account for all four memory parts: weights + KV Cache + runtime overhead + concurrency. Missing any of these can lead to wrong capacity estimates.
    3. Deploy in tiers: shared pools for short requests, quota-based queues for medium and long requests, and physically isolated services for ultra-long requests.
    4. Decouple windows: separate the default serving window from the actual service window so that a capability limit is not mistaken for a default value.
    5. Match parameters to workloads: max-model-len controls the window, while max-num-seqs controls active sequences. Repeatedly load test and adjust based on the workload profile.

    Parameter tuning detached from real workloads can fail during production peaks, no matter how standard the configuration looks. What determines whether a system can go live is not a specification table, but whether the model can reliably serve under real load and business constraints.