How to Deploy Large MoE Models and Plan GPU Configuration

    This guide explains how to plan GPU configuration for deploying large Mixture-of-Experts (MoE) models. It shows why activated parameters should not be used to size memory, how to estimate weight memory, KV Cache, and runtime overhead for each precision format, and how context length and concurrency change the requirements. It also covers server sizing beyond the GPU and a step-by-step method for choosing a configuration and validating it with load testing before production. The estimates and example configurations are for planning reference only. Actual results depend on your model, framework, quantization backend, parallelism strategy, and load-test results.

    When a team prepares to deploy a very large MoE model, the first question is almost always the same: “Can our existing servers run this model?” Without stated assumptions, that question invites misleading answers. Whether a large MoE model can be deployed, and how well it performs, depends on at least three independent variables:

    1. Numerical precision: sets the base size of the weights;
    2. Context length: determines the additional memory pressure from the KV Cache;
    3. Concurrency: drives throughput targets, peak memory usage, and latency.

    The right engineering question is therefore not a simple “yes” or “no.” It is: at a given precision, context length, and concurrency target, how much GPU memory, how many GPUs, and which parallelism strategy are required, and how can load testing confirm the system is ready for production?

    1. Key Principle: Activated Parameters Are Not the Same as Weight Footprint

    The most common sizing mistake with MoE models is treating activated parameters as weight footprint. A model might activate only about 40B parameters per forward pass, while the full set of expert weights could be around 744B. For memory planning, these two numbers must be kept separate:

    • Activated parameters determine the compute per inference, and therefore the compute capacity required and the inference latency.
    • Total weight parameters determine how many weights must be loaded into memory, and therefore the memory budget.

    No matter which experts a given token activates, all expert weights must either be fully loaded or distributed across the GPU nodes. You cannot size a node the way you would for a 40B dense model.

    Using standard precision conversions, the weight footprint of 744B parameters in inference looks roughly like this:

    • BF16: 744B × 2 bytes ≈ 1.49 TB
    • FP8: 744B × 1 byte ≈ 744 GB
    • 4-bit: 744B × 0.5 bytes ≈ 372 GB

    In short: activated parameters determine how fast you can compute, while weight size determines whether the model fits. Capacity planning must start by estimating memory from the total weights. KV Cache and runtime overhead are then added on top.

    2. Deployment Tiers Must Be Conditional: Look at Precision, Memory, and Parallelism Together

    Different precisions and GPU memory tiers are not simply “the higher, the better.” They map to different business stages and acceptance goals. Treat the tiers below as planning references, not unconditional conclusions. Whether a configuration works in practice must be verified through testing with the specific model, inference framework, parallelism strategy, context length, and concurrency.

    Deployment goal Example precision and weight footprint Example memory and nodes Parallelism strategy Typical use case Key prerequisites and risks
    Functional validation and production at moderate concurrency FP8, ~744 GB 8 GPUs with ~141 GB per GPU (e.g., H200 141GB-class) Commonly 8-way tensor parallelism (TP), or TP combined with expert parallelism (EP) Short to medium context, low-to-moderate concurrency in production or pre-production Requires framework support and a compatible model architecture; leave room in memory for runtime overhead; validate against the target model in practice
    Long context and extended sessions Depends on precision: FP8 or lower 8 GPUs with ~180 GB per GPU (e.g., B200, GB200, or comparable high-memory cards) Any needed combination of TP, EP, and pipeline parallelism (PP) Contexts of 128K to 1M tokens KV Cache must be calculated from layer count, KV heads, head dimension, sequence length, bytes per parameter, and concurrency; if memory is insufficient, reduce concurrency or truncate the context
    Low-memory integration testing and algorithm validation 4-bit, ~372 GB 8 GPUs with 80 GB or 96 GB per GPU 8-way TP, or hybrid parallelism supported by the framework Low-concurrency functional validation, algorithm integration testing, offline evaluation Lowest memory pressure, but quality changes from quantization must be accepted and verified, and throughput and latency must meet business requirements

    One point deserves special emphasis. The 141GB and 180GB per-GPU figures are example memory tiers for planning purposes only. They do not mean any model will necessarily fit on any framework, at any context length, under any concurrency. Inference frameworks differ in how well they support MoE expert parallelism, tensor parallelism, KV Cache management, and quantization backends. The final configuration must be based on the target model, the target framework, and load-test results.

    3. Capacity Estimation: Calculate Weights, KV Cache, and Runtime Overhead Separately

    3.1 Weight memory

    The baseline weight memory in inference can be approximated as:

    Weight memory ≈ parameter count × bytes per parameter

    For example, 744B parameters:

    • BF16: 744B × 2 ≈ 1,488 GB
    • FP8: 744B × 1 ≈ 744 GB
    • 4-bit: 744B × 0.5 ≈ 372 GB

    This is only the weight-side baseline. Real deployments must also account for routing parameters, quantization metadata, framework-reserved buffers, communication buffers, and more.

    3.2 KV Cache

    The KV Cache depends closely on model architecture, context length, and concurrency. It can be approximated as:

    KV Cache ≈ 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per parameter × concurrency

    The factor of 2 accounts for the K and V tensors. For long-context scenarios, the KV Cache can become the dominant memory consumer.

    Suppose the layer count, KV heads, and head dimension of a model are fixed, and KV Cache precision is unchanged:

    • Going from an 8K to a 200K context increases sequence length by roughly 25×.
    • Reducing concurrency from 8 to 2 cuts the concurrency-side memory by about 4×.
    • Combined, the KV Cache for 2 concurrent requests at 200K can still require about 6.25× the memory of 8 concurrent requests at 8K.

    Everyday 8K traffic and occasional 200K requests therefore cannot share the same memory assumptions. Long-context capability must be validated through KV Cache calculations and real load tests.

    3.3 Total memory and runtime overhead

    Deployments should use a more complete formula:

    Total memory = weight memory + KV Cache + runtime and communication overhead + safety margin

    Runtime and communication overhead includes:

    • temporary activation buffers;
    • communication buffers for parallel execution;
    • framework scheduling and memory pool overhead;
    • extra metadata from the quantization backend;
    • system overhead such as monitoring, logging, and debugging interfaces.

    For high-load scenarios, reserve a safety margin of at least 10% to 20%. This keeps memory usage away from the limit for long periods.

    4. Which Variables Actually Determine the Configuration

    GPU model and GPU count are outputs, not inputs. The real drivers are business variables, which are best evaluated in this priority order:

    1. SLA and latency targets
    2. Context length distribution
    3. Actual concurrency
    4. Whether quantization is acceptable
    5. GPU interconnect, framework version, and deployment topology

    The specific variables include:

    • Typical vs. peak context length: everyday 8K and occasional 200K represent completely different memory budgets.
    • Maximum output length: longer outputs keep the KV Cache occupied for longer per request.
    • Reasoning mode and agentic workflows: long chains amplify the lifetime of each request and occupy more concurrency slots.
    • Online users vs. actual concurrency: online user count is not the same as concurrency. Capacity should be estimated from the number of requests that actually run inference at the same time.
    • Tool calls and long sessions: multi-turn tool calling and long-lived sessions reduce effective system throughput.
    • GPU interconnect and parallelism strategy: inter-GPU bandwidth, the expert parallelism scheme, and how tensor parallelism is partitioned all affect end-to-end performance.
    • Quantization quality risk: 4-bit lowers the hardware bar, but quality regression must be verified as acceptable before launch.

    5. “The Model Loads” Does Not Mean “The System Is Ready for Production”

    Running a single inference successfully is only the minimum bar. Before production, the system must also meet engineering requirements for throughput, latency, stability, rollback capability, and more. A deliverable capacity plan should cover at least the following acceptance dimensions:

    • TTFT: whether time to first token meets the business SLA;
    • P95/P99 latency: whether tail latency is acceptable;
    • Throughput: overall tokens/s, tokens/s per user, and tokens/s per node;
    • GPU utilization: whether there is headroom at peak load;
    • Memory watermark: whether enough safety space is reserved to avoid OOM or frequent paging;
    • Error rate: whether inference failures, timeouts, retries, and routing errors are under control;
    • Canary release and rollback thresholds: whether the system can roll back quickly if latency degrades, memory behaves abnormally, or error rates rise.

    For example, load testing can use reference acceptance lines like these:

    • peak memory watermark stays below 90%;
    • P95 latency of core APIs stays below the business SLA;
    • error rate stays below 0.1% or the threshold allowed by the business;
    • 30 minutes of continuous load testing with no OOM and no severe throughput drops;
    • canary releases support both version rollback and configuration rollback.

    These numbers are not fixed standards. They illustrate the principle: launch acceptance is not just about whether the model can load. It is about whether the system can serve reliably under target traffic.

    6. Server Requirements Beyond the GPU

    GPUs are only one part of the machine. A node suitable for high-precision deployment also needs adequate CPU, system memory, local storage, and interconnect and cooling capacity.

    6.1 CPU

    • Without CPU offloading, dual-socket CPUs are usually enough for scheduling, data preprocessing, and monitoring.
    • With CPU offloading enabled, add CPU cores and memory bandwidth so that offloading does not become a bottleneck.

    6.2 System memory

    Plan for at least 1.5× to 2× the GPU memory capacity, and adjust according to the offload ratio.

    Rough estimates:

    • without offloading, 1.5TB-class memory is a reasonable starting point for a high-end node;
    • every ~10% of weights offloaded adds roughly 744 GB × 10% ≈ 74 GB of extra host memory;
    • if monitoring, caching, logging, and container orchestration also run on the node, add further safety margin.

    6.3 NVMe storage

    Estimate storage as number of model versions × size per version + dataset cache + runtime cache + 20% margin.

    For example:

    • keeping 3 model versions at about 744 GB each takes about 2.2 TB for weights alone;
    • with datasets, logs, temporary KV caches, or image caches, 8 TB or more of enterprise-grade NVMe is recommended;
    • multi-node training or evaluation environments may need even more capacity.

    7. Putting It into Practice: Set the Goal, Then Choose GPUs, Then Validate with Load Testing

    The recommended decision sequence is:

    1. Define the business goal: target context length, target concurrency, acceptable TTFT and P95/P99 latency, and throughput targets.
    2. Estimate the memory budget: calculate weight memory, KV Cache, runtime overhead, and safety margin separately.
    3. Choose a precision tier: FP8 balances quality and memory; 4-bit suits low-memory validation; BF16 is considered only for special quality or ecosystem requirements.
    4. Work backward to GPU model and count: use per-GPU memory, the parallelism strategy, and framework support to determine the node configuration.
    5. Load-test with real traffic: do not accept a configuration table at face value. Use load-test results as the basis for launch.

    Recommended load-testing steps:

    1. Fix input and output lengths, such as 4K input and 2K output, to simulate a typical workload.
    2. Start at low concurrency and increase it step by step.
    3. Record throughput (tokens/s), TTFT, P95/P99 latency, GPU utilization, memory watermark, and error rate.
    4. Find the balance point between throughput and latency.
    5. Test long contexts separately, such as 128K, 200K, or 1M tokens.
    6. Keep a safety margin of around 20% and produce a final, acceptance-ready configuration table.

    8. FAQ

    Q1: How do I estimate memory usage for an MoE model?
    Use weight memory + KV Cache + runtime overhead + safety margin. Weight memory is roughly the parameter count multiplied by bytes per parameter. KV Cache is roughly 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per parameter × concurrency.

    Q2: Only about 40B parameters are activated. Why can’t I size the node like a small model?
    Activated parameters reflect compute per inference, but all ~744B weights must still be fully loaded or distributed across GPU nodes. The memory budget must be based on total weight size, not activated size.

    Q3: Can 8 high-memory GPUs handle a 1M context directly?
    Not necessarily. Fitting FP8 weights on 8 GPUs and supporting a 1M context are two separate questions. The KV Cache for a 1M context can be very large and must be calculated from layer count, KV heads, head dimension, bytes per parameter, and concurrency. If memory is insufficient, reduce concurrency, shorten the context, or move to higher-memory nodes.

    Q4: What should be validated before launching a 4-bit quantized deployment?
    At minimum, validate quality regression, TTFT, P95/P99 latency, throughput (tokens/s), memory watermark, error rate, and the stability of the quantization backend under different parallelism strategies. Lower memory requirements do not mean the model is ready for production. Launch decisions must be based on business acceptance criteria.

    Disclaimer: The capacity estimates, GPU counts, and deployment tiers in this article are for planning reference only. Actual results depend on the model version, framework version, quantization backend, interconnect topology, parallelism strategy, and load-test results.