Cost Guide for Local Deployment of Ultra-Large MoE Models

    This guide estimates the cost of running an ultra-large Mixture-of-Experts (MoE) model locally. It covers memory needs for weights, KV cache, and workspace, three hardware tiers tied to speed and concurrency targets, expert offloading, communication bottlenecks, quantization, a three-year total cost of ownership (TCO) method, and a checklist to run before launch.

    1. Define the Problem First: Cost Goes Beyond One Accelerator Card

    Estimating the cost of local deployment for an ultra-large MoE model requires separating three questions: whether the model fits in memory, whether it generates output at an acceptable speed, and whether it can serve multiple concurrent users. Each question sets a different hardware threshold.

    A model can “run” with a large amount of host memory and a few accelerators. Reaching 25 tokens per second (tps) for a single user usually requires keeping frequently used experts on high-speed compute devices and solving the bottlenecks of host memory bandwidth and bus transfers. Serving 10 concurrent users, each at 25 tps or more, requires expert parallelism, a high-speed interconnect domain, and stricter capacity reserves. The cost of that setup is an order of magnitude higher.

    This article uses a representative ultra-large MoE model as the basis for its estimates. The model has about 2.8 trillion total parameters, 896 experts, and 16 experts activated per token, an expert activation ratio of about 1.79%. The expert activation ratio is not the same as a strict parameter activation ratio. Storage for an MoE deployment must still be estimated from total parameters. The activation ratio mainly affects compute consumption and routing communication.

    The model weights have not been published. All figures below are capacity estimates. Once the weights are released, verify each figure against the actual file sizes and benchmark results.

    2. Capacity Floor: Calculate Weights, Workspace, and Cache Together

    With 4-bit quantization, the raw weight floor for 2.8 trillion parameters is:

    1. 2.8e12 × 4 bit ÷ 8 ≈ 1400GB

    A real deployment cannot account only for raw weights. With a micro-scaling format that uses 4-bit weights plus one 8-bit scaling factor per 32 elements, the effective bit width is about 4.25 bits:

    1. 2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5GB

    This still excludes non-4-bit tensors, metadata, alignment padding, runtime workspace, KV cache, communication buffers, and fragmentation reserves. A more reliable estimation method is:

    1. Weight storage ≈ total parameters × effective bit width ÷ 8
    2. Total requirement ≈ weight storage × (1 + metadata and alignment factor)
    3. + activation workspace + KV cache + communication buffers + fragmentation reserve

    For a weight set close to 1.5 TB, reserve at least 20% to 40% in additional space. A single machine’s memory or addressable storage should not be sized to exactly 1.5 TB. Otherwise, loading non-4-bit tensors or handling long-context requests can easily trigger out-of-memory errors or frequent paging.

    Estimate the KV cache separately:

    1. KV per token ≈ 2 × number of layers × number of KV heads × head dimension × bytes per element
    2. Total KV ≈ KV per token × sequence length × number of concurrent requests

    Use the actual model configuration for the number of layers, KV heads, and other parameters. With 10 concurrent users and long contexts, the KV cache can grow from tens of gigabytes to several hundred gigabytes. Include it in the memory and storage budget from the start.

    3. Hardware Configurations for Three Performance Targets

    Tier A: Run the Model, Speed Not a Priority

    The goal is to load the roughly 1.5 TB of weights into addressable storage and produce tokens, even if only a few tokens per second. The most direct approach is heterogeneous CPU and GPU inference. Each MoE layer has a shared attention component. It is compute-intensive but makes up a small share of parameters, so it can stay on the GPU. The 896 sparse experts take up most of the storage, but each token activates only 16 of them, so they can be offloaded to CPU memory on a large-memory server.

    A typical configuration for this tier includes:

    • Large host memory, at least 2 TB recommended
    • Local high-speed SSD storage for cold loading and checkpoints
    • A small number of data center accelerators for attention, routing, and frequently used experts
    • Standard in-node interconnect

    This tier has the lowest cost of the three, but it has high latency and low throughput. It suits validation, offline batch processing, and internal demos. It is not suitable for interactive production.

    Tier B: Single-User Output at 25 tps

    25 tps means an average of about 40 ms per token. For an MoE model, each token passes through routing, expert transfer, expert computation, result return, and sampling. If many experts sit in CPU memory, memory bandwidth and host bus transfers can consume most of the latency budget. For this reason, a configuration of large memory plus a few accelerators is not enough. This tier usually requires multiple accelerators with expert sharding. Frequently used experts, or all experts, should go into a pool of GPU memory or high-speed memory. Shorten the context and reduce concurrency as well.

    For quantization, use at least 4-bit micro-scaling, and keep critical paths at 8-bit activation precision or higher. Compared with Tier A, cost typically grows several times. Most of the spending goes to accelerator count, GPU memory capacity, host memory bandwidth, and tuning labor. Verify whether the target is met with batch size 1, the target sequence length, and a realistic routing distribution. Theoretical compute alone is not a sufficient measure.

    Tier C: 10 Concurrent Users, Each at 25 tps or More

    The total throughput target is at least 250 tps, with quantization at 4-bit micro-scaling or higher. A single card or a single machine can hardly support this load cleanly, so the design needs expert parallelism and a high-speed interconnect domain. The 896 experts are spread across multiple cards. After routing, each token’s activations must move to the device that holds the target expert, and then move back after computation. All-to-all communication latency directly reduces the compute gains.

    Public discussions commonly suggest tens of accelerators forming a single high-speed communication domain, so that expert transfers stay within that domain as much as possible. If cross-node communication takes too large a share, end-to-end tps may fall short even when each card has enough compute. This tier has the highest cost. Beyond accelerators, it requires high-speed switching, data center power, cooling, fault recovery, and operations. Evaluate purchase and lease options by the full domain, not by per-card prices.

    4. Key Points for Heterogeneous Inference and Expert Offloading

    Expert offloading is more than moving weights into memory. Pay attention to four points:

    1. Pinned memory and asynchronous transfers to reduce loading jitter.
    2. Expert caching policy to keep high-frequency experts on accelerators and low-frequency experts in host memory.
    3. NUMA affinity to avoid slow memory bandwidth from cross-CPU-node access.
    4. Batching and routing aggregation to combine computation for requests that route multiple tokens to the same expert.

    Pseudocode:

    1. for layer in moe_layers:
    2. shared_out = gpu_attention(layer.shared, hidden)
    3. routes = router(hidden)
    4. for expert_id, tokens in group_by_expert(routes):
    5. if expert_id in gpu_cache:
    6. out = gpu_expert(expert_id, tokens)
    7. else:
    8. weight = async_copy_to_gpu(expert_id)
    9. out = gpu_expert(weight, tokens)
    10. scatter(out, tokens)

    This approach lowers the GPU memory requirement but adds transfer latency. Tier A can accept that latency. Tier B must hide transfers behind computation. Tier C should avoid cross-domain transfers as much as possible.

    5. Communication and Quantization: The Hidden Cost Drivers

    Large-scale MoE inference is very sensitive to all-to-all communication bandwidth. More experts spread across more devices means more frequent routing transfers. When estimating cost, do not ask only how much GPU memory is needed. Also ask how large the communication domain is, what the single-hop latency is, and whether high-bandwidth full-mesh interconnect is supported.

    For quantization, 4-bit weights greatly reduce storage. However, scaling factors, non-4-bit tensors, activation precision, and dequantization overhead all add to the cost. Quantization-aware training, introduced during supervised fine-tuning, can reduce accuracy loss. Task-level regression testing is still required.

    6. Cost Structure and Estimation Method

    Estimate the three-year TCO using the following formula:

    1. 3-year TCO ≈ hardware purchase or lease
    2. + power × PUE × operating hours
    3. + networking and switching
    4. + local storage
    5. + operations staff
    6. + software adaptation, benchmarking, and tuning
    7. + fault redundancy and spare parts

    Tier A has the lowest cost, mainly a large-memory server with a few accelerators. Tier B costs several times more, with most of the spending on accelerators and memory bandwidth. Tier C typically costs an order of magnitude more and requires data center facilities.

    To run a capacity rehearsal in the cloud first, use general-purpose categories such as a container platform, object storage, and monitoring and alerting. Baidu AI Cloud offers general-purpose services of this type. Confirm specifications through actual price quotes and benchmarking.

    7. Pre-Launch Verification Checklist

    1. After the weights are released, check the actual file size, effective bit width, and the share of non-4-bit tensors.
    2. Benchmark batch size 1 and batch size 10, and record the time spent in routing, expert computation, communication, and sampling.
    3. Measure peak KV cache usage with real business sequence lengths.
    4. Run task-level accuracy regression tests before and after quantization.
    5. Verify the expert cache hit rate and the memory bandwidth bottleneck.
    6. Compare purchase and lease options using three-year TCO, not per-card prices alone.

    8. Conclusion

    The cost of local deployment for an ultra-large MoE model depends on four factors: the capacity floor of the weights, the speed target, the concurrency target, and the size of the communication domain. For a goal of simply running the model, a heterogeneous setup with large memory and a few accelerators is enough. A single user at 25 tps needs multi-card expert sharding and hot-expert caching. Ten concurrent users at 25 tps or more require tens of accelerators in a high-speed interconnect domain, with reserves for KV cache, workspace, and communication buffers. Until the weights are published, treat all figures as capacity estimates. Final decisions should rest on actual benchmark results.