Tiered Cost Strategies for Local Deployment of Ultra-Large MoE Models

    Deploying a 2.8-trillion-parameter sparse Mixture-of-Experts (MoE) model locally presents significant challenges in storage capacity and routing efficiency rather than just compute power. This article provides a tiered cost and configuration estimation for deploying a model with 2.8T total parameters, 896 experts, and 16 active experts per token. It covers weight capacity calculation with 4-bit quantization and outlines three service tiers ranging from basic functional verification to single-user high-speed inference and 10-concurrent high-throughput production, detailing the hardware requirements, estimated costs, and implementation steps for each.

    Deploying a 2.8-trillion-parameter sparse MoE model locally introduces a core contradiction: it is not about finding “a single GPU that can compute,” but rather “how to fit nearly 1.5TB of weights into addressable storage and ensure that routed expert computation is not interrupted.” Based on public information about a model with 2.8 trillion total parameters, 896 experts, and 16 active experts per token, this article discusses configurations and costs across three service tiers. Note: The speeds, memory, hardware configurations, and prices discussed below are magnitude estimates based on general public information. Final figures must be verified against actual weights, quantization formats, context lengths, and benchmark tests.

    1. Calculating Capacity: Weight Lower Bound at ~1.5TB, Activation Ratio Only Dictates Compute

    Memory footprint for MoE deployment must be calculated based on total parameters, not the activation ratio. All 896 experts must be addressable; during inference, 16 experts are routed and activated. The expert activation ratio is approximately 1.79%, which does not equate to a strict parameter activation ratio. It primarily affects the compute and weight read volume per token, without changing the fact that all experts must be stored.

    If 4-bit block quantization is used, where every 32 4-bit elements are paired with an 8-bit scaling factor, the equivalent bit width is approximately 4.25 bits. The capacity estimation is as follows:

    1. total_params = 2.8e12
    2. bits_per_weight = 4.25 # 4-bit weights + 8-bit scale per 32 elements
    3. weight_gb = total_params * bits_per_weight / 8 / 1e9
    4. print(weight_gb) # Approx. 1487.5GB

    This is just the raw weight data, excluding non-4-bit tensors, metadata, alignment padding, runtime workspace, communication buffers, and KV cache. Therefore, the practical lower bound for deployment should be viewed as 1.5TB of weights plus a 20% to 50% margin. A single session with a short context may approach 1.8 to 2TB of addressable space, while high concurrency with long contexts will require additional VRAM and RAM. If the model underwent quantization-aware training (QAT) during supervised fine-tuning, the precision risk of 4-bit deployment is relatively manageable, though regression testing on business tasks is still necessary.

    2. Three Service Tiers Correspond to Three Distinct Cost Structures

    Tier 1: Functional Run Only, Speed Ignored.
    The typical approach is CPU+GPU heterogeneous inference: the attention backbone and shared components stay on the GPU, while sparse experts are offloaded to CPU memory, fed by a large-memory server. It is recommended to have at least 2TB of RAM, 2-4TB of local solid-state storage, and a single accelerator card with tens of GB of VRAM for the backbone and scheduling. This setup can handle functional verification, prompt debugging, and precision regression, but the decode speed is usually only single-digit tokens per second (tps), and prefill is bottlenecked by CPU memory bandwidth. Hardware costs are in the tens of thousands, suitable for verifying if the model can be locally ignited, but not for interactive production.

    Tier 2: Single User, Output Speed at 25 tps.
    At this level, relying entirely on CPU offload long-term is not viable. Estimating based on reading activated expert weights per token: 2.8T × 1.79% ≈ 50B activated parameters. Under 4-bit, this means reading roughly 25GB of weights per token; 25 tps corresponds to an effective weight read bandwidth of about 625GB/s. The backbone shared parameters, KV cache, and routing overhead will further raise this requirement.

    • Plan A is All-VRAM: An 80GB card requires about 20 cards just for weights. Factoring in workspace, KV, and redundancy, 24 to 32 cards are recommended, costing in the millions but offering the most stable speed.
    • Plan B is Hierarchical Offload: 8 to 16 accelerator cards plus 2 to 4TB of high-bandwidth memory, keeping hot experts on the GPU and cold experts in memory. This requires testing whether the effective memory bandwidth can support 25 tps. The cost can drop to the lower millions, but tuning is complex.
      Tier 2 represents the inflection point of steep cost increases.

    Tier 3: 10 Concurrent Users, Individual Speed Still ≥25 tps.
    Total throughput must be at least 250 tps. Based on the read estimates above, aggregated weight bandwidth needs to reach about 6.25TB/s. Furthermore, with 896 experts distributed across multiple cards, every routed token requires all-to-all transfers. This demands a super-node composed of 64 or more accelerators, utilizing high-speed interconnects to expand the communication domain and keep expert transfers local as much as possible. Sixty-four cards is the starting point for weight capacity and communication domains; KV cache, peak workspace, and long context will require additional resources, and high-concurrency long contexts may push this number higher. Hardware costs are typically in the tens of millions, with electricity, cooling, switching networks, and operations excluded.

    3. Cost Composition and Implementation Order

    Costs extend beyond accelerator cards to include large-memory servers, local storage, high-speed switching, rack power, software adaptation labor, model licensing, and security compliance. It is recommended to proceed in the order of: “Capacity Calculation → Single Node Power-on → Single User Baseline → Multi-Concurrent Stress Test → Cost Review.”

    Acceptance testing should at least record: Time to First Token (TTFT), prefill tps, decode tps, P95 tail latency, per-user tps under 10 concurrent users, VRAM/RAM peak usage, expert routing all-to-all latency, KV cache growth, and OOM occurrences. Any “25 tps” conclusion must be tied to context length, batch size, quantization level, and sampling parameters.

    Conclusion:
    The conclusion is straightforward: Tier 1 costs in the tens of thousands for verification; Tier 2 requires millions to sustain a stable 25 tps for a single user; Tier 3 costs tens of millions and relies on super-node interconnects. Define the SLA first, then buy hardware; conduct baseline tests first, then discuss full-scale deployment.