Local Deployment Cost Breakdown for a Trillion-Parameter MoE Model

    This article estimates the hardware, memory, bandwidth, networking, and operational costs required to deploy a 2.8T-parameter MoE model locally. It separates confirmed public architecture facts from engineering estimates, explains why total parameters drive memory footprint, why the communication domain is often more critical than raw accelerator count, and provides three deployment tiers with rough cost ranges and optimization levers for inference throughput.

    Local Deployment Cost Breakdown for a Trillion-Parameter MoE Model

    1. Bottom Line and Scope: Which Numbers Are Reliable and Which Are Estimates

    To determine how much it costs to deploy a trillion-parameter sparse Mixture-of-Experts (MoE) model on-premises, the first step is not choosing accelerators. It is separating the unknown from what can be reasonably calculated.

    • Confirmed public information: The target model’s architecture characteristics — about 2.8 trillion total parameters, 896 experts, and 16 activated experts per token — are publicly stated. The model also introduces a quantization-aware scheme from training, using 4-bit weights with 8-bit activations, and the official guidance recommends large-scale deployment with more than 64 accelerators forming a single communication domain.
    • Unconfirmed information: The weights and the full technical report have not been released. Therefore, every memory, bandwidth, concurrency, and cost figure below is an engineering-order-of-magnitude estimate based on public architecture information, not measured data. Once weights are released, these estimates must be validated with benchmarks: actual activated-expert distribution, share of non-quantized tensors, real KV-cache shapes, and cross-node communication efficiency.

    A second concept must be corrected up front: the expert activation ratio is not the parameter activation ratio. The 16/896 ≈ 1.79% figure describes the sparsity of expert routing. However, dense parameters such as the attention backbone, embedding layers, routing gates, and normalization layers are read for every token. If decode bandwidth estimates multiply total parameters by 1.79%, they will underestimate memory traffic by roughly 3–5x.

    2. What 2.8 Trillion Parameters Means for Memory

    MoE memory usage is determined by total parameters; the activation ratio mainly affects compute consumption. The reason is simple: routing is a runtime decision. You cannot know in advance which experts the next token will hit, so all 896 experts must be resident in addressable storage before inference.

    The raw lower bound is:

    2.8T × 4 bit ÷ 8 ≈ 1,400 GB

    But 4-bit floating-point formats usually share an exponent within blocks. With one 8-bit scale factor per 32 4-bit elements, the effective storage is 4.25 bits per parameter:

    1. 2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB # weights + scales only

    Even this is not a runnable number. Additional memory must be reserved for:

    Item Scale / note
    Non-quantized tensors Embedding layers, some attention projections, and routing gates often remain in 8-bit or 16-bit precision
    Metadata and alignment Per-layer/per-expert offsets, tensor headers, and 128B alignment fragmentation
    Runtime workspace Activation buffers and temporary all-to-all send/receive buffers, which grow with concurrency
    KV cache Scales linearly with context length and concurrency; see Section 6
    Weight-loading buffer Extra headroom is needed for cold-start sharded loading

    For engineering planning, it is safer to provision 1.7–1.9 TB of addressable accelerator memory, not 1.49 TB.

    For comparison, another open-source MoE model with about 670 billion parameters requires roughly 685 GB for the main weights plus auxiliary prediction parameters under 8-bit weights. Under 4-bit weights, it needs about 335 GB, plus about 37 GB for activations — around 400 GB total — to run on an 8-accelerator node with 640 GB of memory. The target model is more than four times larger, so this defines the lower bound for full deployment. No single-machine workaround can avoid it.

    3. Why the Communication Domain Is More Critical Than “Do I Have Enough Memory?”

    When 896 experts are sharded across dozens of accelerators, every token’s routing path involves: dispatch → compute on the target accelerator(s) → combine. This is a classic collective communication pattern: many small messages, high frequency, and highly sensitive to tail latency.

    For autoregressive decoding, the latency budget for a single token is divided into many small stages, and each fixed overhead is amplified. Suppose the target is 25 tokens/s:

    1. Per-token budget = 1000 ms / 25 = 40 ms

    That 40 ms must cover routing computation, two all-to-all communication phases (including synchronization with the slowest participant), expert GEMM, and KV-cache writes. When RDMA runs over standard Ethernet across racks, queueing and synchronization overhead for one all-to-all operation can consume more than 10 ms, and jitter directly translates into jitter in token intervals.

    This is the practical reason official guidance requires more than 64 accelerators to form a “super node”: use high-speed interconnects inside the communication domain so that expert movement stays within a single communication domain.

    The implication is important: if the budget is insufficient and you fall back to stitching multiple nodes over ordinary networking, the loss may be greater than simply reducing the number of accelerators.

    4. Three Tiers: Configurations and Cost Ranges

    Tier 1: “It Just Needs to Run,” Without Caring About Speed

    There are only two technical paths:

    1. CPU/memory path: Use a platform with more than 2 TB of memory, load the full weights into memory, and run inference through a CPU-side inference engine. It can run, but for a 2.8T-parameter model, one token requires scanning hundreds of GB of data. Even with multi-TB/s aggregate memory bandwidth, the achievable speed is usually below 0.1 tokens/s. This path is suitable only for offline batch jobs, structural validation, or accuracy comparison.
    2. Accelerator + host-memory offloading path: When accelerator memory cannot hold the weights, offload expert layers to host memory and page them in/out through unified memory or paging. This is somewhat faster, but introduces PCIe-level data movement and is still unsuitable for interactive use.

    Cost range: tens to hundreds of thousands of CNY for a one-time hardware investment — typically a multi-CPU platform, 2–4 TB of memory, and high-capacity storage. Power and operations costs are relatively low. The value of this tier is validating the workflow, not providing a service.

    Tier 2: Single User, 25 tokens/s

    This is the most critical tier, because it shifts the problem from memory capacity to memory bandwidth and latency.

    A rough decode estimate is:

    1. Bytes read per token ≈ dense parameters × 4 bit + expert parameters × (16/896) × 4 bit + KV increment
    2. Effective throughput ≈ aggregate accelerator memory bandwidth / bytes read per token (discounted by communication and scheduling efficiency)

    Assuming dense parameters are about 10% of the total, one token requires reading roughly 300 GB. At 25 tokens/s, this implies about 7–8 TB/s of aggregate read bandwidth, while also requiring 64-level accelerator capacity (about 1.9 TB total divided by ~80 GB per accelerator). On a single accelerator with around 3 TB/s HBM bandwidth, the issue is not raw bandwidth alone. The real constraint is that capacity must be spread across about 64 accelerators, and once spread out, every token’s expert routing becomes a two-stage collective communication across 64 accelerators — so the 40 ms budget is dominated by communication and synchronization.

    Configuration points: 64 or more high-memory accelerators within a single communication domain; high-speed inter-accelerator interconnect; topology alignment across node-internal and node-external domains (try to keep shards of the same expert within the same switching layer); weights resident in accelerator memory without offloading.

    Cost range: a cabinet- or rack-level investment, typically in the several-million to ten-million-plus CNY range, including accelerators, servers, high-speed networking, and datacenter modifications. Annual power usage is calculated for a full-cabinet load in the tens of kilowatts.

    Tier 3: 10 concurrent users, each ≥ 25 tokens/s, quantization no lower than 4-bit

    Concurrency is a nonlinear cost multiplier. It pushes up three things at once:

    • KV cache: Roughly tens to hundreds of KB per token, depending on layer count and KV-compression strategy. 10 concurrent users × 128K context = 1.3M token context, which translates to tens to hundreds of GB of extra accelerator memory, directly squeezing maximum context length or concurrency headroom.
    • Compute: Prefill is compute-intensive. 10 concurrent requests require more prefill compute and more complex scheduling. In practice, prefill/decode separation is often needed.
    • Communication: With larger batches, all-to-all messages are larger and more likely to saturate bandwidth. Single-stream latency and overall throughput require trade-offs.

    Therefore, Tier 3 usually requires 96–128 accelerators, or a lower concurrency/context headroom at the same accelerator count. Cost range: a one-time investment above ten million CNY. Annual operating cost — power, cooling, networking, and operations staffing — is typically 20%–40% of hardware depreciation.

    The Tier 3 constraint “quantization no lower than 4-bit” also locks the memory lower bound: you cannot trade down to 3-bit or more aggressive weight quantization for extra capacity. That means concurrency must be solved mainly by adding accelerators and optimizing scheduling.

    5. Example Estimation Script

    1. def moe_budget(total_params, dense_ratio=0.10, top_k=16, num_experts=896,
    2. bits=4, scale_bits=8, block=32, overhead=0.25):
    3. """Return estimated weight memory, bytes read per token, and required aggregate bandwidth."""
    4. eff_bits = bits + scale_bits / block # 4.25 bit
    5. weight_gb = total_params * eff_bits / 8 / 1e9
    6. plan_gb = weight_gb * (1 + overhead) # Reserve workspace/KV/alignment
    7. expert_params = total_params * (1 - dense_ratio)
    8. read_gb = (total_params - expert_params) * eff_bits / 8 / 1e9 \
    9. + expert_params * (top_k / num_experts) * eff_bits / 8 / 1e9
    10. return plan_gb, read_gb
    11. plan_gb, read_gb = moe_budget(2.8e12)
    12. for tps in (25,):
    13. print(f"Memory plan: {plan_gb:.0f} GB, token read: {read_gb:.0f} GB, "
    14. f"required aggregate bandwidth: {read_gb * tps:.1f} TB/s")
    15. def token_cost(hardware_cny, years=3, power_kw=50, kwh_price=0.8,
    16. utilization=0.5, tokens_per_s=200):
    17. """Estimate the self-hosted cost per million tokens after utilization adjustment."""
    18. yearly = hardware_cny / years + power_kw * 24 * 365 * kwh_price
    19. yearly_tokens = tokens_per_s * 3600 * 24 * 365 * utilization
    20. return yearly / (yearly_tokens / 1e6)

    6. Cost Ledger: Don’t Look Only at the Price of Accelerators

    One-time capital expenditure is only half of the denominator. What really determines whether a deployment is expensive is utilization.

    1. Cost per million tokens = (annual hardware depreciation + annual power + annual operations) / annual effective output tokens

    Using 10 million CNY of hardware, 3-year depreciation, 50 kW full-cabinet power, and 200 tokens/s aggregate throughput, the cost at full load is on the order of a few hundred CNY per million tokens. If utilization drops to 30%, unit cost increases by more than 3x.

    This is the core economic problem of self-hosting a trillion-parameter MoE model: its optimal operating point is continuous high-concurrency online serving, not occasional Q&A from a few users. For low-frequency workloads, per-token cost from hosted inference services is usually clearly lower, because the service provider amortizes idle risk through multi-tenancy.

    The following must also be counted:

    • Storage and loading time for 1.5 TB weights across multiple replicas. Loading 1.5 TB from local disk at 5 GB/s takes about 5 minutes; over a 10 GbE network, it can take more than half an hour. This directly affects elastic scaling and failure recovery.
    • High-speed network switch ports.
    • Datacenter power capacity and cooling.
    • Personnel with experience in distributed MoE inference tuning.

    7. Optimization Checklist: Squeezing More Performance Without Adding Accelerators

    1. Expert parallelism and shard-topology alignment: Place expert shards according to the communication topology to reduce hops across switching layers.
    2. Communication-computation overlap: Pipeline dispatch/combine with expert GEMM to hide communication latency.
    3. Prefill/decode separation: These workloads have very different resource profiles. Co-locating them can interfere with tail latency.
    4. KV-cache compression and quantization: This is the most direct lever for improving Tier 3 concurrency.
    5. Speculative decoding: If the model provides multi-token prediction heads or a small draft model can be used, single-stream latency can be improved significantly.
    6. Continuous batching + chunked prefill: Increase effective throughput and smooth token intervals.
    7. Quantization format validation: Confirm the list of non-quantized tensors and the scale layout to avoid underestimating memory requirements by more than 10%.

    8. Decision Recommendations

    • If the goal is evaluation and validation: Choose Tier 1 and use a memory-heavy platform to get the workflow running. The investment is controllable.
    • If the goal is high-quality interactive use by one person: Tier 2 is the real threshold. Budget should be approved for a full communication domain, not just a card count.
    • If the goal is offering a service to external users: Choose Tier 3, and calculate utilization and cost per million tokens first before deciding between self-hosting and hosted inference.

    Finally, the scope must be stated again: all memory, bandwidth, accelerator-count, and cost figures above are estimates based on public architecture information. After the weights and technical details are released, every assumed parameter must be replaced with real benchmark results — especially actual activation distribution, KV-cache shape, and cross-node communication efficiency. These three factors affect the conclusion far more than fluctuations in accelerator prices.