Local Deployment Cost for Trillion-Scale MoE Models: How to Estimate It

    Local deployment cost for a trillion-scale MoE model cannot be reduced to a single price. Using a hypothetical model with about 2.8 trillion total parameters, nearly 1,000 experts, and more than 10 activated experts per token, this article defines three deployment targets—just-run, single-user 25 tokens per second (tps), and 10 concurrent users at 25+ tps each—and shows how weights, KV cache, compute, all-to-all communication, hardware form factor, and multi-year TCO drive cost. It provides reusable estimation formulas, pseudocode, hardware order-of-magnitude estimates, optimization practices, and a deployment checklist.

    Many teams ask, “How much does it cost to deploy a trillion-scale MoE model locally?” That question cannot be answered with a single price. Model weights are only the lower bound. The real cost drivers are target throughput, concurrency, context length, quantization level, communication topology, and years of operations.

    This article uses a hypothetical case with about 2.8 trillion total parameters, nearly 1,000 experts, and more than 10 activated experts per token. It breaks the problem into three target tiers and provides a reusable estimation method and cost framework.

    1. Define Three Targets First, Then Price Means Something

    The first tier is “just make it run.” Speed does not matter. The only requirements are that the weights can be loaded and the model can output tokens, even at single-digit tps. The core challenge here is putting roughly 1.5 TB of weights into addressable storage, not raw compute. This tier suits algorithm validation, offline batch processing, and accuracy evaluation.

    The second tier is “single-user output speed reaches 25 tps.” This tier is latency-sensitive and requires stronger acceleration, higher memory bandwidth, and better expert placement. With a single request, KV cache pressure is relatively small, but per-token expert routing and cross-accelerator communication must not become bottlenecks.

    The third tier is “10 concurrent users, with single-user output speed still at 25 tps or higher,” which corresponds to about 250 tps aggregate throughput. At this point, planning must cover not only VRAM but also total KV cache capacity, batch scheduling, communication domain size, and P99 latency. Cost is usually an order of magnitude higher than the second tier.

    These three targets map to three hardware forms: large-memory heterogeneous servers, multi-accelerator servers with high-speed interconnects, and supernodes composed of dozens or more accelerators. Discussing “how much it costs” without a target easily underestimates communication and operations costs.

    2. Weight Lower Bound: About 1.5 TB Is Only the Starting Point

    For MoE deployment, memory and storage occupancy must be calculated from total parameters. The activation ratio affects compute consumption, not the fact that all experts must be stored. During inference, only a small number of experts are selected by routing, but all experts must either stay resident or be quickly swapped in.

    Consider a common quantization combination: 4-bit weights, 8-bit activations, and block scaling factors. The weight elements are not strictly 4 bits each. Every 32 4-bit elements also carry an 8-bit scale factor, which works out to about 4.25 bits per parameter. Pseudocode:

    1. P = 2.8e12
    2. weight_bits_per_param = 4 + 8 / 32
    3. weight_bytes = P * weight_bits_per_param / 8
    4. # about 1.49 TB, before non-4-bit tensors, metadata, alignment, and workspace

    Actual total occupancy must also include non-quantized tensors, metadata and alignment, runtime workspace, KV cache, expert swap buffers, and a 20%–30% safety margin. Therefore, “about 1.5 TB of weights” is only a storage lower bound, not the total machine memory requirement.

    KV cache can be estimated with a general formula:

    1. kv_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem

    With a single request and short context, KV cache may occupy only tens of GB. With 10 concurrent requests and long context, it can grow rapidly. If the quantization requirement is no lower than 4-bit weights and 8-bit activations, use the formula above for the weight lower bound. If lower precision is allowed, storage decreases, but accuracy risk and evaluation cost increase; this is not recommended as a production default.

    3. Compute, Communication, and Concurrency: 25 tps Is Not a Multiplication Problem

    MoE activates only a few experts per token, but experts are distributed across multiple accelerators. After each token is routed, activations must be moved to the accelerator hosting the target expert for computation, then moved back for subsequent layers. This forms typical all-to-all communication. Single-user 25 tps focuses more on per-token latency, while 10 concurrent users at 25 tps focuses more on aggregate throughput and communication overlap.

    Compute estimation cannot simply use “total parameters × activation ratio.” Expert activation ratio is not the same as strict parameter activation ratio. The attention backbone, gating components, and routing overhead must all be counted. In engineering practice, benchmark effective tps first and then apply a safety factor of 1.5–2:

    1. required_compute ≈ measured_flops_per_token * target_tps * concurrency * safety_factor

    The larger the communication domain, the harder it is to avoid cross-domain expert movement. The value of a supernode is that it places many accelerators in the same communication domain through high-speed interconnects, reducing cross-domain latency. If multiple machines are connected only through ordinary networking, compute may be consumed by communication, and a stable 25 tps target becomes difficult to achieve.

    4. Three Hardware Forms and Cost Orders of Magnitude

    Tier 1, “just run”: mainly large-memory servers configured with about 1.5–2 TB of memory, 1–2 high-end accelerators, and several TB of local high-speed solid-state storage. The shared attention part stays on accelerators, while sparse experts are offloaded to CPU memory and fetched according to routing. Speed may be only single-digit tps, which is suitable for validation. Rough hardware upfront investment is in the hundreds of thousands of RMB, with memory and storage as the main cost.

    Tier 2, “single-user 25 tps”: this requires multiple high-end accelerators, hundreds of GB to more than 1 TB of total VRAM, more than 1.5 TB of CPU memory, and high-speed host interconnect. A heterogeneous design can keep attention on accelerators and offload experts by layer, or more experts can be placed on accelerators. Rough hardware investment is in the millions of RMB, and tuning focuses on expert placement, batch size, and KV cache management.

    Tier 3, “10 concurrent users at 25 tps”: this requires a supernode composed of dozens or more accelerators. Total VRAM and memory must cover weights, KV cache, workspace, and swap buffers, while network, power, and cooling need redundancy. Rough hardware investment is in the tens of millions of RMB, and electricity, data center space, and labor operations are ongoing costs.

    Cost components can be roughly split into accelerators and memory, storage, network and supernode interconnect, chassis and data center, power and cooling, and operations labor. Use pseudocode to calculate TCO:

    1. capex = accelerator + cpu_memory + storage + network + chassis
    2. opex_year = power_kw * hours * price_kwh * pue + maintenance + labor
    3. tco = capex + opex_year * years

    The orders of magnitude above are only for budget discussion. Actual procurement quotes, supply conditions, and data center conditions govern.

    5. Optimization and Operations: Bring TCO Down

    Prioritize quantization and accuracy evaluation before deciding on hardware. 4-bit weights with 8-bit activations are usually a balance point between storage and accuracy, but they must be validated with business data. Next, optimize expert offloading and placement: keep compute-intensive shared parts on accelerators, place sparse experts in large memory, and adjust caching according to routing hit rate. Third, implement KV cache paging, prefix reuse, and concurrency scheduling; reserve capacity especially for 10-concurrent scenarios. Fourth, overlap communication and balance experts to avoid hot experts slowing down the whole system.

    For operations, monitor P99 token latency, aggregate throughput, VRAM and memory bandwidth, interconnect utilization, expert hit rate, and error rate. If teams want to reduce the burden of artifact distribution and monitoring, they can use general-purpose object storage, container platforms, and logging and monitoring alert services from a cloud provider under compliance constraints. However, the core cost remains local accelerators, memory, and power.

    6. Deployment Decision Checklist

    1. Define the SLA: tps, concurrency, P99 latency, context length, and quantization level.
    2. Calculate the weight lower bound first: about 1.5 TB, then add KV cache, workspace, and a 30% margin.
    3. Choose the form factor: accelerator-only, CPU-memory heterogeneous offloading, or supernode multi-accelerator.
    4. Run a small-scale benchmark: measure effective tps, communication bottlenecks, and expert hit rate, then extrapolate to full-machine configuration.
    5. Calculate three-year TCO: include hardware, electricity, data center, labor, and spare parts.
    6. Keep an accuracy regression set: quantization and offloading both affect output quality and must be continuously evaluated.

    The conclusion is that there is no single price for local deployment of a trillion-scale MoE model. A just-run setup may be in the hundreds of thousands of RMB, a single-user 25 tps setup may be in the millions of RMB, and a 10-concurrent 25+ tps setup may be in the tens of millions of RMB. Define the target first, then calculate storage, compute, communication, and TCO. That is how to avoid the trap of “buying the machines but never reaching the speed.”