Local Deployment Costs for a 2.8-Trillion-Parameter MoE Model: A Cost Breakdown

    This article breaks down the hardware and budget needed to run a 2.8-trillion-parameter sparse MoE model locally at three service levels: a model that simply runs, 25 token/s for a single user, and 25 token/s per request across 10 concurrent requests. It covers the storage floor, all-to-all communication for expert routing, KV cache growth, and rough cost ranges. All figures are engineering estimates to be verified with benchmarks once the weights are released.

    Before the weights and technical report are fully public, the cost of local deployment cannot be estimated from the price of a single accelerator card. For a sparse MoE model with 2.8 trillion total parameters, cost is driven first by the storage floor, since all experts must fit in addressable memory. Next, expert routing requires all-to-all communication between devices. Finally, concurrency, context length, and KV cache determine the remaining requirements. This article covers three tiers: a model that runs at all; a single user receiving 25 token/s of output; and 10 concurrent requests, each still receiving 25 token/s, with weights quantized to at least 4-bit. All speed, memory, and cost figures are engineering estimates. They must be verified item by item with benchmarks once the weights are published.

    1. The Storage Floor: What 2.8 Trillion Parameters Means

    Memory usage in an MoE model depends on total parameters. The activation ratio affects compute cost but does not change the fact that every expert must be addressable. Assume a model with 2.8 trillion total parameters, 896 experts, and 16 experts activated per token. The expert activation ratio is about 1.79%. Note that this is the expert activation ratio, not a strict parameter activation ratio. At deployment, all 896 experts must be loaded into addressable GPU or host memory first. During inference, routing selects 16 experts for each token.

    With 4-bit weights and 8-bit activations using microscaling quantization, the raw 4-bit weight floor for 2.8 trillion parameters is about:

    1. 2.8e12 × 4 bit ÷ 8 ÷ 1e9 ≈ 1400GB

    Block-wise quantization usually also stores one 8-bit scaling factor for every 32 4-bit elements. Under common quantization formats, this brings the average to about 4.25 bits per weight:

    1. 2.8e12 × 4.25 bit ÷ 8 ÷ 1e9 ≈ 1487.5GB

    This figure does not include non-4-bit tensors, metadata, alignment padding, runtime workspace, or KV cache. The weight floor is therefore about 1.5TB. For budgeting, plan for 1.8 to 2.5TB of usable storage. If Tier 1 uses CPU memory offloading, start at 2TB and use 4TB for more headroom.

    For comparison, a model with about 670 billion parameters needs about 685GB of weights at 8-bit and about 335GB at 4-bit. Adding about 37GB of activations gives about 400GB, which fits on eight accelerators with 640GB of total VRAM. The 2.8-trillion-parameter model is more than four times that size, which sets the requirements for full deployment.

    Communication is the more critical constraint. The 896 experts are spread across multiple cards. After routing, each token’s activations must be sent to the card holding the target expert, then sent back after computation. All-to-all bandwidth and tail latency directly reduce effective compute. For this reason, large MoE models usually need 64 or more accelerators in a supernode, connected by high-speed interconnect. This keeps expert transfers inside one communication domain.

    2. Acceptance Criteria for the Three Tiers

    Tier 1: The model loads and generates tokens. Speed is not a priority, and single-digit tokens per second is acceptable. The goal is to verify weight integrity, quantization error, routing logic, and interface integration.

    Tier 2: A single user reaches at least 25 token/s of decoding speed, with quantization at 4-bit or higher. Context length, time to first token, and P95 tail latency must all be defined. Without them, “25 token/s” cannot be compared across setups.

    Tier 3: 10 concurrent requests each reach at least 25 token/s, for a total throughput of at least 250 token/s. KV cache grows linearly with concurrency and context length, and all-to-all communication for expert routing becomes the main bottleneck.

    3. Tier 1: Heterogeneous Offloading to Run the Model

    The most direct approach is heterogeneous inference across accelerators and CPUs. Keep the shared attention and gating layers on accelerators, since they have high compute density and fewer parameters. Offload the sparse experts, which account for most of the storage and only a few of which are active per token, to CPU memory served by a large-memory server. This allows a small number of accelerators to hold nearly 1.5TB of weights. The trade-offs are memory bandwidth limits and the latency of swapping experts in and out.

    A typical configuration is a dual-socket server processor, multi-channel high-capacity memory, more than 2TB of fast local storage, and one or two high-VRAM accelerators for attention and KV cache. The cost is on the order of hundreds of thousands of yuan, mainly for memory capacity, the server platform, and accelerators. The risks are that insufficient CPU memory bandwidth can slow expert swapping to single-digit token/s, and long contexts can push KV cache into the limited memory.

    A configuration sketch:

    1. tier: runnable
    2. weight_storage: cpu_memory
    3. experts: cpu
    4. attention: gpu
    5. min_ram_gb: 2048
    6. recommended_ram_gb: 4096

    This tier suits validation and integration testing, not production. For a 4-bit block-quantized model with about 670 billion parameters, the weights take about 377GB. That model can run on a dual-socket server processor with one high-VRAM accelerator. The 2.8-trillion-parameter model requires memory and storage budgets to scale up accordingly.

    4. Tier 2: 25 token/s for a Single User

    To keep a single user at a steady 25 token/s, reduce CPU offloading and keep experts resident on accelerators as much as possible. With a 1.5TB weight floor plus KV cache and workspace, total VRAM must be at least 1.6 to 2TB. Plan for 2 to 2.5TB. With 80GB-class accelerators, that means about 20 to 32 cards. Larger VRAM per card reduces the count, but interconnect bandwidth must not drop.

    The main challenge at this tier is not single-token compute. It is expert parallel scheduling during small-batch decoding. Expert parallelism, tensor parallelism, and pipeline parallelism must be combined so that all-to-all communication does not become a bottleneck. The cost is on the order of several million yuan, mainly for accelerators, high-speed interconnect between cards, switches, and servers. For acceptance, fix the context length. Measure P50 and P95 decoding speed, expert hotspots, load balance, and KV cache usage. Common risks include enough VRAM but insufficient interconnect bandwidth, which keeps throughput below 25 token/s, and quantization scaling factors adding overhead that makes actual throughput lower than estimated.

    5. Tier 3: 10 Concurrent Requests at 25 token/s

    Ten concurrent requests, each at 25 token/s, require a total throughput of at least 250 token/s. Single-path optimization is not enough at this scale, so larger parallelism and batching are needed. This usually means a supernode of 64 or more accelerators, with high-speed interconnect spanning the expert parallel groups, plus high-bandwidth storage, redundant power supplies, and cooling. The cost is on the order of tens of millions to hundreds of millions of yuan. It covers accelerators, supernode racks, high-speed switching, storage, power and cooling, spare parts, and operations.

    The core risk at this tier is tail latency. Small jitter in all-to-all communication can increase P95 latency. Expert hotspots can overload specific cards. Concurrent KV cache can consume VRAM quickly. As the fault domain grows, a single card failure can affect an entire parallel group. For temporary validation, you can use Baidu AI Cloud’s general-purpose GPU instances, object storage, and high-speed network services for elastic stress testing, then move stable workloads back to local infrastructure. This section covers only general capabilities and does not address specific interfaces or billing.

    6. Where the Money Goes

    A simplified three-year total cost of ownership:

    1. tco = accelerator_count × unit_price
    2. + memory_capacity × unit_price
    3. + storage_capacity × unit_price
    4. + switches_and_optical_modules
    5. + data_center_power_and_cooling
    6. + software_operations_and_staff
    7. + spares_and_fault_redundancy

    Rough cost ranges by tier: Tier 1 is on the order of hundreds of thousands of yuan, Tier 2 is on the order of several million yuan, and Tier 3 is tens of millions to hundreds of millions of yuan. These are not quotes. Market prices, accelerator supply, and data center conditions can change the results substantially. Do not overlook power and cooling. For supernodes with 64 or more accelerators, power and heat dissipation are the largest facility upgrade costs. Include UPS, power distribution, and air or liquid cooling in the plan early.

    7. Rollout Path: Validate Before Buying

    Follow five steps. First, define service-level objectives, including context length, concurrency, quantization precision, and tail latency. Second, run synthetic workloads to measure prefill, decoding, concurrency, and expert hotspots. Third, validate on a small scale using rented or borrowed general-purpose accelerator environments. Fourth, verify each item: weight loading time, time to first token, decoding speed, all-to-all bandwidth, memory swap frequency, and power consumption. Fifth, decide on the scale of local purchases.

    In summary, Tier 1 can run with large memory and a few accelerators, at a cost of hundreds of thousands of yuan. Tier 2, which targets 25 token/s for a single user, usually reaches several million yuan. Tier 3, which targets production throughput for 10 concurrent requests, requires a supernode of 64 or more accelerators, with a budget in the tens of millions to hundreds of millions of yuan. Until the weights are released, treat all figures as estimates to be verified. The actual budget should be determined by benchmark results.