Local Deployment of Trillion-Parameter MoE Models: Three-Tier Solutions and Cost Estimation

    Deploying trillion-parameter Mixture-of-Experts (MoE) models locally requires distinguishing between merely loading weights and delivering a usable service. This article introduces a three-tier deployment framework—L1 for feasibility, L2 for single-user usability, and L3 for production-grade concurrency. It details capacity estimation methods for weights and KV cache, outlines hardware and software requirements for each tier, and provides a comprehensive cost breakdown to prevent common deployment pitfalls like out-of-memory (OOM) errors and performance degradation.

    Local Deployment of Trillion-Parameter MoE Models: Three-Tier Solutions and Cost Estimation

    The parameter scales, number of experts, and quantization formats discussed in the following estimations are hypothetical constructs from public discussions. They are used to illustrate capacity estimation methods and do not correspond to any specific commercial model. Until official weights and technical reports are fully released, any VRAM, throughput, and hardware figures should only serve as budgetary references and must be verified item by item using actual weights and benchmark testing.

    1. Separating “Can Run” from “Works Well”

    The most common mistake in locally deploying ultra-large MoE models is equating “weights can be loaded” with “the service is available.” It is recommended to define three tiers of acceptance targets:

    • L1: Can run, regardless of speed. Single request, low concurrency, or single-digit tps is acceptable. The focus is on fully loading all experts without OOM and ensuring coherent output. Suitable for feasibility verification, internal evaluation, and offline tasks.
    • L2: Single-user output speed reaches 25 tps. Stable single-request throughput of 25+ tps, acceptable time-to-first-token (TTFT) latency, and no significant jitter in P99 metrics. Suitable for individual research, small team trials, and low-traffic services.
    • L3: 10 concurrent users each achieving 25+ tps, with quantization at least at the 4-bit weights / 8-bit activations level. Total throughput of ~250 tps, requiring continuous batching, KV cache management, expert parallelism, and high-speed interconnect coordination. Suitable for production-grade intranet services.

    These three tiers do not run on the same machine. L1 proves “feasibility,” L2 enters “usability,” and L3 achieves “production-grade.” Budgeting must be derived from the target tier, rather than purchasing hardware first and figuring out performance later.

    2. Capacity Estimation: Activation Ratios Do Not Reduce Memory

    The key characteristic of the MoE architecture is that while each token only activates a few experts, all experts must be addressable. Assuming a total of 2.8 trillion parameters, where 16 out of several hundred experts are activated per token, the expert activation ratio is about 1.79%. This ratio primarily affects compute consumption, not weight storage. During inference, all experts must still be loaded into VRAM or high-speed accessible memory, and the target experts are called out based on routing.

    Using 4-bit weights as an example, the raw data lower limit is about 1400GB. If block-level quantization is used—where every 32 4-bit elements are accompanied by an 8-bit scale factor—calculated by weight elements plus the scale factor, it is about 2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB. This does not account for non-4-bit tensors, metadata, alignment, runtime workspaces, or KV cache. In engineering practice, 1.7 to 2.0TB should be reserved for weights and workspaces, depending on the actual weight manifest.

    KV cache must be estimated separately:

    1. def estimate_capacity(total_params=2.8e12,
    2. weight_bits=4,
    3. scale_bits=8,
    4. group_size=32,
    5. nonquant_ratio=0.05,
    6. workspace_ratio=0.15,
    7. kv_cache_gb=0):
    8. scale_overhead = scale_bits / group_size
    9. weight_gb = total_params * (weight_bits + scale_overhead) / 8 / 1e9
    10. total_gb = weight_gb * (1 + nonquant_ratio + workspace_ratio) + kv_cache_gb
    11. return weight_gb, total_gb
    12. weight_gb, total_gb = estimate_capacity(kv_cache_gb=80)
    13. print(weight_gb, total_gb)

    This function only provides a lower-limit capacity estimate. Real deployments must also add communication buffers, temporary expert routing tensors, CUDA graph/operator workspaces, fragmentation reserves, and monitoring overhead. Under L3 high concurrency, the KV cache grows linearly with context length, concurrency level, and network layers, and cannot be ignored.

    3. Three-Tier Deployment Solutions and Cost Logic

    L1: CPU-GPU Heterogeneous Offloading to Output Tokens

    The most straightforward approach is heterogeneous inference: keeping compute-intensive but low-proportion components like the attention backbone, routing, and shared calculations on the GPU, while offloading the storage-heavy sparse experts to CPU memory, feeding them to the GPU via large-memory servers. The GPU handles high-bandwidth computing, the CPU memory handles capacity, and NVMe serves as a secondary cache.

    The hardware focus is not on top-tier compute, but on large memory, memory bandwidth, PCIe/NUMA topology, and stable power supply. It is recommended to prepare at least 1.5 to 2TB of RAM, several high-bandwidth accelerators, local NVMe drives, and adequate power supply. Software focus should be on expert routing, asynchronous data transfer, operator fusion, and memory pool management. The risk is that CPU memory bandwidth and PCIe can become bottlenecks, and routing jitter can cause tps fluctuations.

    Costs are primarily driven by large memory and accelerators, far lower than full-scale multi-node supercomputers, but you must accept low tps. Acceptance criteria: full weights loadable, no swapping, coherent single-request output, and no memory leaks over long runtimes. L1 is ideal for initially verifying model capabilities and business feasibility.

    L2: Multi-GPU Expert Parallelism for Stable 25 tps

    Achieving a single-request 25 tps requires keeping weights in high-bandwidth GPU VRAM as much as possible and limiting expert parallelism communication to a single node or a single high-speed interconnect domain. Estimating based on 1500GB of weights, 80GB-class accelerators would require about 20 units just for weights. Factoring in workspaces, KV cache, and redundancy, 24 to 32 accelerators within the same interconnect domain are recommended as a starting point. The exact number depends on card capacity, quantization precision, and actual weights.

    The key at this tier is parallelization strategy: tensor parallelism splits the attention backbone, expert parallelism distributes experts, and activated values are transferred via all-to-all communication after routing. Communication latency will directly eat into compute capacity, so expert parallelism must not leave the high-speed interconnect domain. Software-wise, continuous batching, paged KV cache management, expert load balancing, and operator optimization are required.

    Costs enter the realm of multi-GPU servers or small supercomputers, with primary cost items being accelerators, high-speed interconnects, switches, and power/cooling. In the cloud, this corresponds to multi-node GPU clusters, high-speed networking, and large-memory instances. L2 acceptance is not “average tps,” but stable single-request 25 tps, controllable P99 Time Per Output Token (TPOT), no VRAM swapping, and no long-tail latency in expert routing.

    L3: Production-Grade Supernodes for 10 Concurrent Users at 25 tps

    L3 total throughput is about 250 tps, requiring that 10 concurrent paths each maintain 25+ tps, with quantization precision no lower than 4-bit weights / 8-bit activations. At this point, the bottleneck typically shifts from single-card compute to interconnects, KV cache, and scheduling. Following deployment suggestions from public discussions, full-scale high concurrency usually requires 64+ accelerators forming a single high-speed interconnect domain; the specific scale must be determined by stress testing.

    Production-grade solutions require: joint splitting via expert parallelism and tensor parallelism, continuous batching, paged and quantized KV cache, request scheduling and prioritization, expert load balancing, and full-link monitoring and alerting. Stress testing must cover 10 concurrency, long contexts, burst traffic, and single-card failure degradation. Acceptance metrics include per-path tps, P99 TTFT, P99 TPOT, peak VRAM, all-to-all bandwidth utilization, and error rates.

    Costs are the highest, including supernode hardware, high-speed switching, data center power/cooling, spare parts, and operations. If using cloud resources, evaluate combinations of general GPU instances, large-memory instances, high-speed networking, object storage, container platforms, and monitoring/alerting. For example, cloud providers like Baidu AI Cloud offer these general resource categories, though specific specifications and pricing must be verified through actual quoting and stress testing. For self-hosted solutions, depreciation, electricity, bandwidth, operations personnel, and software adaptation costs must be calculated.

    4. Costs Extend Beyond Accelerators

    Total costs should be broken down into: weight storage and memory, accelerators, CPU and NVMe, high-speed interconnects, power and cooling, software adaptation, stress testing and tuning, operations and monitoring, and spare parts. A common pitfall is purchasing cards based purely on “weight lower limit ÷ per-card VRAM,” ignoring KV cache, workspaces, and communication buffers, leading to frequent OOM errors or performance degradation post-launch.

    It is recommended to create three tables first: Capacity, Throughput, and Cost. The Capacity table lists weights, KV cache, activations, and workspaces; the Throughput table lists single-request tps, concurrent tps, and P99; the Cost table lists one-time investments and monthly operations. Start with L1 to verify feasibility, use L2 for single-user acceptance, and finally use L3 stress testing to decide whether to enter production. The three tiers correspond to three separate budgets and should not be confused with one another.