The Real Cost of On-Premises Deployment for Trillion-Parameter MoE Models

    This article breaks down what it actually costs to deploy a trillion-parameter sparse Mixture-of-Experts (MoE) model on-premises, using a recently open-sourced model with roughly 2.8 trillion parameters, MXFP4 weights (~1.56 TB), and million-token context as a reference. It works through the VRAM budget that determines card count, the sizing math for 128 GB accelerators, the memory-bandwidth and expert-parallel All-to-All communication limits that shape performance, the software stack needed for efficient inference, and the supporting infrastructure (storage, network, power) that teams routinely under-count. With a full budget breakdown and a three-year TCO view, it explains why a minimal "can it run" environment starts around RMB 8 million while a production-grade system lands near RMB 30 million, and closes with six practical paths to bring costs down.

    The bottom line: expect ~RMB 8 million to start, ~RMB 30 million for a production recommendation

    TL;DR — Using a recently open-sourced trillion-parameter sparse MoE model as a reference (referred to below as the “target model”):

    • A local, private deployment starts at a minimum of ~RMB 8 million;
    • A production-grade recommended configuration totals around RMB 30 million.

    Keep in mind that RMB 8 million only buys a validation environment that “gets the model running.” To also satisfy throughput, latency, and availability targets, total spend approaches the recommended figure. This article lays out the full calculation so the numbers are defensible and can be used directly for internal budgeting.

    Over the past year, the open-source race has shifted from parameter count to inference efficiency, and a recently released trillion-parameter MoE model is a good example:

    • ~2.8 trillion total parameters, a sparse MoE architecture, and support for million-level token context;
    • weights released in native 4-bit floating point (MXFP4), ~1.56 TB in size, split across 96 shards;
    • an attention section built on a hybrid of “gated linear attention + sparse attention,” with support for multi-token prediction.

    On several public coding and terminal-task benchmarks, it performs in the same tier as leading closed-source models and even leads on some individual tests. On release day, a cloud provider’s supernode product completed Day0 adaptation — matching the inference framework and running full-chain validation the same day the model shipped.

    Which raises the real question: open weights are free, but how much genuine money does it take to turn them into an enterprise-ready service?

    1. Three deployment models, three very different budgets

    Deployment model Hardware starting point Budget scale Typical use case
    Fully on-premises, self-built 32+ accelerators RMB 8M–30M Data must stay in-domain, steady load, 3+ year plan
    Public cloud API No hardware purchase Pay per token Small or spiky volume, fast time to launch
    Rented cloud instances On-demand or reserved Per card-hour Temporary stress tests, peak events, POC validation

    The key point: free weights do not mean free deployment. VRAM, networking, storage, data-center space, power, and an operations team all cost real money.

    2. Start with the hard constraint: the VRAM math decides “how many cards”

    2.1 What makes sparse MoE different: total params eat VRAM, activated params eat compute

    An MoE model holds a large number of parallel “expert” sub-networks, and each token is routed to only a few of them for computation:

    • Total parameters determine the size of all model weights, which must reside entirely in VRAM;
    • Activated parameters determine per-token compute, usually only a small fraction of the total.

    So the binding constraint for a trillion-parameter model is often not compute at all — it is VRAM capacity and memory bandwidth.

    2.2 The four components of VRAM demand

    ① Model weights

    The target model’s weights are ~1.56 TB, the dominant share of VRAM usage.

    ② KV cache and sequence state

    The hybrid attention structure is designed specifically for long context:

    • Linear attention layers replace per-token caching with a fixed-size state matrix, so sequence length has little effect on them;
    • Sparse attention layers first use a lightweight indexer to score all historical tokens, then select only the most relevant few for computation. They still retain a compressed cache that scales with sequence length, but it is far smaller than full attention.

    A rough estimate for a hypothetical configuration: 10 sparse layers, 64 KV heads per layer, head dimension 128, cache precision 8-bit.

    • A single million-token session needs ~160 GB of cache;
    • 32 concurrent sessions need ~5 TB.

    At this scale, concurrency affects the VRAM budget even more than weights do. Many vendors also enable request-level logging and audit by default for compliance, which should be evaluated when planning capacity.

    ③ Runtime overhead

    The inference framework needs to reserve room for:

    • Communication buffers: expert-parallel All-to-All, plus the draft model for multi-token prediction;
    • Memory-pool fragmentation;
    • Temporary activation space.

    Typically reserve 10%–20% of a single card’s VRAM.

    ④ Redundancy margin

    • Degraded operation after a single-card failure;
    • Future upgrades to larger context or higher concurrency.

    Reserve at least 15%.

    Bottom line: with a target of “million-token context, dozens of concurrent requests,” actual VRAM demand is roughly 3–4 TB, and it grows with concurrency. Weights alone exceed 1.5 TB, which means a trillion-scale MoE model must be deployed with multi-card tensor parallelism plus multi-card expert parallelism — a single 8-card machine can no longer hold it.

    3. Sizing example: with 128 GB cards, 32 cards is just the starting line

    To quantify these constraints, assume:

    • 128 GB VRAM per card;
    • A single-rack supernode form factor with a 64-card high-speed interconnect domain — low-latency, high-bandwidth inside the rack, standard data-center networking across racks;
    • Target total VRAM demand of 3.5 TB (weights, cache, and reserve included).

    Available VRAM = card VRAM × card count × (1 − reserve ratio) = 128 GB × N × 0.85 = 108.8N GB

    Required card count N ≥ 3584 ÷ 108.8 ≈ 33 cards

    Expert parallelism also requires the card count to divide the number of experts evenly. If there are 256 experts, the card count can only be a multiple of 32:

    • 32 cards: available VRAM 3.48 TB, right at the threshold;
    • 64 cards: available VRAM 6.96 TB, leaving roughly double headroom for weights.

    In practice, you also subtract full-precision parameters the framework loads on top, and account for CUDA Graph and memory-pool fragmentation losses, so 32 cards is “runnable but with almost no margin.” The starting configuration for any trillion-scale MoE model is at least one rack of 32 cards. The card count at the rack level is itself a hard floor derived from the VRAM constraint.

    4. Then performance: bandwidth and communication decide “how well it runs”

    VRAM capacity decides whether the model boots; VRAM bandwidth decides how fast a single request decodes; inter-card communication decides the ceiling on high-concurrency throughput.

    4.1 Decoding is memory-bound

    Each generated token has to traverse the selected experts’ parameters. Roughly, the bytes moved per token ≈ activated parameters × bytes per parameter:

    • If a token needs to move ~50 GB of weights, at ~8 TB/s of single-card bandwidth the theoretical ceiling is ~160 tokens/s;
    • Real-world bandwidth utilization is typically only 30%–40%;
    • 64 cards aggregate to ~512 TB/s, so with an ideal expert split and no communication blocking, a single request’s theoretical decode ceiling can reach hundreds of tokens/s. The real number needs a stress test.

    This shows the trillion-parameter model’s advantage: a single request can be fast. But to sustain high throughput, the bottleneck moves from VRAM to the network.

    4.2 Expert-parallel All-to-All is the main communication bottleneck

    Each decoding layer typically involves two collective communications:

    • Dispatch: send each token’s hidden state to the card holding its selected expert;
    • Combine: gather the expert results back to the token’s original position.

    With 32 cards, ~10 MB-class communication per card, and ~400 GB/s of effective network bandwidth, a single All-to-All takes sub-millisecond time. That sounds small, but the model has dozens of layers and this sits on the serial critical path: accumulated latency can reach tens of milliseconds per second, directly determining a single request’s token output rate.

    Across racks, RDMA end-to-end latency is typically a few to tens of microseconds; add protocol overhead and congestion, and long-tail latency can amplify several times over. This is exactly the value of the supernode’s “single-rack, multi-card” form factor:

    • Converge high-frequency, fine-grained inter-card communication inside the rack;
    • Confine cross-rack communication to low-frequency work like pipeline parallelism.

    The planning principle that follows: the card count should be an integer multiple of the high-speed interconnect domain width. For example, with a 64-card high-speed domain, 32, 64, and 128 cards are all satisfied directly; if the business needs 96 cards, it is forced into a 64 + 32 cross-domain split, with clear performance degradation.

    4.3 From “runnable” to “usable”: define the business SLO, then back out the card count

    Card counts should not be guesswork — derive them from business metrics.

    Metric Meaning Notes
    TTFT (time to first token) From sending a request to receiving the first token Drives interactive experience; mainly determined by prefill
    TPOT (time per output token) Average time per token in the generation phase Drives reading fluency
    Output rate Tokens generated per second Roughly 1000 ÷ TPOT
    Max concurrency Number of requests that can meet the SLO simultaneously Determines queue length and queuing delay

    Steps to back it out:

    1. Determine peak QPS, the input/output length distribution, and acceptable TTFT, TPOT, and queueing time;
    2. Stress-test the target model to get “max concurrency of a single card group at a specific TPOT”;
    3. Divide peak concurrency by per-group concurrency, multiply by a 1.3–1.5 redundancy factor to get the required card count;
    4. Round up to an integer multiple of the high-speed interconnect domain width.

    5. The soft layer: behind Day0 adaptation is a complete engineering stack

    Having the hardware in place is only the first step; whether it runs efficiently depends on how mature the inference engine is. Efficient inference for a trillion-parameter MoE model requires at least four coordinated layers:

    ① Operator layer

    • Attention: sparse-attention indexing operators, plus linear-attention state updates;
    • MoE compute: 4-bit weight dequantization + grouped GEMM, where the key is to expand weights only briefly in registers/shared memory and avoid extra round trips to VRAM;
    • Sampling and speculative decoding: draft and verify for multi-token prediction.

    ② Parallelism strategy layer

    • Tensor parallelism: split attention and dense layers;
    • Expert parallelism: assign cards by expert, usually combined with tensor parallelism into a 2D split;
    • Pipeline parallelism: split layers across racks;
    • Sequence parallelism: handle the prefill of million-token contexts.

    ③ Scheduling layer

    • Continuous batching;
    • Chunked prefill: split long inputs into chunks and interleave them with decoding so long requests don’t block others;
    • Prefix caching: reuse repeated prefixes such as system prompts and long documents;
    • Prefill/Decode disaggregation: the two workload types have different compute profiles, so separating them allows independent optimization.

    ④ Platform layer

    • A unified model-serving gateway;
    • Elastic scaling;
    • Monitoring and alerting centered on TTFT, TPOT, and queue length.

    For a self-built team, the point of Day0 adaptation is this: when a mainstream inference framework adapts on the day the model ships, the engineering stack can be reused directly. Teams building or deeply customizing their own inference engines typically need an extra adaptation and tuning cycle of several weeks to months, and that labor cost should be counted in the total budget.

    6. Supporting infrastructure: three cost blocks that are easy to miss

    Storage

    Loading 1.56 TB of weights across 64 cards means each card reads hundreds of GB.

    • If everything is pulled from remote storage, at 10 GB/s aggregate bandwidth, loading alone takes minutes — and expansion or failure recovery is slower still;
    • Recommend local high-speed caching, or loading multiple replicas in parallel;
    • For capacity, reserve at least 3× the weight size to hold multiple versions, quantized copies, and rollback packages.

    Network

    • East-west: inference traffic stays within the rack, putting little pressure on external switches;
    • North-south: user traffic and operations traffic need to be isolated separately;
    • Cross-facility: if you run active-active across two sites, evaluate synchronization latency.

    Data center and power

    At ~700 W per card and ~8 kW per machine, 64 cards is ~64 kW of IT load:

    • At PUE 1.3, total power is ~83 kW;
    • Annual consumption is ~730,000 kWh;
    • At RMB 0.6/kWh, that’s ~RMB 440,000/year in electricity.

    On top of this, factor in rack rental, cooling upgrades, UPS, and operations headcount.

    7. Budget breakdown: from RMB 8M to RMB 30M, where the money goes

    Prices below are estimates based on public market rates and are meant only to illustrate budget structure; actual figures depend on quotes.

    Item Minimum config (1 supernode) Recommended config (2 supernodes)
    Accelerator servers 32 cards, ~RMB 4.5M–6M 64 cards, ~RMB 9M–12M
    High-speed network Included in the supernode Cross-node fabric, ~RMB 0.5M–1.2M
    Storage and local disks ~RMB 0.3M–0.6M ~RMB 0.6M–1.2M
    Data center and power ~RMB 0.2M–0.5M ~RMB 0.4M–1M
    Inference engine and ops tooling Mostly open source; labor listed separately Same
    Implementation and adaptation labor (one-time) 2–4 people × 2–3 months 4–6 people × 3–6 months
    3-year electricity ~RMB 0.8M–1.3M ~RMB 1.3M–2M
    Total (incl. 3-year electricity) ~RMB 7M–10M ~RMB 13M–18M

    On top of the recommended configuration, budget for:

    • A standby node or failover capability;
    • Capacity headroom and stress testing;
    • Post-launch tuning.

    Altogether, the total budget lands around RMB 25M–35M. There’s a caliber distinction here worth spelling out for decision-makers:

    • A ~RMB 10M budget corresponds to a validation environment that “can run”;
    • A ~RMB 30M budget corresponds to a production system that is “usable and stable.”

    The gap between them is mostly not in the cards, but in networking, storage, data center, redundancy, and labor.

    8. Three-year TCO: don’t let one-time procurement hijack long-term cost

    Cost item One-time Annual recurring
    Hardware purchase Yes Depreciation (over 3–5 years)
    Data center and power Retrofit cost Electricity, rack rental
    Software and tooling Mostly open source Commercial support subscription (optional)
    Labor Implementation, adaptation Ops, tuning, tracking model iterations
    Network and storage Yes Expansion, maintenance

    Compared with paying per use for a cloud API, self-building usually has a cost advantage under a long-term steady load — but only if:

    • Utilization is high enough (recommend sustained levels above 40%–50%);
    • The usage period covers the depreciation window;
    • There is a stable operations team.

    Otherwise, depreciation on idle compute is the most expensive waste of all.

    9. Six paths to cut costs

    ① Define the SLO first, then the card count

    Many budgets blow up because targets are set too high: million-token context, thousands of concurrent requests, millisecond-level first token — each one multiplies cost. Clarifying which scenarios genuinely need these numbers can usually cut 30%–50% of the hardware requirement.

    ② Use quantization and sparsity well

    The target model ships with native 4-bit weight quantization, already saving half the VRAM; the hybrid attention structure sharply reduces long-context cache. When evaluating other sizes or higher-throughput versions in the same family, prioritize whether the quantization format is native and whether the attention structure is friendly to long context — not just the parameter count.

    ③ Make inference-framework maturity a hard selection criterion

    Prefer models that already have Day0 adaptation on mainstream inference engines and extensive production validation, which significantly lowers adaptation risk and labor cost.

    ④ Deploy in a hybrid model to shave peaks

    • Run steady baseline load on the local cluster;
    • For business peaks, temporary stress tests, or POC phases, rent on demand via cloud GPU servers, elastic bare-metal instances, or container platforms.

    Major cloud providers (such as Baidu AI Cloud) offer mature GPU compute and high-performance storage products that can directly support this hybrid mode.

    ⑤ Trade inference optimization for hardware

    • Improve batching efficiency;
    • Enable prefix caching;
    • Use speculative decoding.

    With the same hardware, these usually deliver a meaningful gain in effective throughput, effectively lowering per-token cost.

    ⑥ Procure in phases

    Validate business value at small scale first, then expand on a rolling basis based on utilization data — avoiding a one-time heavy-asset purchase that sits idle for a long time.

    10. Four sentences for decision-makers

    1. Calculate VRAM first, then card count. A trillion-scale MoE model’s VRAM demand is set jointly by total parameters, context length, and concurrency — a single 8-card machine can’t hold it.
    2. Look at communication first, then performance. The card count within a high-speed interconnect domain is a hard constraint; cross-domain deployment causes a cliff in performance.
    3. Clarify the caliber before reporting a budget. RMB 8 million is “can run”; RMB 30 million is “can be used.” Always make the caliber explicit in external communication.
    4. Calculate three years before signing the contract. Electricity, labor, expansion, and depreciation together are the real cost.

    Open-source large models are rapidly closing the capability gap with closed-source ones, but “free weights” is not the same as “free deployment.” Getting the budget right, choosing the right architecture, and staffing the operations team well are the prerequisites for enterprises to make good use of open-source large models.