How to Calculate the Cost of On-Premises Deployment for Trillion-Parameter MoE Models

    This article breaks down how to estimate the cost of deploying trillion-parameter Mixture-of-Experts (MoE) models on-premises. It explains why the budget is a range rather than a single number, the distinction between total and activated parameters, a five-step VRAM estimation method, decode-throughput and interconnect bottlenecks, a three-year TCO model with a worked example, common costing mistakes, software-hardware co-tuning strategies, and a phased rollout checklist. It helps readers build a reproducible, verifiable cost model instead of chasing a single "starting price."

    1. The Short Answer: Your Budget Is a Range, Not a Single Number

    When enterprises see open-weight trillion-parameter MoE models, they tend to ask the same question: how do you calculate the cost of deploying a trillion-parameter MoE model on-premises? If you focus only on “how big are the weights,” you reduce the problem to “how much VRAM do I need.” But the real cost of private deployment is not a single number — it’s a range shaped by VRAM capacity, concurrency, context length, interconnect bandwidth, utilization, data-center conditions, and the software platform.

    More precisely, deploying a trillion-scale sparse MoE model privately is usually not a question of “can one machine run it?” but rather “how many concurrent requests must it serve reliably, how long a context must it support, and what service level must it meet?” Depending on the assumptions, the cost can range from the single-digit-millions of RMB to the tens of millions — sometimes several times apart.

    Deployment positioning Typical goal Reference hardware scale* Investment magnitude*
    Functional validation Get the inference pipeline working; validate results and security/compliance When a single machine can’t hold it, at least 3 8-GPU nodes or equivalent VRAM, using multi-node expert/tensor parallelism; compute virtualization can’t replace VRAM capacity Millions of RMB
    Department-level pilot A team of dozens; concurrency 10–50 ~4 nodes, scaling up for long context or high concurrency Early tens of millions of RMB
    Production service High-concurrency online service, high availability and elastic scaling ~8 nodes to start, usually needing rack-scale interconnect and multi-replica redundancy Tens of millions of RMB, growing with scale

    *Reference scale assumes 8 GPUs per node with ~80GB VRAM per card. The “investment magnitude” in the table is illustrative: by default it covers only compute hardware procurement and essential networking; it excludes electricity, operations, software platforms, storage, data-center retrofit, redundancy, and spares that belong in a three-year TCO. For a full three-year TCO, add those separately. Actual costs vary widely with hardware market conditions, data-center constraints, concurrency, context length, and how you scope the build.

    One point deserves special emphasis: compute virtualization can partition compute, but it cannot replace VRAM capacity. A large share of the expert weights in a trillion-scale MoE model must stay resident in VRAM at all times. Even if the business only wants a small-scale proof of concept, as long as a single machine can’t hold the full weights, you must deploy across multiple nodes.


    2. The Underlying Logic: Total Parameters Determine VRAM, Activated Parameters Determine Compute

    To estimate the on-premises cost of a trillion-parameter MoE model, start by separating two concepts: total parameters and activated parameters.

    In a dense model, every token passes through all parameters, so total parameter count, VRAM usage, and compute are tightly correlated. A sparse MoE model works differently: each layer has multiple expert subnetworks, and the gating network selects only a few experts to compute for each token. Many parameters sit idle, and only a subset is activated per token.

    This leads to a key “law of separation”:

    How many cards you buy depends on total parameters; how fast it runs depends on activated parameters.

    Specifically:

    • VRAM side: No matter how many experts each token activates, all expert weights must stay resident in VRAM — otherwise any token routed to any expert could fail. So VRAM demand is driven mainly by total parameters.
    • Compute side: Each token only reads and computes the experts it activates, so decode compute and VRAM-bandwidth pressure are driven mainly by activated parameters.

    If a trillion-scale model has an activation ratio around 10–15%, the total parameter count might reach 2.8 trillion, yet the parameters actually involved in computing each token might be only around 300 billion. That’s why you can’t estimate speed from total parameters alone, nor VRAM from activated parameters alone.

    2.1 Native Low-Precision Weights: The Starting Point for VRAM Estimation

    To shrink weight size, trillion-scale MoE models are usually released in a native low-precision format. Using a common assumption: 32 elements share one 8-bit scaling factor, so each parameter averages about 4.25 bits — 4 bits of weight plus 0.25 bits of scaling overhead.

    Estimating a total of 2.8 trillion parameters:

    2.8×10¹² × 4.25 ÷ 8 ≈ 1.49×10¹² bytes ≈ 1.49 TB

    This is roughly consistent with the publicly reported sharded weight total of about 1.56 TB. With FP16, the weight size is about 5.2 TB; with 8-bit, it’s also close to 3 TB. The low-precision format is what makes a trillion-scale model able to run on a limited number of machines.

    But there’s a hidden cost here: the hardware must natively support this kind of low-precision compute format. If it doesn’t, the inference engine may have to temporarily dequantize weights back to higher precision, or fall back to a less efficient low-precision compute path. The result can be higher VRAM usage, lower throughput, and a forced increase in node count. So when selecting hardware, don’t look only at peak single-card compute — look at how well the hardware natively supports the model’s weight format.

    2.2 Hybrid Attention: Long Context Doesn’t Explode, but It Isn’t Free

    These models often use a hybrid attention structure — for example, a combination of gated linear attention and sparse attention. The former compresses history with a state matrix; the latter keeps only a few key pieces of information for precise computation. This reduces attention overhead at long context, so “supporting long context” doesn’t immediately blow up VRAM.

    But that doesn’t mean long context is free. Every active request must hold its own attention state or KV cache. The higher the concurrency and the longer the context, the larger this usage. So capacity estimation can’t look only at weights — you also have to account for:

    Context capacity = average context length × number of concurrent requests

    In long-context scenarios, the KV cache and attention states are the “second block of VRAM” — and the part most easily underestimated in budget calculations.

    2.3 Multi-Token Prediction: A Throughput Lever with a Small VRAM Ledger

    Multi-token prediction (MTP) lets the model predict multiple subsequent tokens at one position; during inference, it can serve as a draft model paired with speculative decoding. For suitable tasks, it can significantly speed up single-request generation.

    Its essence is raising the output per single weight read: in the decode phase, reading the weights once on a GPU usually produces just one token, leaving compute largely idle; if you can verify multiple tokens at once, hardware output is amplified. But the MTP module itself has parameters that must be counted in the VRAM budget — though its share is usually small.

    2.4 The Four-Quadrant Resource Bottleneck

    Dimension Dense model Trillion-scale sparse MoE
    Main VRAM driver Total parameters Total parameters + KV cache/states + communication buffers
    Main compute/bandwidth driver Total parameters Activated parameters
    Key network capability Mostly tensor-parallel communication Mostly All-to-All communication, sensitive to latency jitter
    Scaling method Adding nodes scales near-linearly All-to-All latency rises across nodes; efficiency loss exists

    So the on-premises cost of a trillion-scale sparse MoE is essentially determined by VRAM capacity × interconnect bandwidth, not by a single-card peak-compute number.


    3. VRAM Estimation: Working Backward from Business Goals to the Number of Machines

    Recommend a five-step approach: define SLOs, estimate VRAM, assess throughput, plan interconnect, and fill in platform capabilities.

    3.1 Step 1: Define Business Goals and Write Down the SLOs First

    Many budgets go wrong because business goals aren’t quantified. Recommend clarifying at the project-initiative stage:

    • Concurrency: daily active users and peak concurrent requests;
    • Context distribution: are most requests within 4K, or is there a lot of 32K, 128K, or even longer context;
    • Latency requirements: targets for time to first token (TTFT) and time per output token (TPOT);
    • Availability: whether cross-rack or cross-availability-zone disaster recovery is needed;
    • Compliance requirements: whether data can leave the domain, whether full audit logging is needed;
    • Budget scope: hardware procurement only, or three-year TCO; whether networking, storage, software platform, data-center retrofit, and operations are listed separately;
    • Build timeline: hardware lead time, data-center power capacity, floor loading, cooling capacity, and network cabling readiness;
    • Storage and network: weight-loading speed, model-repository capacity, and whether high-speed network cards and switches are in the budget.

    If these assumptions aren’t aligned, you’ll easily end up with a “minimum config” and a “recommended config” that differ by several times.

    3.2 Step 2: Estimate VRAM Demand with the Five-Item Method

    Private-deployment VRAM demand breaks into five parts:

    VRAM demand = weights + KV cache/attention states + communication & runtime buffers + runtime overhead + safety margin

    Component Meaning Estimation notes
    Weights All experts, attention layers, embedding layers, etc. Based on actual sharded-file size
    KV cache/states Attention states for all concurrent requests Related to layers, KV-head count, context length, concurrency
    Communication & runtime buffers All-to-All, attention operators, custom memory pools, etc. Reserve several GiB per card; can aggregate to tens of GiB across cards
    Runtime overhead Compute graphs, framework-resident memory, CUDA Graph, etc. Small share, but can’t be ignored
    Safety margin Prevent OOM, handle traffic spikes Recommend reserving 15–20%

    If you can’t get accurate KV-head counts, layer counts, and head dimensions, you can estimate with an order-of-magnitude formula:

    Bytes per token of KV ≈ 2 × layers × KV heads × head dimension × bytes per element

    where K and V each account for one share. If the initial estimate is about 320 KB per token, then a single 32K-context request uses about 10 GiB of VRAM; with 64 concurrent requests, KV cache alone approaches 640 GiB. Adding the ~1.5 TB of weights, communication buffers, and safety margin, the node count will be well above the theoretical minimum of “weights only.”

    So capacity estimation must use “context length × concurrency,” not just weight size.

    3.3 Step 3: Use Utilization as a Ceiling, Not a Target

    Theoretical calculations usually assume 100% VRAM utilization, but engineering practice must reserve room for fragmentation and traffic spikes. A common recommendation is to keep VRAM utilization around 85–90%. So after rounding the node count up, you often need to add one more.

    Estimating with 8 GPUs per node and ~80GB per card, one node has about 640GB of VRAM. Roughly:

    Weight size Minimum nodes (weights only) Realistic starting point (with normal concurrency)
    ~1.5 TB 3 ~4, scaling up for long-context/high-concurrency
    ~3 TB 5 ~6
    ~5 TB 8 9–10

    This is just an order-of-magnitude table. A real budget still has to be recomputed with actual weight size, context length, concurrency, and measured throughput.


    4. Decode Throughput, Interconnect, and Network: The Hidden Big-Ticket Items Behind the Cost

    4.1 The Decode Phase Is Memory-Bound, Not Just a Matter of Piling on Compute

    To generate each token, the decode phase must read the activated weights. Compute units spend most of their time waiting for data, so single-request speed has a near-theoretical ceiling:

    Single-request speed ceiling ≈ effective VRAM bandwidth ÷ weight bytes read per token

    Suppose the activated parameters are about 300 billion, stored in a ~4-bit format, so each step reads about 150GB; if the experts are spread across 8 cards with an aggregate bandwidth of about 20TB/s, the theoretical ceiling is about 130 tokens/s. Real implementations usually reach only 50–70% of the theoretical value, i.e., about 60–90 tokens/s, which is roughly enough for an interactive experience.

    As concurrency rises, batching lets one weight read serve multiple requests, raising overall throughput. But as concurrency climbs further, total bandwidth and compute get progressively saturated, showing “slower per request, plateauing overall throughput.” So the decode-side bottleneck usually isn’t insufficient compute, but:

    • KV cache/attention-state capacity: determines how many requests can be held at once;
    • VRAM bandwidth: determines how many tokens each step can produce.

    4.2 Prefill Isn’t Simply Cheaper if You Split It into Small Pools

    A common past idea was to use a few high-end cards for prefill and a large number of ordinary cards for decode. But with a trillion-scale MoE model, all weights must stay resident in VRAM, and prefill nodes themselves must hold the full model, so the minimum scale is locked by the VRAM threshold. A more realistic approach is a unified resource pool with dynamic scheduling by load.

    When a long document is fed in all at once, TTFT lengthens noticeably. You need:

    • Chunked prefill: split long prefill into small chunks interleaved with other requests;
    • Prefix reuse: cache computed results for fixed prefixes such as system prompts, code-repository context, and knowledge-base snippets.

    4.3 The Network Is More Likely to Be the Bottleneck Than “Not Enough Cards”

    MoE expert parallelism pushes All-to-All communication onto the critical path. At every step, tokens are sent to the card hosting the target expert, and results are collected back. The more experts and the more cross-node traffic, the more easily communication latency is amplified.

    The node form factor often affects cost more than the node count:

    Node form factor Interconnect characteristics Suitable scenarios
    General 8-GPU GPU server Good intra-node interconnect; cross-node depends on external high-speed network Small-to-medium scale, multi-model co-location
    Rack-scale integrated node Low full-mesh latency, high bandwidth; supports expert parallelism over a larger scope Trillion-scale MoE, high-concurrency production

    If the machine count goes up but the machines are scattered across multiple ordinary racks and can only interconnect via the external network, cross-node All-to-All easily becomes a performance bottleneck — the “plenty of machines, low throughput” situation. This is one of the fundamental reasons why the “recommended config” is often well above the “minimum config.”

    Three rules of thumb for network planning:

    • High-speed NICs must be in the budget. NICs with optical modules, plus switches and port counts, are often underestimated and can account for 10–15% of the total hardware budget in large clusters.
    • Prefer “fewer but stronger” over “more but weaker.” Integrated nodes trade scale for efficiency and suit throughput-first scenarios; small-to-medium deployments can start with general-purpose nodes.
    • Storage mustn’t hold things back. Loading 1.5TB-scale weights on ordinary storage can take tens of minutes; high-speed parallel storage can compress that to minutes. For a platform with frequent canary releases, this money is well spent.

    Data-center facilities are also easy to leave out. A fully loaded 8-GPU server typically draws 6–10kW; rack-scale nodes draw more. At 10kW full load and 8,760 hours a year, one server uses about 87,600 kWh a year, with electricity costing from tens of thousands to over a hundred thousand RMB; add cooling, power redundancy, and floor-loading retrofit, and an older data center may “have power but no space.”


    5. Cost Calculation: Turning a Few Machines into a Clear Ledger

    5.1 The TCO Model: Three-Year Total Cost of Ownership

    Don’t look only at the purchase amount; use a three-year TCO:

    TCO (3 years) = hardware procurement + software & platform + data-center retrofit + 3 years of electricity & cooling + 3 years of operations labor + expansion reserve

    Cost category Main content Share reference
    Compute hardware GPU servers/integrated nodes, CPU and memory 50–65%
    Network equipment High-speed NICs, switches, optical modules and cables 10–15%
    Storage equipment Parallel/object storage, backup 5–10%
    Data-center facilities Power, cooling, rack retrofit 5–10%
    Software & platform Inference engine, scheduling, observability, security Depends on build vs. buy
    Operations (3 years) Electricity, labor, maintenance Comparable to or higher than hardware procurement

    Beyond hardware procurement, the following items are most easily left out:

    • Redundancy and failure cost: the larger the cluster, the more common hardware failures; N+1 redundancy, a spares pool, and fault isolation all need budget.
    • Software-adaptation labor: from model release to stable launch, operator adaptation, communication optimization, and stress-test tuning usually take weeks to months.
    • Electricity: at 10kW per server, 3 years, and RMB 0.8/kWh, one server costs about RMB 210,000 in electricity; an 8-node cluster’s electricity alone can exceed RMB 1.6 million.
    • Model-iteration cost: open-source model versions update frequently, and changes in architecture or quantization format may require re-adaptation.

    5.2 A Simplified Example: How the Budget Is Built

    Suppose a team builds a department-level private inference platform:

    • Config: 4 general-purpose 8-GPU nodes, ~80GB per card; 100Gbps-class RDMA network; 50TB usable parallel storage;
    • Hardware procurement: ~RMB 1.5 million per node, ~RMB 6 million for 4 nodes;
    • Other one-time costs: network ~RMB 400,000, storage ~RMB 300,000, software & platform ~RMB 800,000, totaling ~RMB 7.5 million;
    • Annual operating cost: electricity ~RMB 280,000/year, operations labor ~RMB 600,000/year, maintenance ~RMB 300,000/year, totaling ~RMB 1.2 million/year.

    From this:

    • Build-period investment: ~RMB 7.5 million;
    • Three-year TCO: RMB 7.5 million + RMB 1.2 million/year × 3 = RMB 11.1 million;
    • Per-year cost after depreciation: RMB 6 million of hardware depreciated straight-line over 3 years ≈ RMB 2 million/year, plus ~RMB 1.2 million/year operating ≈ RMB 3.2 million/year.

    If annual token consumption is 50 billion, i.e., 50,000 million tokens:

    Cost per million tokens = RMB 3.2 million ÷ 50,000 million tokens = RMB 64 per million tokens

    If actual utilization is only about 50%, annual token consumption drops to 25 billion, i.e., 25,000 million tokens, and the unit cost rises to:

    RMB 3.2 million ÷ 25,000 million tokens = RMB 128 per million tokens

    This example shows: the break-even point of private deployment isn’t “can the model run” but “can utilization be maxed out.” Utilization is the most sensitive and most easily underestimated variable in the TCO model.

    5.3 Sensitivity Analysis: Which Three Variables Matter Most

    Variable Impact mechanism Impact level
    Concurrency × average context length Directly determines KV cache/attention states, which determines node count Highest; can change node count by multiples
    Hardware form factor Determines cross-node communication efficiency and single-rack ceiling High; affects unit cost and scaling method
    Hardware market conditions Prices fluctuate notably when accelerator supply is tight High and uncontrollable

    Conversely, the following two have relatively limited impact on the total budget:

    • Total model parameters: weights are already compressed to the TB scale and are no longer the sole bottleneck;
    • Single-card peak-compute number: the decode phase is memory-bound, so simply piling on compute yields limited returns.

    5.4 Decision Framework: Build, Rent, or Hybrid

    Option Suitable scenarios Main risk
    Fully self-built Sensitive data, stable predictable load, heavy usage High upfront cost; unit cost spikes when utilization is low
    Fully rented Short-term validation, volatile usage, no GPU ops team Long-term high-frequency usage accumulates cost; data-egress compliance risk
    Hybrid deployment Most enterprises’ realistic choice Needs unified scheduling and a routing layer to avoid two siloed systems

    For most enterprises, hybrid deployment is the more prudent starting point:

    • Self-hosted baseline: put predictable routine workloads on your own cluster — e.g., internal coding assistants, document processing — and try to max out utilization;
    • Cloud for peaks: overflow burst traffic, ultra-long contexts, and offline batch jobs to elastic public-cloud compute, paying for actual usage;
    • Sensitive data stays in-domain: core code and customer data go only through the local link.

    Before building your own, recommend first validating results and cost with a public-cloud managed inference service or API, focusing on weight-format support, concurrency SLOs, data compliance, and cost per token — to avoid “buy the cards first, experiment later.”


    6. Common Mistakes and Cautions

    In on-premises deployment of trillion-parameter MoE models, the following mistakes most easily distort the budget:

    1. Only asking whether it can load, not how much concurrency it supports. Running is just the minimum bar, not a production standard.
    2. Looking only at weight size, not at KV cache/attention states. Long context and high concurrency significantly amplify VRAM demand.
    3. Assuming compute virtualization can replace VRAM capacity. Weights must stay resident in VRAM; virtualization can’t conjure VRAM out of nothing.
    4. Counting only GPUs, not the network. All-to-All communication, high-speed NICs, switches, and optical modules can all become bottlenecks.
    5. Counting only procurement, not TCO. Electricity, operations, storage, data-center retrofit, software platforms, and model iteration should all be included.
    6. Computing unit cost only at full load. When utilization is low, the cost per million tokens can double.
    7. Ignoring compliance cost. Licenses, content safety, supply-chain verification, and audit-log storage should all be planned in advance.

    7. Software-Hardware Co-Tuning: Squeezing Down Cost with Engineering

    With the same hardware, throughput before and after tuning can differ by 2x. For a trillion-scale MoE, focus on four layers.

    7.1 Operator Layer: Actually “Compute” in Low Precision

    Low-precision weights must not only “fit” but also compute efficiently. You need matrix-multiply operators optimized for the low-precision format so the hardware computes natively in low precision; and custom operators for the hybrid attention structure to reduce state-update and KV-access overhead.

    7.2 Communication Layer: Make All-to-All “Run Along the Topology”

    Under expert parallelism, communication is often the critical path. You can use topology-aware placement to co-locate frequently co-occurring experts and reduce cross-node communication; use communication-computation overlap to hide communication latency inside computation; and use capacity factors and expert bias to ease the long-tail latency caused by hot experts.

    7.3 Scheduling Layer: Decouple and Combine on Demand

    You can separate prefill from decode and configure parallelism for each: prefill emphasizes compute, decode emphasizes bandwidth. With high-speed networking, let the two use different parallelism groups to avoid competing for interconnect bandwidth; and dynamically scale decode instances with real-time load, giving idle resources to offline jobs.

    7.4 Service Layer: Trade Caching and Speculation for Throughput

    Do KV reuse for fixed prefixes such as system prompts, code context, and knowledge-base snippets; use multi-token prediction to generate drafts and verify multiple tokens per weight read; and dynamically adjust batch size by business latency requirements rather than always going bigger.

    7.5 Observability: Make Every Unit of Compute Investment Visible

    Recommend at least monitoring TTFT, TPOT, output throughput, VRAM utilization, KV cache/prefix hit rate, expert-routing balance, and GPU power and temperature, with alerts. Without an observability system, cost optimization easily becomes guesswork.


    8. Rollout Checklist: From Validation to Scale

    Phase Key tasks Deliverables
    Validation Validate results, compliance, and unit cost with public cloud or small-scale private deployment Feasibility report, preliminary cost model
    Pilot Use hybrid deployment to carry stable baseline traffic; get monitoring and operations working Usable inference service, SLO report
    Scale Once business volume is stable, expand the self-hosted cluster using the TCO model Capacity model, cost dashboard, scaling plan

    You can use the following checklist directly to decide the next step:

    • Validate first: is the business value clear? is the cost per token acceptable? does data compliance pass?
    • Then pilot: can utilization be raised step by step? do TTFT and TPOT meet targets? can operations handle it?
    • Then scale: is business volume stable? does unit cost keep falling? is rack-scale interconnect and multi-replica redundancy needed?

    9. Conclusion: The Right Question Isn’t “How Much,” but “What Are You Buying It For”

    Back to the original question: how do you calculate the on-premises deployment cost of a trillion-parameter MoE model?

    The answer depends on three questions:

    • What are you buying it for? An internal productivity tool, an external AI service, or a domain-specific model?
    • What service level must it reach? How much concurrency, how long a context, how much latency, how high availability?
    • How do you use it without waste? Can utilization be maxed out? is elastic capacity in place? can cost per token keep falling?

    For most enterprises, the more pragmatic path is: first validate value with a public cloud or a small-scale private deployment, then use hybrid deployment to carry stable baseline traffic, and finally scale out the self-hosted cluster using the TCO model once both business volume and unit cost are under control.

    The arrival of trillion-scale open-source models has turned “putting a strong model in your own data center” from an idea into an engineering problem. But the answer to an engineering problem is never a string of numbers — it’s a method that can be recomputed, verified, and continuously optimized. Getting this ledger right is worth more than memorizing any “starting price.”