How to Estimate the Cost of Deploying Trillion-Parameter MoE Models Locally

    This article explains how to estimate the cost of deploying trillion-parameter Mixture-of-Experts (MoE) models on-premises. It shows why parameter volume alone can't determine deployment cost, breaks requirements into three tiers—minimum viable, single-user usability at 25 token/s, and production-grade performance at 10 concurrency—and maps each tier to its dominant bottleneck and cost range. It also presents a structured cost estimation model, five practical strategies for reducing cost, and a recommended workflow of defining SLOs before hardware and budget.

    1. Why You Can’t Estimate Local MoE Deployment Cost from Parameter Size Alone

    When teams evaluate the cost of local deployment, the first instinct is often to divide the model’s parameter count by available GPU memory to decide “whether we have enough cards.” For ultra-large-scale MoE (Mixture-of-Experts) models, this estimate is far from sufficient. MoE models are characterized by a very large total parameter count, but each token only activates a small subset of experts. Take a typical ultra-large-scale MoE scenario as an example: the total parameter count can reach 2.8 trillion, with 896 experts, of which each token activates 16 experts—an expert activation ratio of roughly 1.79%. This structure introduces two key differences:

    1. VRAM usage must be calculated from total parameters
      Although inference only activates a few experts, all expert weights must still be addressable, loadable, and schedulable. In other words, the routing mechanism decides “how much you compute,” while storage capacity decides “whether it can be loaded at all.” If not all experts can be held, inference degrades and may even fail to start.

    2. Compute consumption depends only on the activated part, but communication cost is significantly amplified
      Once each token is routed to a few experts, the system must send activations to the hardware where those experts reside and retrieve the results after computation. The more experts there are and the more widely they are distributed, the more pronounced the communication overhead. If inter-card interconnect, inter-node networking, or host memory bandwidth is insufficient, GPU compute is consumed waiting on communication.

    Therefore, local deployment cost is not a single hardware-price question. It is a systems-engineering problem determined jointly by weight storage, VRAM/memory capacity, inter-card communication, expert scheduling, quantization precision, concurrency, and operational complexity.


    2. Three Tiers of Goals Lead to Completely Different Cost Ranges

    Before discussing “how much it will cost,” it helps to define the goal first. Common local deployment requirements can be broken into three tiers:

    1. Minimum viable: it just needs to run, speed doesn’t matter
    2. Single-user usable: single-user output speed reaches around 25 token/s
    3. Production viable: with 10 concurrent users, per-user output speed still stays above 25 token/s, and quantization quality stays above an acceptable threshold

    These three tiers don’t scale linearly—they rise in steps. The higher the tier, the greater the demands on VRAM capacity, bandwidth, communication-domain size, and scheduling capability, and the faster costs escalate.


    3. Tier One: Just Running It—the Core Cost Is “Fitting the Model”

    The first tier has a very clear goal: get the model fully loaded and able to output tokens. Speed requirements are low here; a few tokens per second, or even single-digit tokens per second, is acceptable.

    3.1 VRAM and Memory Lower Bound

    If the model has 2.8 trillion total parameters and uses 4-bit weight quantization, the bare weight capacity lower bound is:

    1. 2.8 trillion × 4 bit / 8 = 1.4 TB

    But a real deployment can’t be estimated from bare weights alone. Low-bit quantization typically introduces group scaling factors, alignment padding, metadata, runtime buffers, KV cache, and other overhead. If each 32 weight elements carry one 8-bit scaling factor, that adds about 0.25 bit per parameter on average, bringing capacity to:

    1. 2.8 trillion × 4.25 bit / 8 ≈ 1.49 TB

    And that’s just the weights themselves. For stable operation, you also need to reserve space for:

    • KV cache: depends on context length, concurrency, and batch size;
    • Inference runtime buffers: for activations, temporary tensors, and communication buffers;
    • Framework overhead: memory pools, handles, alignment regions;
    • Safety margin: it’s recommended to reserve an additional 20%–40% of available capacity.

    So the actual usable storage requirement for the first tier is usually not 1.4 TB but closer to 2 TB or more. If it relies entirely on accelerator VRAM, it needs a compute node built from multiple high-capacity accelerators; if it uses a CPU memory offloading approach, it needs at least one high-memory server plus a small number of accelerators for the key computation.

    3.2 Two Typical Routes

    Route A: All-Accelerator VRAM Deployment

    Advantages:

    • Data stays in high-bandwidth storage as much as possible;
    • Expert scheduling paths are short;
    • Relatively better latency.

    Disadvantages:

    • High hardware cost;
    • High demands on chassis form factor, power, cooling, and communication topology;
    • If VRAM capacity is insufficient, it won’t start.

    Applicable scenarios:

    • When there’s a plan for later performance upgrades;
    • When the team is willing to pay more for hardware to get lower latency;
    • When the model needs long-term, high-frequency use.

    Route B: CPU Large Memory + Accelerator Heterogeneous Offloading

    This route keeps the compute-intensive parts on accelerators and offloads the sparse expert weights—which make up the bulk of storage—to CPU memory. At runtime, expert weights are loaded dynamically based on routing results.

    Advantages:

    • More easily meets the minimum “does it fit” requirement;
    • Leverages the capacity advantage of high-memory servers;
    • Requires relatively fewer accelerators.

    Disadvantages:

    • Usually slower;
    • Highly sensitive to memory bandwidth, CPU scheduling, and expert caching strategy;
    • Tail latency may fluctuate significantly.

    Applicable scenarios:

    • Validating feasibility;
    • Low-frequency use;
    • Internal research and testing/evaluation;
    • Limited budget but wanting to get the pipeline working first.

    3.3 Where the First Tier’s Cost Goes

    Cost at this tier goes mainly to:

    • High-memory servers;
    • A small or moderate number of accelerators;
    • Memory capacity expansion;
    • Basic cooling and power;
    • Inference framework adaptation and debugging effort.

    Roughly estimating with common market configurations, this tier typically falls into the range of tens of thousands to over a million yuan. The exact figure depends on accelerator capacity, memory capacity, whether a fully integrated system is delivered, and whether redundancy is needed. At this stage, don’t chase extreme performance—prioritize validating model loading, routing correctness, and output stability.


    4. Tier Two: Single User at 25 token/s—Cost Begins to Be Driven by Bandwidth

    When the goal shifts from “it runs” to “a single user reaches 25 token/s,” the bottleneck moves from capacity to bandwidth and scheduling efficiency.

    4.1 Why a Speed Target Significantly Raises Cost

    In MoE inference, every token goes through routing, expert selection, expert computation, and result merging. Although only 16 experts are activated, those experts may be spread across different accelerators, different memory regions, or even across nodes. If data movement is frequent, GPU compute units will often sit idle waiting for data.

    Key factors affecting speed include:

    • Expert weight read bandwidth;
    • Inter-accelerator interconnect bandwidth;
    • Transfer bandwidth between CPU memory and accelerators;
    • KV cache read/write efficiency;
    • Routing/scheduling overhead;
    • Batching and concurrency strategy.

    If the tier-one solution relies on CPU memory offloading, expert weights must be moved from CPU memory to the compute device every time they’re used, so speed is limited by host memory bandwidth and interconnect link bandwidth. Reaching 25 token/s usually requires increasing the proportion of expert weights that reside in high-bandwidth media.

    4.2 Common Optimization Approaches

    Increasing Accelerator VRAM Capacity

    Placing more hot experts directly in accelerator VRAM reduces the frequency of loading from CPU memory. The larger the VRAM, the more experts can be cached and the faster routing hits.

    Expert Affinity Caching

    Speed isn’t only about adding hardware. By tracking hot experts, you can keep frequently called experts resident in high-bandwidth regions first. For example:

    1. hot expert set = experts frequently routed in the last N requests
    2. cold expert set = experts that can be offloaded to CPU memory or local high-speed SSD

    This can improve the actual hit rate at relatively low cost.

    Routing Prefetch

    If requests have continuity, or session context follows patterns, you can predict which experts the next token is likely to activate and load the weights in advance. It won’t guarantee a 100% hit rate, but it can effectively reduce waiting time.

    Improving Communication-Domain Quality

    For scenarios requiring multi-card deployment, inter-card interconnect quality matters a lot. If communication latency is high, even with ample compute you’ll see slow output and noticeable jitter. When deploying, pay attention to:

    • Intra-node interconnect topology;
    • Inter-node network latency;
    • Whether a unified communication domain is formed;
    • Communication hotspots caused by routing concentration;
    • Whether expert distribution is balanced.

    4.3 Where the Second Tier’s Cost Goes

    The second tier adds cost mainly in:

    • Higher-capacity accelerators;
    • More accelerators;
    • Higher-quality host interconnect;
    • Faster network equipment;
    • More complex scheduling and caching strategy development;
    • More detailed performance-tuning effort.

    This tier typically starts in the low millions of yuan and rises further if you demand stable low latency or high throughput. It’s no longer just about “getting it to start”—it must continuously satisfy single-user interactive experience.


    5. Tier Three: Holding 25 token/s at 10 Concurrency—Production Cluster Cost

    The third tier is the most complex: it demands concurrency, per-user speed, and controlled quantization quality at the same time. This is no longer something a single server can easily solve—it’s close to a small inference service system.

    5.1 Concurrency’s Pressure on VRAM

    In single-user inference, KV cache is relatively small; as concurrency rises, each request occupies its own context cache. 10 concurrent requests means KV cache, runtime buffers, and communication buffers all grow proportionally.

    A rough estimate:

    1. total capacity = weights + per-request KV cache × concurrency + runtime buffer + communication buffer + safety margin

    With long contexts, KV cache can become the second-largest memory consumer after weights. At that point, focusing only on expert weights is not enough—you must also evaluate:

    • Maximum context length;
    • Average request length;
    • Concurrency peak;
    • Batching strategy;
    • KV cache eviction mechanism;
    • Layered use of VRAM and memory.

    5.2 Concurrency’s Pressure on Communication

    In single-user scenarios, expert routing is relatively linear. Under concurrency, multiple requests trigger different expert combinations simultaneously, which can cause:

    • Routing hotspots;
    • Communication-domain congestion;
    • Uneven expert load;
    • Severe queuing on some accelerators;
    • Significantly higher tail latency.

    To hold 25 token/s per user at 10 concurrency, routing, expert distribution, and communication topology must stay balanced. This typically requires:

    • More accelerators;
    • More balanced expert sharding;
    • A more efficient scheduler;
    • More stable high-speed networking;
    • More complete monitoring and alerting;
    • More mature failover and degradation strategies.

    5.3 Quantization Quality Can’t Be Ignored

    The third tier requires quantization quality to stay above an acceptable standard. This is critical. Low-bit quantization reduces storage cost but may introduce precision loss. When deploying, establish acceptance criteria, such as:

    • Whether accuracy drops on key tasks;
    • Whether long-form generation shows repetition or breaks;
    • Whether code, math, and logical reasoning degrade;
    • Whether output stability drops under high concurrency;
    • Whether result distributions shift noticeably before vs. after quantization.

    “Whether it runs” can’t be the only acceptance criterion. In production, quantization strategies must pass regression testing on business samples.

    5.4 Where the Third Tier’s Cost Goes

    The third tier adds cost mainly in:

    • Multiple high-spec servers;
    • High-bandwidth, low-latency networking;
    • A more capable cluster scheduling system;
    • Monitoring, logging, alerting, and stress-testing infrastructure;
    • Model caching and request-routing optimization;
    • Long-term operations and stability assurance.

    This tier is usually not a single-device purchase but a system investment in the range of several million to tens of millions of yuan. The exact figure depends on whether you pursue extreme stability, whether you build your own data center, whether redundancy is needed, and whether you need to serve high-concurrency requests long-term.


    6. Cost Estimation Model: Don’t Just Ask “How Much”—Ask “Which Resource Is Most Expensive”

    To estimate budget more accurately, use a structured formula like the following:

    1. total cost = accelerator cost
    2. + host and CPU memory cost
    3. + high-speed network cost
    4. + storage cost
    5. + power and cooling cost
    6. + software optimization and labor cost
    7. + operations and monitoring cost

    Broken down further into capacity formulas:

    1. deployable total capacity = total accelerator VRAM + usable CPU memory + usable high-speed cache
    2. minimum capacity requirement = weight capacity + KV cache capacity + runtime buffer + communication buffer
    3. recommended safety margin = 20%–40%

    If heterogeneous offloading is used, bandwidth constraints also come into play:

    1. theoretical speed ceiling ≈ expert weight read bandwidth / average bytes read per token
    2. actual speed ceiling = theoretical speed ceiling × scheduling efficiency × communication efficiency × cache hit rate

    This formula shows that increasing memory capacity is only the first step—what really determines speed is effective bandwidth. If expert weight loading can’t keep up, no amount of compute will reach the target token/s.


    7. How to Reduce Local Deployment Cost: Five Practical Directions

    7.1 Prioritize Quantization, but Control Precision Regression

    4-bit and 8-bit mixed quantization are important means of reducing weight capacity. But lower isn’t always better—especially for complex reasoning, code generation, and long-form tasks, prioritize output quality. Recommended process:

    1. Compare before and after quantization on a small sample set;
    2. Choose representative business tasks;
    3. Evaluate error rate, latency, repetition rate, and logical consistency;
    4. Then decide the final quantization scheme.

    7.2 Expert Hotness Tiering

    In MoE models, different experts are called at different frequencies. Experts can be split into:

    • Hot experts: resident in high-bandwidth accelerator VRAM;
    • Warm experts: resident in host memory;
    • Cold experts: resident in local high-speed SSD or loaded on demand.

    This balances capacity and speed.

    7.3 Request-Level Routing Optimization

    Don’t only look at the model’s internal routing—also look at service-layer request scheduling. You can:

    • Route session requests to affinity instances;
    • Replicate high-frequency experts;
    • Prevent a single expert from becoming a hotspot;
    • Put long-context requests in a separate queue;
    • Give short requests a fast lane.

    7.4 Control Concurrency and Batching Strategy

    Batching improves throughput but increases VRAM usage and latency. For interactive scenarios, don’t blindly pursue large batches. Adjust dynamically based on SLO:

    1. low-latency scenario: small batch, prioritize single-request speed
    2. high-throughput scenario: large batch, prioritize overall throughput

    7.5 Establish a Performance Baseline

    Before deployment, run benchmarks covering at least:

    • Single-request time-to-first-token;
    • Single-request output speed;
    • 10-concurrency output speed;
    • Long-context performance;
    • Expert routing distribution;
    • VRAM/memory peak;
    • Communication wait time;
    • Failure rate and recovery capability.

    Without a baseline, you can’t tell whether cost optimization is effective.


    8. Practical Recommendations: Define SLO First, Then Hardware, Then Budget

    When deploying ultra-large-scale MoE models, many teams fall into the trap of “buy hardware first, then figure out how to use it.” A more sensible path is:

    1. Define business SLOs
      For example: must a single user hit 25 token/s? Must it support 10 concurrency? What’s the maximum context length? Is quantization precision loss acceptable?

    2. Do the capacity math
      Compute weights, KV cache, runtime buffers, and safety margin to determine the minimum deployable boundary.

    3. Choose the deployment form
      If you just need to get it running, prioritize a high-memory server with heterogeneous offloading; if you want interactive experience, raise accelerator capacity and interconnect quality.

    4. Run a small-scale benchmark
      Under real or near-real load, validate routing, communication, cache hit rate, and output quality.

    5. Evaluate total cost
      Look beyond procurement cost—also factor in power, cooling, operations, optimization labor, and stability.

    6. Set degradation strategies
      For example: reduce context length, cut concurrency, switch to a lower-precision model, or throttle high-frequency requests—so the service doesn’t collapse under peak load.


    9. Conclusion: The Essence of Local Deployment Cost Is “Paying for Experience”

    Deploying ultra-large-scale MoE models locally isn’t simply “buy cards and load a model.” Different goals map to different cost tiers:

    • Just running it: the focus is capacity; cost lies mainly in memory, servers, and basic accelerators;
    • Smooth single user: the focus is bandwidth and scheduling; cost lies mainly in high-capacity accelerators, interconnect quality, and optimization capability;
    • Stable concurrency: the focus is clustering, networking, monitoring, and scheduling; cost approaches a production-grade inference platform.

    So before answering “how much will it cost,” you must first answer three questions:

    1. Do you need “it runs,” or “it’s usable”?
    2. Is your bottleneck VRAM capacity, or expert read bandwidth?
    3. Where’s your acceptable lower bound for quantization quality?

    Only when these are clear can you spend your budget on the most critical resources. For most teams, the safest approach isn’t to jump straight to the highest configuration, but to start by defining SLOs and then, through capacity estimation, small-scale stress testing, and phased optimization, find the best balance between performance and cost.