Trillion-Parameter MoE Local Deployment: Cost and Performance Path Assessment

    Deploying a trillion-parameter Mixture-of-Experts (MoE) model locally is fundamentally different from running an ordinary model. This article explains why memory and storage capacity are governed by total parameter count rather than sparse activation, why 4-bit quantized weights still approach 1.5TB, and how the bottleneck shifts across three deployment targets: a minimum runnable setup, single-user inference at 25 tokens/s, and production-grade service at 10 concurrent users each sustaining 25 tokens/s. It covers capacity planning, bandwidth and communication costs, quantization precision, a budget estimation framework, recommended configurations, and a staged validation process.

    1. Framing the Problem: Local Trillion-Parameter MoE Deployment Is About Placing Weights, Feeding Bandwidth, and Controlling Latency

    When a model reaches the trillion-parameter scale and adopts a Mixture-of-Experts (MoE) architecture, local deployment becomes significantly harder. Many people intuitively assume that because each token only activates a small number of experts, memory and VRAM requirements should also drop sharply in proportion to that activation ratio. This understanding is inaccurate.

    For MoE models, sparse activation mainly reduces the compute per inference, not the weight residency requirement. During inference, the model must first know where all experts’ parameters are, then route each token’s activations to the storage location of its target expert. As a result, as long as you want the model to retain full capability, all expert weights must be addressable, schedulable, and readable by the system.

    This is the first principle of local trillion-parameter MoE deployment: VRAM or memory capacity is determined first by total parameter count. Activation sparsity only affects compute consumption and communication pressure; it will not shrink weight usage down to the activation ratio.

    While the full weights and technical report have not been published, any cost estimate should be treated as an engineering projection rather than a final quote. A truly reliable budget must be verified item by item through benchmarking once the weights become loadable: weight loading time, first-token latency, per-token generation latency, long-context stability, concurrent throughput, communication overhead, memory bandwidth utilization, and so on.

    This article discusses three typical targets:

    1. Minimum target: it only needs to run, regardless of speed;
    2. Medium target: single user, output speed reaching 25 tokens/s;
    3. High target: at 10 concurrency, each user’s output speed still reaches 25 tokens/s or above, with quantization precision no lower than a reasonable floor.

    The cost differences across these targets are not simple linear growth, but a progression from “single-machine memory engineering” to “multi-card bandwidth engineering,” and further to “multi-machine communication engineering.”

    2. The Capacity Floor: Why ~1.5TB of Storage Is Still Needed After 4-Bit Quantization

    Assume a trillion-parameter MoE model has a total of 2.8 trillion parameters, and the common practice is to compress weights with 4-bit quantization. If you calculate only the ideal bare weights, the theoretical floor is:

    1. 2.8T parameters × 4 bit / 8 = 1.4TB

    In actual deployment, however, 4-bit quantization usually does not store the weights alone; it also retains scale factors or calibration parameters. For example, group-wise quantization commonly attaches an 8-bit scale factor to every 32 4-bit weight elements. This makes the effective bit width per weight element higher than 4 bits, approaching:

    1. 4 bit + 8 bit / 32 = 4.25 bit

    So the weight capacity estimate becomes:

    1. params = 2.8e12
    2. effective_bits = 4 + 8 / 32
    3. weight_gb = params * effective_bits / 8 / 1e9
    4. print(weight_gb) # ~1487.5 GB

    In other words, under 4-bit group-wise quantization, the weight body alone already approaches 1.5TB. This does not yet include:

    • Non-4-bit tensors: some embedding layers, normalization layers, gating layers, positional encodings, or special structured parameters may still retain higher precision;
    • Metadata: layer indices, expert routing tables, quantization parameters, tensor layout descriptions, etc.;
    • Alignment overhead: for memory access efficiency, actual storage often needs block alignment;
    • Runtime workspace: including temporary activations, routing buffers, communication buffers, and intermediate operator results;
    • KV cache: the longer the context and the higher the concurrency, the larger the KV cache footprint;
    • Safety margin: memory fragmentation, burst requests, and dynamic scheduling all require reserved space.

    Therefore, if the goal is a “full-capability, usable” deployment, you cannot plan based on just 1.4TB or 1.5TB; you should design for higher capacity. For a minimum runnable scenario, memory or addressable storage capacity should be provisioned starting at least at 2TB; if long context, multi-concurrency, or dynamic batching is involved, 3TB or more should be considered.

    The key conclusion here is: an MoE model’s weight storage pressure does not disappear because the activation ratio is low. Even if 896 experts each activate only a fraction per token, all of them still need full residency or on-demand scheduling. The first layer of local deployment cost is “fitting the entire model into the system.”

    3. Minimum Runnable: Getting Tokens Out Is Enough, and the Cost Focus Is Memory and Heterogeneous Offloading

    The first target is very clear: it does not care about speed, it only requires the model to load and produce results. Even a single-digit number of tokens per second is fine, as long as it runs first.

    In this scenario, the most typical approach is heterogeneous GPU-CPU inference. The attention modules in the model are usually compute-intensive and suited to high-speed compute devices like GPUs; the many sparse expert layers, though large in capacity, activate only a few experts at a time and can be placed in CPU memory or large-capacity storage, dynamically loaded or cached according to routing results.

    The advantage of this architecture is controllable cost; the downside is that performance depends heavily on memory bandwidth, expert hit rate, data movement efficiency, and scheduling strategy.

    1. Capacity Planning

    The core of the minimum runnable scenario is not compute but large-capacity memory.

    A rough guideline is as follows:

    1. required memory ≈ weight capacity + non-quantized parameters + runtime workspace + KV cache + system reserve

    If the weight capacity is close to 1.5TB, actual memory planning usually should not be below 2TB. If the context is long or you want to reduce swapping overhead, 2.5TB to 3TB can be considered.

    2. The Role of the GPU

    In this mode, the GPU does not necessarily need to hold all experts. A more common approach is:

    • Keep the shared attention portion on the GPU;
    • Cache the frequently accessed expert layers into GPU VRAM or high-bandwidth memory as much as possible;
    • Place low-frequency experts in CPU memory;
    • Allow extremely cold experts to be loaded from local storage, at the cost of lower speed.

    Therefore, the minimum runnable approach does not have high requirements on the number of GPUs, but places heavy demands on host memory capacity, memory channel bandwidth, and the CPU memory controller.

    3. Performance Characteristics

    Deployments of this kind usually show clear bottlenecks:

    • Before generating the first token, large amounts of weights need to be loaded or caches established;
    • Each token generation may require moving data across CPU memory and GPU;
    • Expert routing can cause random access;
    • If expert caching is insufficient, frequent swapping in and out slows things down;
    • Long context further amplifies KV cache pressure.

    Ultimately this may manifest as single-digit tokens per second, or obvious stutter under complex contexts.

    4. Cost Characteristics

    The cost of the minimum runnable approach is concentrated mainly in large-capacity servers, memory modules, motherboards, CPUs, and multi-channel memory bandwidth. Its advantage is a low initial barrier, suitable for validating whether the model can load, whether it can complete a full inference, and whether output quality can be assessed.

    But for real production use, it is usually not a production-grade solution, because users cannot accept excessively low speed, and it cannot support multi-concurrency.

    4. Single User at 25 tokens/s: From “Runs” to “Usable,” With the Bottleneck Shifting to Bandwidth and Hot-Spot Caching

    The second target is single-user use with an output speed of 25 tokens/s. This target is already close to the experience threshold of general interactive applications, and is suitable for local-assisted coding, document analysis, private knowledge Q&A, and similar scenarios.

    Compared with minimum runnable, the core change at this level is not “whether it can load,” but “whether it can continuously and stably emit tokens.”

    1. Why 25 tokens/s Is an Important Threshold

    In a single-user interactive scenario, perceived experience depends not only on first-token latency but also on the generation speed of subsequent tokens. If generation speed is too low, the experience will be poor even if the final answer is correct.

    25 tokens/s means that, ideally, about 25 tokens can be stably generated per second. For scenarios like long-text generation, code completion, and reasoning-chain output, this speed noticeably improves usability.

    But MoE models expose a characteristic during the decode phase: each generated token may require accessing large amounts of expert weights. Even if only a few experts are activated, the system still needs to complete routing, addressing, data movement, computation, and result merging. If the data cannot stay as close as possible to the compute units, speed will be limited.

    2. Key Optimization Directions

    To reach single-user 25 tokens/s, optimization is usually needed around the following aspects:

    1) Increase the residency ratio of hot-spot experts

    Not all experts are accessed with equal probability. Some experts may be hit more frequently by specific tasks or specific token patterns. In engineering, you can analyze the routing distribution and place high-frequency experts in high-bandwidth storage as much as possible.

    1. high-frequency experts → accelerator VRAM or high-bandwidth memory
    2. medium-frequency experts → CPU memory
    3. low-frequency experts → large-capacity local storage

    This tiered caching strategy can significantly reduce cold starts and cross-layer movement.

    2) Reduce memory bandwidth pressure

    The decode phase is a typical bandwidth-sensitive task. Each token needs to read a large amount of parameters and context state, so memory bandwidth and VRAM bandwidth become key metrics.

    If model weights are scattered across ordinary memory and CPU memory bandwidth is insufficient, the target speed will be hard to reach even with abundant GPU compute.

    3) Reduce cross-device communication

    If experts are distributed across multiple devices or nodes, activations must be sent to the target expert’s location after routing, and results retrieved after computation. If this communication happens frequently, it will significantly lengthen single-token latency.

    Therefore, the single-user 25 tokens/s scenario is better suited to converging the critical path within fewer devices or a higher communication domain.

    4) Reasonable batching

    Although batching yields less benefit in a single-user scenario than in multi-concurrency, overall efficiency can still be improved through dynamic scheduling. For example, compute-intensive parts can be executed together to reduce launch overhead; or short requests can be micro-batched to improve device utilization.

    3. Cost Characteristics

    The cost of single-user 25 tokens/s is usually higher than the minimum runnable approach. It requires a higher-quality hardware combination:

    • Higher-capacity accelerator VRAM;
    • Higher-bandwidth CPU memory;
    • A stronger GPU-CPU data path;
    • A better local storage and caching system;
    • More detailed quantization, routing, and scheduling optimization.

    If the budget is limited, you can lower the barrier by reducing context length, increasing the quantization compression ratio, and optimizing the expert caching strategy. But too low a quantization compression ratio may affect precision, so a balance must be struck between output quality and cost.

    5. 10 Concurrency With 25 tokens/s per User: Production-Grade Deployment, Where the Bottleneck Becomes Communication, Scheduling, and KV Cache

    The third target is that at 10 concurrency, each user’s output speed still reaches 25 tokens/s or above. This level moves from “personally usable” into the category of “multi-user production system.”

    After concurrency increases, the system must not only handle more token-generation requests but also maintain multiple context states simultaneously. Each request has its own KV cache, routing state, temporary activations, and output buffer. At this point, simply adding VRAM or memory capacity is not enough; communication, scheduling, and cache management must be optimized together.

    1. New Pressures Introduced by Concurrency

    1) KV cache grows multiplicatively

    Each concurrent request produces its own context cache. The longer the context, the larger the KV cache. 10 concurrency is not simply multiplying single-request VRAM by 10, because different requests may differ in context length, generation length, and attention distribution.

    A simplified estimate is as follows:

    1. KV cache ≈ layers × attention heads × head dimension × sequence length × concurrency × bytes per element

    In real systems, paging management, fragmentation, reserved buffers, and dynamic growth must also be considered.

    2) Expert routing communication becomes more frequent

    10 concurrency means more tokens passing through the router at the same time. Each token may activate different experts, making expert access patterns more scattered. If experts are distributed across different devices or nodes, cross-device communication will increase noticeably.

    Large-scale MoE inference is very sensitive to communication bandwidth. Experts do not simply finish local computation; activations must be sent to the location of the target expert, and results returned after computation. Communication latency directly stacks onto each token’s generation path.

    3) Scheduling complexity rises

    With a single user, the system can prioritize minimizing latency; with multiple users, the system must balance fairness, throughput, and latency. If scheduling is poor, the following may occur:

    • One long-context request monopolizes too much cache;
    • Some short requests are blocked by long requests;
    • Expert caches are frequently flushed;
    • Communication queues build up;
    • VRAM fragmentation reduces available capacity.

    Therefore, production-grade deployment usually needs a more complete request queue, priority policies, context paging, and cache reclamation mechanisms.

    2. Architecture Recommendations

    For the 10-concurrency target, a common approach is a large-scale multi-card, multi-node deployment architecture.

    1) Keep weights resident in high-speed storage as much as possible

    If all weights must reside in high-speed VRAM, then total effective VRAM needs to cover:

    1. weight capacity + KV cache for each concurrent request + temporary workspace + communication buffers + safety margin

    Based on nearly 1.5TB of weights, a production-grade deployment usually needs to plan for a total capacity far above the weights themselves. If the context is long and concurrency is high, actual capacity requirements may exceed 2TB.

    2) Expert parallelism and communication-domain optimization

    Expert parallelism is a common approach for MoE inference, distributing different experts across different devices. But expert parallelism brings routing communication pressure. To reduce latency, high-frequency communication should occur within high-bandwidth interconnect domains as much as possible.

    The key here is not simply stacking up device counts, but building a reasonable communication domain:

    • Place high-frequency experts in the same high-speed communication domain as much as possible;
    • Keep cross-node communication limited to low-frequency paths as much as possible;
    • Aggregate or group routing results to reduce meaningless data movement;
    • Batch expert computations to lower the per-token communication cost.

    3) Dynamic batching and paged caching

    The 10-concurrency scenario requires finer-grained resource management. Dynamic batching can improve accelerator utilization, but must also avoid batches so large that single-request latency rises. KV cache is best managed with paging or block-based management to reduce fragmentation and improve reuse.

    A common flow is:

    1. request arrives → routing analysis → expert cache hit check → dynamic batching → expert computation → result merging → output buffer

    Each step needs to monitor latency, VRAM, memory, and communication queues; otherwise local optimizations may be offset by the overall bottleneck.

    3. Cost Characteristics

    The third level is the most expensive. It usually requires:

    • Multiple servers or multi-node accelerator clusters;
    • High-speed interconnect networks;
    • Large-capacity, high-bandwidth memory;
    • A reliable storage system;
    • A complete monitoring, logging, and alerting system;
    • More complex deployment, tuning, and operations workflows.

    For a self-built approach, the upfront hardware investment is significantly higher than the first two levels; for cloud resources, you can validate workloads via on-demand or reserved instances, but for long-term high-frequency use, you still need to pay attention to resource utilization efficiency and instance-spec matching.

    6. Cost Is Not Only About Accelerator Cards: What Really Determines the Budget Is Capacity, Bandwidth, and Communication

    Many people, when assessing local deployment cost, only focus on the number of accelerators, which is incomplete. For a trillion-parameter MoE model, cost is made up of at least the following categories:

    1. Storage and Memory Cost

    Weight capacity is a hard constraint. Even after 4-bit quantization, roughly 1.5TB-level weight space is still needed, and actual capacity requirements are higher with runtime overhead.

    If there is not enough memory or VRAM, the system can only constantly swap data in and out from low-speed storage, and speed will drop sharply.

    2. Bandwidth Cost

    The decode phase is a bandwidth-intensive task. Memory bandwidth, VRAM bandwidth, and the data channel bandwidth between GPU and CPU all directly affect token generation speed.

    The difference between single-user 25 tokens/s and 10-concurrency 25 tokens/s comes largely from bandwidth pressure. After concurrency increases, the system needs to move data for multiple requests simultaneously, and insufficient bandwidth will quickly become the bottleneck.

    3. Communication Cost

    An MoE model’s expert routing brings extra data movement. The more widely experts are distributed, the more frequent the communication. In multi-machine deployments, the quality of the inter-node network directly affects overall throughput.

    Therefore, a production-grade MoE deployment cannot look only at single-card performance, but must look at the design of the entire communication domain.

    4. Power and Cooling Cost

    High VRAM, multiple accelerators, and multi-node deployments bring significant power consumption. Racks, power supplies, cooling, and data-center or home-environment cooling conditions all affect long-term operating cost.

    5. Operations Cost

    Local deployment is not a one-time task. Model upgrades, driver updates, quantization-file replacement, failure recovery, monitoring and alerting, and log analysis all require continuous investment. For a 10-concurrency production environment, operations cost cannot be ignored.

    7. A Budget Estimation Framework: How to Break “How Much Will It Cost” Into Computable Problems

    To avoid quoting by gut feeling, you can adopt the following estimation framework.

    1. Weight Capacity Estimation

    1. def estimate_weight_gb(total_params, bit_width, scale_bits=0, group_size=1):
    2. effective_bits = bit_width
    3. if scale_bits > 0:
    4. effective_bits += scale_bits / group_size
    5. return total_params * effective_bits / 8 / 1e9
    6. weight_gb = estimate_weight_gb(
    7. total_params=2.8e12,
    8. bit_width=4,
    9. scale_bits=8,
    10. group_size=32
    11. )
    12. print(weight_gb)

    This result represents only the weight body and cannot be used directly as whole-machine capacity.

    2. Total Capacity Estimation

    1. required_capacity =
    2. weight capacity
    3. + non-quantized parameter capacity
    4. + metadata and alignment overhead
    5. + runtime workspace
    6. + KV cache
    7. + safety margin

    For the 10-concurrency scenario, KV cache and safety margin must be estimated separately and cannot simply reuse the single-user configuration.

    3. Throughput Estimation

    1. system throughput ≈ concurrency / average per-request token generation latency

    If the target is 10 concurrency with 25 tokens/s per user, the system needs to support an aggregate generation capability of about 250 tokens/s while keeping each request’s tail latency acceptable.

    4. Cost Structure Estimation

    1. one-time cost =
    2. accelerator devices
    3. + servers and motherboards
    4. + memory
    5. + local storage
    6. + network equipment
    7. + power and racks
    8. + software and deployment services
    9. recurring cost =
    10. electricity
    11. + operations labor
    12. + failure replacement
    13. + network and data center
    14. + monitoring and logging

    For cloud resources, recurring cost can be estimated by dimensions such as instance hours, storage capacity, and network traffic. Whether self-built or rented, small-scale benchmarking should be done first before deciding the scale of long-term investment.

    Path 1: Minimum Runnable

    Applicable scenarios:

    • Validating model output quality;
    • Algorithm experiments;
    • Offline batch processing;
    • Research tasks not sensitive to speed.

    Recommended configuration approach:

    • A large-capacity memory server;
    • A moderate number of accelerators for the shared attention module;
    • Tiered storage for expert layers;
    • Prioritizing weight loadability and result correctness.

    Cautions:

    • Do not expect high concurrency;
    • Do not use overly long contexts;
    • Be prepared to accept low generation speed;
    • Suitable for validating first, then deciding whether to scale up.

    Path 2: Single User at 25 tokens/s

    Applicable scenarios:

    • Local-assisted development;
    • Single-user knowledge Q&A;
    • Private document analysis;
    • A personal research assistant.

    Recommended configuration approach:

    • Increase high-bandwidth memory and VRAM capacity;
    • Cache hot-spot experts;
    • Optimize routing and scheduling;
    • Reduce cross-node communication;
    • Balance quantization format and precision.

    Cautions:

    • Single-user experience depends heavily on decode speed;
    • Memory bandwidth and expert hit rate need to be monitored;
    • Long context will noticeably increase pressure;
    • If pursuing a stable experience, relying on the minimum configuration is not recommended.

    Path 3: 10 Concurrency With 25 tokens/s per User

    Applicable scenarios:

    • Internal services for small teams;
    • Multi-user knowledge bases;
    • Enterprise private inference services;
    • Test platforms needing stable concurrency.

    Recommended configuration approach:

    • Multi-card, multi-node deployment;
    • A high-speed interconnect communication domain;
    • Combining expert parallelism with caching;
    • Dynamic batching;
    • Paged KV cache management;
    • A complete monitoring system.

    Cautions:

    • Communication and scheduling matter more than single-card performance;
    • Enough capacity must be reserved to handle burst concurrency;
    • Quantization precision, context length, and concurrency need to be optimized jointly;
    • Better suited to a production-grade operations system.

    9. Quantization Precision: Don’t Look Only at Capacity, Also Look at Output Usability

    4-bit quantization is currently a common direction for compressing large-model weights, but lower is not always better. For trillion-parameter MoE models, quantization affects three types of objects:

    1. Expert weights: the largest share of capacity, usually quantized first;
    2. Attention and gating-related parameters: sensitive to routing and context understanding, and may require higher precision;
    3. KV cache: occupies noticeable space under long context and high concurrency, and its precision and capacity also need to be considered separately.

    If quantization is too aggressive, the following may occur:

    • Unstable routing;
    • Repeated output or semantic drift;
    • Degraded long-context capability;
    • Reduced code-generation accuracy;
    • Worse numerical stability.

    Therefore, in engineering, higher precision is usually retained for key layers, group-wise quantization is applied to the capacity-heavy expert layers, and output quality is verified with evaluation sets.

    In budget estimation, capacity cannot be calculated based only on the lowest quantization level; the extra space from keeping sensitive layers at high precision must also be considered.

    10. Validation Steps: Don’t Procure First, Run Staged Stress Tests First

    Before formally investing in large-scale hardware, it is recommended to validate in the following order:

    Step 1: Weight Loading Test

    Verify that the system can fully load the weights, and observe:

    • Loading time;
    • Peak memory;
    • Peak VRAM;
    • Whether there are alignment or metadata errors;
    • Whether the first token can be generated normally.

    Step 2: Single-Request Short-Text Test

    Use short input and short output to measure:

    • First-token latency;
    • Average token generation speed;
    • Whether long-sentence generation is stable;
    • Whether there are abnormal stalls.

    Step 3: Single-Request Long-Context Test

    Gradually increase context length and observe:

    • KV cache growth;
    • Memory fragmentation;
    • Whether long-text understanding degrades;
    • Whether generation speed decays noticeably.

    Step 4: Concurrency Stress Test

    Start from 2 concurrency and gradually increase to 10 concurrency, recording:

    • Aggregate throughput;
    • Per-user speed;
    • Tail latency;
    • Communication queue length;
    • VRAM and memory usage;
    • Expert cache hit rate.

    Step 5: Stability Test

    Run continuously for hours or days and observe:

    • Memory leaks;
    • Cache invalidation;
    • Routing drift;
    • Device temperature;
    • Network packet loss or jitter;
    • Failure recovery capability.

    Only after these tests can an “estimated cost” be turned into an “executable budget.”

    11. Conclusion: The Money for Local Trillion-Parameter MoE Deployment Mainly Goes to Capacity, Bandwidth, and Communication

    Overall, the cost of local trillion-parameter MoE model deployment can be summarized in three layers:

    1. Minimum runnable: the core is memory capacity and heterogeneous offloading. The budget is relatively the lowest, but speed may be only single digits, suitable for validation;
    2. Single user at 25 tokens/s: the core is bandwidth and hot-spot caching. Higher memory bandwidth, accelerator VRAM, or cache hit rate is needed, suitable for single-user interaction;
    3. 10 concurrency with 25 tokens/s per user: the core is communication, scheduling, and KV cache management. A multi-card, multi-node architecture, high-speed interconnect, and production-grade operations are needed; the cost is the highest.

    If you only ask “how much will it cost,” the answer is not a single number, but depends on the target level. Minimum runnable mainly invests in large-memory servers; single-user 25 tokens/s needs higher-bandwidth nodes; stable output at 10 concurrency needs a multi-machine, multi-card communication cluster.

    A more accurate expression is:

    1. cost ≈ weight capacity + runtime capacity + bandwidth capability + communication capability + operations capability

    For real users, the recommendation is not to pursue the highest configuration from the start, but to scale up gradually along the path of “first validate loading, then validate single-request speed, then validate concurrency throughput.” This controls trial-and-error cost and finds the most suitable hardware combination under real workloads.