How Much Does It Cost to Deploy a Trillion-Parameter Open-Source LLM On-Premises?

    This article breaks down the real cost of privately deploying a trillion-parameter open-source MoE large language model on your own hardware. It explains how three hard constraints—memory capacity, memory bandwidth, and cross-machine communication—drive the hardware requirements, then walks through parallelism strategies, an illustrative inference configuration, and a hardware bill of materials with starter (~RMB 8 million) and recommended (~RMB 30 million) budget tiers plus a 3-year TCO estimate. It also compares self-built, cloud, and hybrid routes, offers practical cost-optimization techniques, and closes with a 10-point selection checklist to help you justify the budget and pick the right deployment path.

    1. The Bottom Line: The Hardware Bill for On-Premises Deployment

    TL;DR: Based on current mainstream high-end accelerator cards, private deployment of a trillion-parameter-class open-source MoE model starts at roughly RMB 8 million in hardware and requires around RMB 30 million for a production-ready configuration. Ongoing costs—electricity, data center upgrades, and operations staff—are calculated separately. The budget figures and hardware list here can be used directly to explain the cost breakdown to management.

    This cost is not determined by how “good” the model is, but by three hard constraints:

    • Memory capacity: determines the minimum number of accelerator cards needed;
    • Memory bandwidth: determines how fast a single request can run;
    • Cross-machine communication: determines whether multiple machines can work together efficiently as one unit.

    The rest of this article breaks down each of these points and closes with selection recommendations and a checklist for different use cases.

    2. Deployment from an Architectural Perspective: What Makes Trillion-Parameter MoE Different

    As a compute target, the open-source model referenced in this article (hereafter “the model”) has the following typical characteristics:

    • Scale: A sparse Mixture-of-Experts (MoE) architecture with roughly 2.8 trillion total parameters. Only a small fraction of experts are activated per token, so compute consumption is far lower than that of a dense model of equivalent size.
    • Context: Natively supports an approximately 1-million-token context window.
    • Weight format: Uses native 4-bit floating-point quantization; the weight files total about 1.56 TB across 96 shards.
    • Attention structure: A hybrid design combining linear attention and sparse attention, with support for multi-token prediction.
    • Capability focus: Compared with the previous generation, improvements come mainly from large-scale post-training and cross-domain reinforcement learning. In public benchmarks, its Terminal-category scores lead among open-source models, with some areas approaching top closed-source models; coding benchmark scores improved by about 5 percentage points, and an accelerated version with several-fold throughput gains is also available.

    2.1 Hybrid Attention: Lightening the Load for Long Contexts

    Traditional attention must cache the keys and values (KV) of all historical tokens, and memory usage grows quadratically with context length, making a 1-million context nearly impossible to sustain on a single machine. The model’s hybrid attention architecture splits the work between two types:

    • Linear attention: uses state compression to carry historical information in a fixed-size state;
    • Sparse attention: attends to only a small number of key historical segments.

    Combined, a million-scale context becomes feasible in both memory and compute. The tradeoff is that the implementation details of the two structures differ greatly, so the inference framework must provide optimized fused kernels—otherwise frequent reads/writes and switching can actually slow down overall speed.

    2.2 Multi-Token Prediction: Getting More Output from a Single Computation

    Autoregressive generation must read the activated weights from memory once for every token output—a classic “memory-bound” workload. Multi-token prediction lets the model produce multiple candidate tokens in a single forward pass, then uses a speculative-decoding-style verification mechanism to accept the correct ones. It improves throughput, but adds extra prediction-head parameters and verification overhead, placing higher demands on the inference framework’s scheduling and batching logic.

    2.3 Native 4-bit Weights: Half the Memory, No Lower Software Bar

    Storing weights directly in 4-bit saves about three-quarters of the memory compared with 16-bit precision, and this is the key prerequisite for running a trillion-scale model on a limited number of accelerator cards. But it also means:

    • Matrix computation must run in low-precision format, with block scaling factors to restore numerical precision;
    • Accelerator cards and inference frameworks must natively support this format. If they can only dequantize first and then compute, both compute and bandwidth get eaten by the extra overhead.

    2.4 Post-Training and Reinforcement Learning: Deciding Whether Deployment Is Worth It

    Large-scale post-training and cross-domain reinforcement learning mainly improve instruction-following, multi-step reasoning, and tool-calling on complex tasks—precisely the capabilities most valued in coding assistants, ops agents, and similar scenarios. For evaluators, this means selection cannot rely on parameter scale alone; you must benchmark with your own business data. It also means confronting another reality: the more capable the model, the more it tends to push up token consumption per request, and inference costs rise accordingly.

    3. Doing the Math on the Three Constraints: How Many Machines Do You Actually Need?

    3.1 Memory Capacity: At 141GB per Card, Weights Alone Eat Up 80%

    Memory is the first threshold for any inference workload; a single “out of memory” error can prevent the service from starting at all. A simplified model can be written as:

    1. Memory requirement ≈ weights + KV cache + runtime and communication buffers + reserve margin

    Taking a high-end accelerator card with about 141GB of HBM per card (about 3.35TB/s of HBM bandwidth per card) as an example:

    • Weights only: 1.56TB ÷ 141GB ≈ 11.3, so rounding up requires at least 12 cards. With 8-card nodes, the minimum configuration is 2 nodes, 16 cards total.
    • Adding KV cache and buffers: 16 cards total about 2256GB; after subtracting the weights, about 660GB remains, to be split among the KV cache, framework overhead, and a roughly 15% safety margin.
    • Production recommendation: If there are many concurrent users and long average contexts, 2 nodes will quickly approach the memory ceiling, so 4 nodes (32 cards) is a safer starting point.

    Note that if you switch to domestic or previous-generation accelerator cards, per-card memory is typically only one-third to one-half that of this high-end card. Meeting weight loading alone requires 4–8 full nodes; the budget looks lower, but you pay a different price in communication, bandwidth, and ecosystem compatibility (see Section 8).

    3.2 Memory Bandwidth: Determining the Single-Request “Ceiling”

    During the autoregressive decoding phase, generating each token requires reading the activated expert weights of that layer in full. The theoretical generation speed of a single request can be approximated as:

    1. Single-request decode speed (tokens/s) ≈ total HBM bandwidth of accelerator cards within the node ÷ bytes of weights read per step

    Since experts are distributed across multiple cards, “total bandwidth” here should be understood as the effective bandwidth determined jointly by communication and compute bottlenecks. A rough estimate: assuming about 100 billion parameters are activated per token, at 4-bit storage this requires reading about 50GB; 16 cards provide a theoretical aggregate bandwidth of about 53.6TB/s, giving a theoretical single-request upper limit of about 1000 tokens/s. Actual values are also affected by:

    • Kernel implementation and communication efficiency;
    • Additional KV reads from long contexts;
    • Batching and scheduling overhead.

    This also explains why “can run” and “runs smoothly” differ by an order of magnitude: 2 machines can handle short-context, low-concurrency tasks, but struggle to support hundreds or thousands of employees online for everyday work.

    3.3 Cross-Machine Communication: The Most Underestimated Cost for Trillion-Parameter MoE

    When an MoE model uses expert parallelism, each token must be routed to the accelerator card holding the corresponding expert for computation and then sent back to the original card—a process involving two all-to-all communication steps. For deployments of 16 cards or more, cross-machine communication is unavoidable; it directly amplifies latency jitter and drags down overall throughput.

    The industry commonly uses three strategies:

    1. High-speed intra-node interconnect: 8 cards in a full node are fully connected via high-bandwidth interconnect, keeping the bulk of expert routing inside the machine;
    2. High-speed lossless networking: cross-machine links use 400G/800G-class RDMA networks, combined with non-blocking switching and congestion control to reduce tail latency;
    3. Communication topology optimization: optimizing the parallel mapping and scheduling of expert groups to reduce unnecessary cross-machine transfers.

    For this reason, a “super-node” form factor has emerged in the market, integrating dozens of accelerator cards through high-speed interconnect into a single unit, bringing cross-card communication closer to intra-node levels. This is also the underlying support that lets many cloud providers complete adaptation and full-pipeline validation on the very day a new model is released.

    3.4 Don’t Forget KV Cache: A Million Context Is Not a “Free Lunch”

    A model “supporting” a million context does not mean you can use a million context for “free.” A commonly used estimation formula is:

    1. KV cache memory ≈ 2 × num_layers × num_kv_heads × head_dim × sequence_length × bytes_per_element

    Because the model uses hybrid attention, actual usage is significantly lower than a full-attention estimate. But assuming FP8 storage (1 byte per element), 60 layers, 16 KV heads, and a head dimension of 128, a single 100,000-token request alone needs about 25GB of cache—nearly one-fifth of a single card’s memory. With dozens of such concurrent requests, the KV cache alone could consume half a machine’s memory.

    Therefore, setting a reasonable input-length limit on the business side and summarizing or truncating over-long conversations is the first gate for controlling memory and cost.

    4. Inference Architecture: How to Configure Parameters for Good Performance

    4.1 Five Parallelism Strategies, Each with Its Own Role

    Deploying a trillion-parameter MoE model is essentially “splitting one model across multiple cards and making them cooperate like a single card.” Common parallelism methods are:

    Parallelism What it splits Communication characteristics Role in deployment
    Tensor Parallelism (TP) Weight matrices within a layer Multiple collective communications per layer; frequent Not suitable across machines; usually done within an 8-card node
    Expert Parallelism (EP) Experts in MoE layers Two all-to-alls per step Main means of scaling card count; network-sensitive
    Pipeline Parallelism (PP) Split by layer Point-to-point transfer between adjacent stages Cross-machine scaling; low communication volume; pipeline bubbles
    Data Parallelism (DP) Full model replicas Gradient sync in training; front-end distribution in inference Horizontal scaling of concurrency; linear throughput gains
    Context Parallelism (CP) A single request’s long sequence Attention-related communication Shares KV and compute in ultra-long-context scenarios

    4.2 An Illustrative Deployment Configuration

    Most mainstream inference frameworks provide compatible interfaces, and the core parameters can be abstracted into four categories: parallelism dimensions + quantization + memory utilization + scheduling limits. The configuration below is illustrative only, to help you understand the relationships between parameters:

    1. # Illustrative inference service launch parameters (field names are generic, not the syntax of any specific framework)
    2. model: /models/oss-trillion-moe
    3. parallel:
    4. tensor: 8 # Tensor parallelism: equal to the number of accelerator cards within a node, avoiding frequent cross-machine communication
    5. expert: 4 # Expert parallelism: distribute MoE-layer experts across 4 nodes, i.e., 4 machines of 8 cards each
    6. pipeline: 1 # No pipeline parallelism needed when a single node has enough capacity; can be set to 2~4 when scaling larger
    7. quantization: w4_mx # Enable the 4-bit micro-scaling floating-point format; must match native hardware and framework support
    8. gpu_memory_utilization: 0.90 # Upper limit of per-card memory usage; reserve 10% for activations, buffers, and fragmentation
    9. max_model_len: 262144 # Maximum context length; adjustable between 128K~1M per business needs
    10. scheduler:
    11. enable_chunked_prefill: true # Chunked prefill for long requests, avoiding a single long request blocking others
    12. max_num_batched_tokens: 32768 # Maximum tokens per scheduling batch
    13. enable_prefix_caching: true # Reuse the KV cache of common prefixes such as system prompts
    14. kv_cache_dtype: fp8 # Store the KV cache in 8-bit, halving memory usage

    This configuration corresponds to the following logic:

    • 16 cards (TP=8 × EP=2): meets the minimum threshold;
    • 32 cards (TP=8 × EP=4): the recommended starting point for production;
    • Service replicas (DP): scale horizontally based on concurrency pressure, with a front-end gateway distributing requests by session affinity so that a user’s consecutive conversations hit the same cache as much as possible.

    4.3 Front End: Turning a “Model Service” into a “Usable Service”

    Being able to run inference is only half the work. A usable private inference platform typically also includes:

    • Unified gateway: handles authentication, tenant isolation, rate limiting, and audit logging;
    • Task queues and scheduling: support priority separation between online requests and offline batch jobs, avoiding long tasks dragging down the interactive experience;
    • Object storage: holds model shards and business corpora, working with a high-performance parallel file system for fast reads by compute nodes;
    • Monitoring and alerting: collect metrics such as GPU utilization, memory watermark, network retransmissions, throughput, and latency distribution, with threshold-based alerts.

    5. Budget Breakdown: From Accelerator Cards to 3-Year Total Cost of Ownership

    5.1 The Seven Parts of the Hardware Bill of Materials (BOM)

    • Compute nodes: accelerator cards and full nodes (CPU, memory, system disk, intra-node high-speed interconnect);
    • Network: intra-node high-speed interconnect comes with the full node; cross-machine relies on RDMA NICs, switch stacks, and optical modules;
    • Storage: an all-flash parallel file system plus object storage, for storing the ~1.5TB of weights and the multiple versions produced by model iterations;
    • Racks and data center: power, cooling, and cabling upgrades;
    • Spare parts and redundancy: a spare pool for wear-prone parts such as accelerator cards and power supplies;
    • Software and operations: commercial support for the inference framework, platform software, integration and tuning services;
    • Personnel: at least 2–3 engineers familiar with large-model inference, networking, and systems operations.

    5.2 Two Configuration Tiers and Budget Ranges

    Item Starter configuration (~RMB 8 million) Recommended configuration (~RMB 30 million)
    Compute nodes 2 eight-card full nodes (16 141GB-class accelerator cards) 4–8 eight-card full nodes (32–64 accelerator cards)
    High-speed network One 64-port-class 400G/800G switch + NICs and optical modules Two core switches stacked, with redundant key links
    Storage All-flash parallel file system (~100TB usable) All-flash parallel file system + object storage, supporting multiple model versions
    Data center Use existing high-power racks Rack power, cooling, and cabling upgrades
    Software and operations Open-source inference framework + third-party technical support Platform software, commercial support, and dedicated tuning
    Use case Small-scale internal validation, offline analysis, coding-assistant pilots Company-wide multi-business access, high-concurrency online services

    5.3 Why Prices Fluctuate

    Hardware prices are affected by multiple factors, including supply and demand, exchange rates, product-line iteration, and procurement scale. Take high-end accelerator cards as an example: mass production of new-spec products and the concentrated release of industry compute demand can significantly change the unit price of the previous-generation flagship card. Therefore:

    1. Use a range rather than a single-point quote for budget estimation;
    2. Reserve a 10%–15% price-fluctuation margin;
    3. Require suppliers to re-verify quotes quarterly during the tendering process.

    5.4 3-Year Total Cost of Ownership (TCO)

    The total cost of a self-built private inference platform can be estimated by the following structure:

    1. 3-year TCO ≈ one-time hardware investment + 3 years of electricity + 3 years of personnel + data center and bandwidth + software and integration + upgrade reserve

    Taking 4 eight-card machines as an example, a full node’s power consumption is typically in the 10–15kW range. Assuming an average annual load of 60% and an electricity price of RMB 0.7/kWh, 3 years of electricity costs about RMB 1.3–2 million; the labor cost for 3 engineers over 3 years is typically also in the range of several million RMB. Together, ongoing operating costs often reach 20%–30% of the hardware investment, which should not be omitted at the project-initiation stage.

    5.5 Concurrency Goals Determine the Budget Tier

    Beyond the hardware list, what matters more is “how many people this configuration can support.” You can back-calculate the required number of machines using:

    1. Required full nodes = (target concurrent requests × average output length per request ÷ throughput a single replica can sustain) × redundancy factor

    For example, with measured data: assume a single service replica made of 4 eight-card machines can provide about 2000 tokens/s of effective output throughput at an average 8K context. If the goal is 2000 employees online simultaneously at 20 tokens/s each, the demand is 40000 tokens/s, requiring 20 replicas (i.e., 160 cards), plus about 20% for peak and failure redundancy—actually needing around 24 nodes. If the business targets the whole company with concurrency at this scale, the budget jumps from “millions” into the “tens of millions” range or higher. Therefore, the concurrency goal is the first variable that determines the budget tier.

    6. Self-Built, Cloud, or Hybrid: How to Choose Among Three Routes

    6.1 Self-Built Private Deployment: For Data-Sensitive, Stable Long-Term Core Business

    The advantages are that data stays within your domain, resources are dedicated, and long-term unit costs are controllable; the cost is high upfront investment and long delivery cycles. When models and hardware iterate quickly, self-purchased equipment may become relatively outdated within two to three years.

    6.2 Cloud Rental: For Validation Periods, Variable and Peak-Burst Workloads

    Cloud providers typically offer compute forms matching trillion-parameter MoE models—via super-node instances, high-performance storage, and high-speed networking—and tend to complete adaptation and tuning relatively quickly after a new model is open-sourced. Pay-as-you-go or hourly billing can significantly lower the startup threshold, and elastic scaling can handle peaks like month-end settlement and concentrated training; but long-term, high-frequency use requires careful calculation of unit prices, and data leaving your domain must meet compliance requirements.

    6.3 Hybrid Deployment: The Mainstream Choice Balancing Cost and Elasticity

    A common approach is “private deployment for baseline traffic + public cloud for peaks and experimental workloads”:

    • Everyday requests are handled by the private cluster at the lowest unit cost;
    • Emergencies and month-end peaks automatically spill over to the cloud;
    • New models and versions are first validated in the cloud before deciding whether to update the private cluster’s weights.

    7. Cutting Costs and Boosting Efficiency: Getting the Most from Every Card

    7.1 Choose Quantization Precision Wisely

    4-bit weights already approach full-precision results on most general tasks, but on numerically sensitive tasks such as mathematical reasoning and code generation, quality regressions may occur. It is recommended to compare 4-bit vs. 8-bit results on your own evaluation set to find the balance point between quality and throughput.

    7.2 Separate Prefill and Decode Deployment

    Large-model inference has two phases:

    • Prefill: processes all input tokens at once; compute-intensive, quickly saturating single-card compute;
    • Decode: generates tokens one by one; memory-intensive, mainly consuming memory bandwidth.

    When the two phases are mixed in the same batch of requests, the prefill of long inputs blocks requests that are being generated, causing a “slow first token, uneven output” experience. Splitting them into separate resource pools, each with different parallelism strategies and batching parameters, can significantly improve overall utilization and latency stability.

    7.3 Continuous Batching and Chunked Prefill

    Traditional static batching must wait for all requests in the batch to finish before releasing resources, so short requests get “held back” by long ones. Continuous batching dynamically adjusts batch membership at each decode step; chunked prefill breaks long inputs into small chunks and alternates scheduling with decode requests. Combined, these can keep GPU utilization at a high level over the long term.

    7.4 Fine-Grained KV Cache Management

    • Paged management: borrowing from the idea of OS virtual memory, split the KV cache into fixed-size blocks allocated on demand to reduce memory fragmentation;
    • Low-precision storage: store the KV cache in FP8, halving memory usage with usually controllable precision loss;
    • Prefix cache reuse: compute common prefixes such as system prompts and code-repository context only once, with subsequent requests hitting the cache directly;
    • Popularity-based offloading: migrate inactive session caches to host memory or even storage media, avoiding recalculation after outright discarding.

    7.5 Tuning Speculative Decoding and Multi-Token Prediction

    By sampling an appropriate number of draft tokens and adjusting parameters such as prediction-head length, actual throughput can be improved. The key is to continuously monitor the “candidate token acceptance rate”: a drop in acceptance indicates that model behavior or business distribution has changed and parameters need re-tuning.

    7.6 Expert Parallelism Topology and Communication Optimization

    Placing frequently activated experts on the lowest-communication-cost paths reduces cross-machine transfers; combined with compute-communication overlap and communication-primitive fusion, this can suppress the latency jitter from all-to-all. Such optimization typically requires collaboration among cloud providers, chip vendors, and model teams, and is also an important dimension for evaluating ecosystem support during procurement and selection.

    7.7 Capacity Elasticity and Offline Batching

    Moving non-real-time tasks such as code completion and document summarization to nighttime troughs, combined with tidal scheduling and resource-pool reuse, can absorb an extra 20%–30% of the load without adding hardware (the specific ratio depends on business distribution).

    8. Selection Checklist: Answer These 10 Questions Before Ordering

    1. Is the accelerator-card ecosystem complete: Does it natively support the kernels needed for 4-bit micro-scaling floating-point, hybrid attention, and multi-token prediction? Missing any one could cost you several-fold throughput.
    2. Do per-card memory and bandwidth match: Enough cards but insufficient bandwidth causes obvious lag in long-context scenarios; enough bandwidth but insufficient memory means you can’t even load the weights.
    3. Is the network topology non-blocking: Do cross-machine bandwidth and switch oversubscription ratios meet all-to-all needs? Are the number of optical modules and cable lengths included in the budget?
    4. Can storage load weights quickly: The ~1.5TB of weights needs to be quickly restored after a failure reboot—is the combination of an all-flash parallel file system and local NVMe cache in place?
    5. Do data center power and cooling meet requirements: High-power-density racks often need power and cooling upgrades, and this hidden cost is easily overlooked.
    6. How mature is the software stack: The inference framework’s support speed for new models, kernel coverage, and compatibility with training frameworks directly determine delivery cycles.
    7. Can you get vendor-level support: Is there original-vendor technical support, version upgrades, and a fault-response mechanism?
    8. Is observability complete: Are GPU utilization, memory watermark, network retransmissions, and latency distribution quantifiable and alertable?
    9. Do security and compliance meet requirements: Do weights and data stay within your domain? Does the model service have authentication, audit, and content-safety capabilities?
    10. Is iteration cost reserved: Are model version updates, hardware expansion, and spare-part replacement included in the 3-year budget?

    9. Final Thoughts: Do the Math Before You Launch

    Back to the original question: how much does it actually cost to deploy a trillion-parameter open-source LLM on-premises?

    • Starter threshold: about RMB 8 million, 2 eight-card full nodes, suitable for internal validation and offline tasks;
    • Recommended configuration: about RMB 30 million, 4–8 eight-card full nodes plus high-speed networking and storage, supporting medium-scale multi-business access;
    • Long-term ledger: electricity, personnel, data center, and upgrade iterations typically require several million RMB more over three years;
    • Scaled service: if serving thousands of employees company-wide online simultaneously, the number of machines may expand to dozens, pushing the budget into the tens-of-millions range or higher.

    More importantly, this number is not as simple as “large models are expensive”—it is the inevitable result of three overlapping constraints: memory capacity, memory bandwidth, and cross-machine communication. Understanding these three points lets enterprises break the budget into a verifiable hardware list, and find the option best matching their own data requirements, concurrency scale, and pace of development among self-built, cloud, and hybrid deployment.

    If you’re about to apply for a large-model private-deployment budget for your team, it is recommended to organize your materials in five steps—conclusion, constraints, configuration, budget, route: first give a reportable budget range, then explain the cost sources using the three points of memory, bandwidth, and communication, and finally attach the configuration list and 3-year TCO. Such materials are often more likely to pass review than a vague quote sheet.