Trillion-Parameter LLM Deployment: Budget Estimation and Three Hardware Tiers

(Note: Output string begins below)

1. Open-Source Weights ≠ Free Lunch

“The model is open-source, so if we buy a few servers and deploy it ourselves, we won’t have to pay API fees, right?”

This is a common statement heard during project kickoff meetings. However, a detailed cost breakdown reveals a different reality: for trillion-parameter models, open source merely provides an “entry ticket.” At current hardware prices, this ticket starts at eight figures (in RMB equivalent, easily translating to millions of USD).

Using a public trillion-parameter MoE open-source model as a reference (weight size ~1.56TB; all subsequent figures are based on this baseline and may vary across models), this article breaks down the budget layer by layer to answer three questions:

• What is the absolute minimum cost to just “get it running”?
• How much investment is needed for a “comfortable” production environment?
• Beyond the initial hardware quote, what other hidden costs must be accounted for?

2. The First Cost: How “Heavy” is the Model?

The first step in deployment is not selecting servers, but calculating the weight size. The estimation formula is straightforward:

Weight Size ≈ Total Parameters × Bytes per Parameter

Taking a trillion-parameter MoE model as an example:

• BF16 (2 bytes per parameter): Size exceeds 2TB;
• FP8 (1 byte per parameter): ~1.5TB, aligning with our 1.56TB reference case;
• INT4 (0.5 bytes per parameter): Can be compressed to under 1TB, but the precision loss is typically unacceptable for initial production environments.

Here is a reusable estimation script:

def estimate_vram_tb(n_params_b, bytes_per_param, overhead=0.3):
    """Roughly estimate production-level VRAM requirements (TB)
    n_params_b: Total parameters in billions
    bytes_per_param: Precision, FP8=1, BF16=2
    overhead: Reserved ratio for KV Cache/activations/communication buffers
    """
    weight_gb = n_params_b * bytes_per_param
    total_gb = weight_gb * (1 + overhead)
    return total_gb / 1024
# Trillion-parameter (FP8) deployment, reserving 30% overhead
print(f"{estimate_vram_tb(1560, 1):.2f} TB")   # ≈ 1.99 TB

Two easily overlooked details: First, quantization itself introduces additional overhead from scale parameters and packing alignment. Second, after multi-node sharding, each GPU must have enough room for activations and communication buffers in addition to its allocated weights.

3. The Second Cost: The VRAM Wall

Knowing the weights take up 1.56TB, the next step is to compare this against hardware. Current mainstream high-end inference servers are designed with 8 GPUs per node. Using high-end accelerators with a common 140GB+ VRAM per card, a single node provides approximately 1.15TB of VRAM.

The conclusion is immediate: 1.15TB < 1.56TB. A single node cannot even hold the weights, let alone the KV Cache.

This brings us to an engineering rule of thumb:

Actual VRAM Requirement ≈ Weight Size × 1.2 ~ 1.5

Where does the extra 20%~50% go?

1. KV Cache: The higher the concurrency and longer the context, the larger the cache. In long-context scenarios, this can reach hundreds of GBs.
2. Activations and Intermediate Tensors: MoE routing and attention calculations generate temporary VRAM usage.
3. Framework and Communication Buffers: Cross-GPU communication for tensor parallelism and expert parallelism requires pre-allocated buffer space.

With a conservative 1.3x multiplier, the production-level VRAM requirement for this model is roughly 2TB. Compared to a single node’s 1.15TB, this means starting with at least two machines—this is the mathematical basis for the “minimum 8 million RMB” entry cost.

4. The Third Cost: Three Hardware Tiers

Based on public estimates, three typical configuration tiers can be outlined (prices are hardware reference costs; actual prices depend on procurement):

Tier	Scale	Reference Budget	Positioning
Entry	2 servers / 16 GPUs total	~8.2M - 8.5M RMB	Can run, but concurrency and context are limited
Comfort	4 servers / 32 GPUs total	~16M RMB	Has headroom for medium-concurrency production
Recommended	8 supernodes / 64 GPUs total	~31M - 32M RMB	Production-grade recommended setup

The differences between these tiers go beyond just “more GPUs”:

Entry Tier (~8M RMB): The weights fit exactly, leaving very little headroom. The trade-offs are limited concurrency, compromised long-context capabilities, and virtually no redundancy or gray release capacity. Suitable for POC validation and internal pilot testing, but not recommended for direct external services.

Comfort Tier (~16M RMB): VRAM headroom doubles, providing space for the KV Cache. This enables longer contexts and higher concurrency. However, single points of failure remain a concern, and future scaling requires topology replanning.

Recommended Tier (~32M RMB): Uses a supernode architecture to connect 8 machines via high-bandwidth interconnects into a single “logical machine.” This is the standard posture for trillion-parameter MoE model inference, for reasons explained in the next section.

5. Why Supernodes are Recommended: The Communication Math for MoE

Trillion-parameter models almost universally adopt the Mixture of Experts (MoE) architecture: the total parameter count is massive, but each token only activates a small fraction of experts. This introduces a set of contradictions:

• Compute is sparse: The compute volume per token is manageable and doesn’t require astronomical compute power.
• Parameters are dense: All experts must reside in VRAM constantly.
• Routing is dynamic: Every token must be sent to the GPU hosting the correct expert.

As a result, the bottleneck shifts from “compute” to “communication.” The core communication pattern for MoE inference is all-to-all: at each routing layer, token features are shuffled back and forth across expert GPUs in the cluster. If cross-node communication relies on standard Ethernet, bandwidth and latency will directly cripple throughput.

The supernode architecture utilizes high-bandwidth, low-latency inter-node interconnects to bring dozens of GPUs into a single communication domain, allowing Expert Parallelism (EP) to scale confidently across 64 GPUs or more. This explains why deployment recommendations accompanying model releases often specify supernodes—a loosely coupled multi-node cluster with the same number of GPUs might only achieve a fraction of a supernode’s throughput.

Here is a schematic pseudo-configuration describing this deployment topology:

# Multi-node inference deployment schematic (pseudo-config)
cluster:
  nodes: 8
  gpus_per_node: 8
  interconnect: supernode      # High-bandwidth inter-node interconnect
parallel:
  expert_parallel: 64          # Expert parallelism spread across the cluster
  tensor_parallel: 1
runtime:
  dtype: fp8
  kv_cache_budget: 0.25        # VRAM ratio reserved for KV Cache

An additional engineering perspective: major cloud providers now commonly offer “Day 0 adaptation” for leading open-source models—meaning inference frameworks, operators, and cluster configurations are ready simultaneously with the weight release. For teams lacking in-house tuning capabilities, this path can save weeks or even months of adaptation costs.

6. Beyond the Quote: Mandatory Hidden Costs

If you plan to request that 8 million RMB budget from management, make sure to include the following items; otherwise, you will likely run out of money after going live:

1. Datacenter and Power: Estimating hundreds of watts per GPU, a 64-GPU cluster will draw tens of kilowatts just for the accelerators. Combined with CPUs, networking, and cooling, daily power consumption easily exceeds thousands of kilowatt-hours. Older datacenters will likely need power, UPS, and cooling capacity upgrades.
2. Network Equipment: Supernode high-speed interconnects, management networks, and storage networks require three separate sets of switches and cables, which are costly.
3. Storage and Distribution: 1.56TB of weights, plus version management and hot standby, require high-performance distributed storage. Optimizing model distribution and loading times also requires investment.
4. Software and Personnel: Framework selection and tuning, operator adaptation, monitoring and alerting, logging, and capacity planning require at least a small, dedicated algorithm engineering team. Personnel costs are recurring annual expenses.
5. Depreciation and Evolution: Hardware lifecycles are typically depreciated over 3 to 5 years, while model iterations occur monthly. Whether your existing cluster can accommodate the weights of next year’s new model release must be considered during project initiation.

7. On-Premise vs. Cloud: A Decision Checklist

Suitable for On-Premise: Strict compliance requirements where data must not leave the network domain; stable and sufficiently large internal inference traffic to keep the machines fully utilized; a 3-5 year long-term plan with hardware O&M and framework tuning capabilities.

Suitable for Cloud Hosting: Traffic has obvious peaks and valleys, making on-premise utilization inefficient; small team size incapable of maintaining multi-node inference clusters; need to quickly upgrade with model versions without bearing hardware depreciation.

Cloud platforms, such as Baidu AI Cloud, offer general capabilities including GPU compute, container platforms, object storage, and monitoring/alerting. These transform the aforementioned hidden costs into pay-as-you-go operational expenditures. For most small and medium-sized enterprises, the lower-risk path is: “Run on cloud → Validate internally → Evaluate on-premise after traffic stabilizes.”

8. Conclusion

Open source does not mean free, but for capable teams with clear use cases, open source shifts the question from “is it possible?” to “is it expensive?” More notably, as trillion-parameter weights become open, supernode-level compute and Day 0 cloud adaptations are ready simultaneously. The combination of LLMs and compute power is turning past frustrations—like unobtainable GPUs and incompatible operators—into a genuinely deliverable engineering reality.

If you are evaluating such a project, the recommended action sequence is: Calculate the VRAM budget using the formulas in this article → Frame your budget range using the three hardware tiers → List hidden costs in your project proposal → Use cloud resources for a POC → Let the data decide whether that 8 million RMB entry ticket is worth buying.