On-Premises Deployment of Trillion-Parameter Open-Source LLMs: Cost Estimation and Hardware Selection in Practice

1. Background: When an Open-Source Model No Longer Fits on a Single Server

Over the past couple of years, the parameter counts of open-source large language models have kept climbing. Once total weight files reach the TB scale, “on-premises deployment” stops being a slogan and becomes a very concrete list of questions: which accelerators to buy, how many machines, where to host them, what the power bill looks like, and who will run the system.

Take a typical trillion-parameter open-source model as an example: its total weight files span roughly 1.5 TB. Even a flagship server loaded with eight high-memory accelerator cards offers just over 1 TB of total GPU memory—not enough to hold the weights, let alone the KV cache, activations, and framework overhead that inference also requires.

This is not a problem specific to any single model or hardware vendor; it is the first wall that every on-premises deployment of a trillion-parameter model runs into. The good news is that as open-source model ecosystems and the surrounding compute stacks mature, the old predicament of being unable to buy accelerators or get operators to run is fading, and private deployment is shifting from a technical aspiration into a deliverable engineering project. This article follows that same order: work out the numbers, choose the configuration, then put it into production.

2. Do the Math First: How Much GPU Memory You Actually Need

2.1 Weights: the baseline calculation

Weight memory can be estimated with a simple formula:

Weight memory ≈ number of parameters × bytes per parameter

Bytes per parameter at common precisions: 2 bytes for BF16/FP16, 1 byte for FP8, 0.5 bytes for INT4. For example, a 700-billion-parameter model stored in BF16 needs about 1.4 TB for weights alone—and once you add the runtime overhead of the inference framework, actual usage is even higher.

2.2 KV cache: the part everyone underestimates

Many teams budget only for weights and forget the KV cache. It can be estimated as:

KV cache ≈ 2 × layers × KV heads × head dimension × sequence length × concurrency × bytes per element

With a typical configuration—80 layers, 8 KV heads, a head dimension of 128, BF16 storage—a single 32K-token long-context request can require on the order of 10 GB of KV cache. At 32 concurrent requests, KV cache alone can consume more than 300 GB. In long-context scenarios, this overhead can even exceed the weights themselves.

def weight_gb(params_billion: float, bytes_per_param: float = 2.0) -> float:
    """Estimate GPU memory used by model weights (GB)."""
    return params_billion * bytes_per_param
def kv_cache_gb(layers: int, kv_heads: int, head_dim: int,
                seq_len: int, batch: int, bytes_per: float = 2.0) -> float:
    """Estimate GPU memory used by the KV cache (GB)."""
    per_token = 2 * layers * kv_heads * head_dim * bytes_per
    return per_token * seq_len * batch / 1024 ** 3
print(weight_gb(700))                     # 700B params, BF16 weights ≈ 1400 GB
print(kv_cache_gb(80, 8, 128, 32768, 32)) # 32K context × 32 concurrent requests ≈ 320 GB

2.3 Conclusion: a single server is out of the question

Add up the three ledgers: roughly 1.5 TB of weights, tens to hundreds of GB of KV cache, plus a 10–20% reserve for activations and framework overhead. Any single-server design can be ruled out immediately—a multi-node cluster is the only option. That is the starting point for every configuration discussed below.

3. Three Reference Configurations: the Entry Ticket, the Comfort Tier, and the Recommended Setup

The table below reflects a reference estimate circulating in the industry (quotes vary widely by period and sales channel; treat the numbers as order-of-magnitude references only):

Tier	Topology	Reference budget	Positioning
Entry	2 × 16-GPU machines	≈ RMB 8.0–8.5 million	Minimum viable; the “entry ticket”
Comfort	4 × 32-GPU machines	≈ RMB 16 million	Balances concurrency and redundancy
Recommended	8 × 64-GPU supernodes	≈ RMB 30–32 million	Vendor-recommended topology; best experience

3.1 Entry tier: get it running first

This is the smallest possible cross-node topology. It can only run with low concurrency and a conservative parallelization strategy, and it has essentially no redundancy—one failed GPU or machine degrades or takes down the service. It suits POC validation, small-scale internal trials, and generating real performance and cost data for a formal project approval. In short, at this tier you are buying qualification, not experience.

3.2 Comfort tier: keep it alive inside the organization

Memory and compute headroom improve significantly. You can serve higher concurrency and longer contexts, with room left over for canary releases, capacity drills, and failover. This tier fits medium-scale internal production scenarios such as enterprise knowledge bases or R&D coding assistants, and it is the realistic starting point for most large enterprises that build their own clusters.

3.3 Recommended tier: a validated engineering form factor

Eight 64-GPU supernodes make up the topology repeatedly validated by the model’s publisher and compute platforms: latency, throughput, and stability are pre-tuned, and the form factor naturally supports fault tolerance and rolling upgrades. It is intended for production services exposed to external users under high concurrency. Note that the recommended tier is not simply “more GPUs”—it is “more GPUs that can work together efficiently,” for the reasons in the next section.

3.4 Beyond the hardware budget: the hidden costs

The hardware purchase is only the beginning. Real total cost of ownership (TCO) also includes:

• Facilities and power: an 8 × 64-GPU cluster can draw on the order of 100 kW, which calls for dedicated racks and upgrades to power delivery and cooling.
• Electricity: a rough calculation—assume an average draw of 100 kW, 8,760 hours per year, and an electricity price of RMB 1 per kWh; power alone then costs on the order of RMB 800,000–900,000 per year.
• Networking and storage: RDMA high-speed fabrics, plus fast TB-scale storage and backups for the weights.
• Staffing: an engineering team familiar with parallelization strategies, inference frameworks, and cluster operations.
• Depreciation and iteration: hardware is typically depreciated over 3–5 years while model generations turn over roughly every six months—at procurement time you should already be planning how the next-generation model will be served.

So “RMB 8 million to get in the door” is just the ticket price. Once TCO is laid out, the real gap between the three tiers is larger than the sticker prices suggest.

4. Why “Supernode” Became the Keyword

4.1 The dividing line between scale-up and scale-out

Traditionally, compute capacity scaled by adding machines, connected over general-purpose networks. For trillion-parameter models, however, tensor parallelism (TP) and expert parallelism (EP) generate heavy AllReduce and All-to-All traffic on every forward pass; cross-node bandwidth and latency determine whether that communication stays a gap between compute steps or becomes the bottleneck itself.

The supernode idea expands the high-speed interconnect domain from “8 GPUs inside one machine” to “dozens or hundreds of GPUs across one or more racks,” so that even coarse-grained parallel strategies run inside a high-bandwidth, low-latency domain. That is why the recommended tier points to a supernode form factor—not simply more GPUs, but more GPUs that can cooperate efficiently.

4.2 MoE architectures amplify interconnect requirements

Trillion-parameter models generally use a Mixture-of-Experts (MoE) design: each inference pass activates only a subset of parameters, which reduces compute demand, but routed tokens must be shuttled between experts, so All-to-All traffic takes a significantly larger share. If interconnect bandwidth is insufficient, no amount of extra GPUs helps—they simply idle while waiting for data.

4.3 Day-0 adaptation: shrinking the window from “released” to “usable”

Another trend worth watching is “Day-0 adaptation”: compute clusters and inference frameworks complete adaptation and validation on the same day the model weights are open-sourced, so users no longer face a long “wait for compatibility” cycle. What this really tests is a vendor’s engineering depth in parallel strategies, operator libraries, and inference engines—and adaptation capability itself is becoming a core competitive differentiator for compute platforms.

5. An Engineering Checklist: From Downloading Weights to a Stable Service

1. Weight acquisition and verification. For TB-scale weights, use object storage as a staging relay and download shards in parallel; after the download, verify per-shard hashes against a complete manifest, and keep an offline backup in production environments. What takes days to distribute over a 100 Mbps public link takes minutes over a datacenter network.

2. Parallelization strategy and inference framework. The usual combination is tensor parallelism (TP) × pipeline parallelism (PP) × expert parallelism (EP) × data parallelism (DP). The rule of thumb: TP is the most communication-sensitive and should stay within a single machine; use PP/EP for cross-machine splits. Example configuration (pseudo-code, fields shown for illustration only):

deployment:
  replicas: 2            # number of inference instances
  tensor_parallel: 8     # 8-way tensor parallelism within a machine
  pipeline_parallel: 2   # pipeline parallelism across machines
  expert_parallel: 4     # expert parallelism
  max_context_len: 32768
  quantization: fp8      # weight quantization

3. Quantization and cost optimization. FP8/INT4 weight quantization can cut the weight footprint by half or more, but the accuracy trade-off must be evaluated. KV cache quantization and paged management pay off noticeably in long-context scenarios. Splitting prefill and decode onto separate pools (PD disaggregation) lets each phase run on the topology that suits it best, while continuous batching and speculative decoding raise effective throughput.

4. Serving and capacity planning. Run concurrency load tests before going live, pin down SLOs for throughput, time to first token, and end-to-end latency, and then size the cluster from those results—rather than “serving as much as the GPUs happen to allow.”

5. Observability and operations. Monitoring and alerting should cover GPU utilization, memory headroom, network throughput, and request queue length; retain logs and request traces for incident forensics; and establish canary releases plus fast rollback for model versions.

6. Security and compliance. One of the biggest values of on-premises deployment is that data never leaves your environment, so network isolation, permission auditing, and content-safety policies are all mandatory. When the service is exposed externally, gateway-level rate limiting, authentication, and audit logging are equally non-negotiable.

6. Self-Host or Use an API? A Simple Decision Path

Not every company needs to sink a RMB 30 million budget into a datacenter. Three questions are enough for a quick decision:

1. Must your data stay inside your environment? In strongly regulated scenarios such as finance, healthcare, or government, on-premises or dedicated compute is the precondition.
2. Is your call volume large and stable? With high, predictable volume, the marginal cost of self-hosting is lower; with spiky, exploratory usage, an API is the better deal.
3. Can your team operate a cluster? Without a dedicated team, the money saved on hardware gets lost many times over filling operational pitfalls.

A middle path is also common: start with managed cloud services to validate the business—major cloud platforms, including Baidu AI Cloud, offer general-purpose capabilities such as GPU compute instances, model hosting, and monitoring and alerting—and only decide whether to move to a fully self-built on-premises cluster once call volume and business value are proven.

7. Conclusion

From “can’t buy the accelerators, can’t get the operators to run” to “adaptation completed on release day and a checklist you can order against,” on-premises deployment of trillion-parameter models is turning into a standard engineering problem. It is still expensive—RMB 8 million to start, and RMB 30 million for the recommended configuration—but it is now computable, plannable, and deliverable.

For any enterprise, what matters is not “everyone else deployed it, so we should too,” but working out your own three ledgers: the memory ledger, the cost ledger, and the value ledger. Once those numbers are clear, whether RMB 8 million is an entry ticket or wasted money will answer itself.