Cost Breakdown and Engineering Approaches for Local Deployment of Very Large Open-Source Models

1. Background: When Model Weights Exceed Single-Node GPU Memory

As open-source large model capabilities improve, parameter counts and weight file sizes continue to grow. Some recently released open-source models now have weight files exceeding 1.5 TB, while mainstream high-performance computing nodes typically offer around 1 TB of total GPU memory. This means that even without accounting for KV cache, activations, and temporary buffers during inference, loading the model weights alone already exceeds the capacity of a single node.

Under these conditions, the traditional approach of loading a model on one node and running inference on one GPU is no longer viable. The model must be partitioned across multiple nodes, with strategies such as tensor parallelism, pipeline parallelism, or expert parallelism. At the same time, hardware costs rise sharply, and local deployment shifts from a purely technical problem into a budget problem.

2. Core Constraints: Weights, GPU Memory, and Inference Overhead

2.1 Rapid Growth in Weight File Size

Taking leading open-source models as examples, weight files can reach approximately 1.5 TB. This figure usually refers to the storage size at the original precision or after compression. When actually loaded into GPU memory, additional factors must be considered:

• Parameter precision conversion, such as BF16, FP16, or quantized INT8 and FP8 formats;
• Optimizer states, which are only required during training and do not consume memory during inference;
• The distribution of parameters across embedding layers, attention layers, feed-forward networks, and other model components.

If loaded in FP16, 1.5 TB of weights corresponds to roughly 750 billion parameters. With INT8 quantization, the weight size can be reduced to approximately 750 GB. However, quantization introduces accuracy loss and must be evaluated based on task tolerance.

2.2 GPU Memory Usage Is Not Just Weights

GPU memory consumption during inference typically consists of four parts:

1. Model weights: determined by parameter precision;
2. KV cache: related to batch size, sequence length, number of layers, and number of attention heads;
3. Activations: relatively small during inference, but still require reserved space;
4. Communication buffers and temporary tensors: especially important in multi-GPU parallel setups.

Therefore, even when weights fit exactly into GPU memory, additional space must be reserved for KV cache and communication buffers. In practice, it is advisable to keep GPU memory utilization at 70–80%. Otherwise, long-sequence requests can easily trigger out-of-memory errors.

2.3 The Gap Between Single-GPU Memory and Whole-Node Memory

Assume a single node is equipped with 8 accelerators, each with approximately 140 GB of memory. The theoretical total memory of one node would be about 1.12 TB. If the model weights are 1.5 TB, a single node cannot hold the entire model. Even with 8-way tensor parallelism within the node, the remaining space would not be sufficient for normal inference.

At this point, there are two possible paths:

• Multi-node scaling: partition the model across two or more nodes connected by high-speed interconnects;
• Low-bit quantization: compress the weights to under 1 TB, accepting some accuracy degradation.

3. Minimum Viable Approach: Multi-Node Multi-GPU Partitioning

3.1 Tensor Parallelism and Pipeline Parallelism

For very large models, common parallelism strategies include:

• Tensor parallelism: partitioning matrix operations within individual layers across multiple GPUs. This requires high inter-GPU bandwidth because of substantial communication;
• Pipeline parallelism: partitioning by layers, with different devices handling different layers. Communication volume is relatively lower, but pipeline bubbles can reduce utilization;
• Expert parallelism: for Mixture-of-Experts models, distributing different experts across different devices. Active parameters are smaller, but routing and scheduling become more complex.

Real deployments usually combine these strategies. For example, tensor parallelism can be used within a node, while pipeline parallelism is applied across nodes. This hybrid approach balances communication pressure and device utilization.

3.2 Minimum Hardware Configuration Estimate

Taking a 1.5 TB model loaded in FP16 as an example:

• Each GPU has approximately 140 GB of memory. At 75% practical utilization, about 105 GB is available per GPU;
• Weights alone require roughly 15 GPUs;
• After accounting for KV cache and communication buffers, at least 20 GPUs are needed;
• Assuming 8 GPUs per node, at least 3 nodes are required.

If a single node costs approximately 2.6 million CNY at market price, three nodes would cost around 7.8 million CNY. Adding switches, optical modules, data center modifications, and operational costs, the initial investment approaches 8 million CNY. This estimate is consistent with the commonly cited figure of “8 million CNY minimum” in the community.

3.3 Considerations for Recommended Configurations

A recommended configuration is usually intended for production environments and must provide:

• Concurrent request handling capability;
• Sufficient KV cache headroom for long-sequence inference;
• Multiple replicas or partial redundancy for higher availability;
• Expansion room for future model upgrades.

As a result, recommended configurations are often 2–4 times larger than the minimum configuration. If the minimum configuration is 3 nodes, a recommended configuration may reach 8–12 nodes, corresponding to roughly 20–30 million CNY. The key bottleneck here is GPU memory, not peak compute. In many scenarios, the computational capacity of the GPUs is more than sufficient, but if GPU memory is insufficient, inference cannot start at all.

4. Cloud Adaptation and Day-0 Support

4.1 Supernodes and High-Speed Interconnects

For teams that cannot deploy locally, cloud-based high-performance computing clusters provide an alternative. Cloud providers typically offer “supernode” products that combine a large number of accelerators into a logical supercomputing unit through high-speed interconnect networks.

The core value of a supernode lies in:

• Providing a GPU memory pool far larger than any single node;
• Offering much higher inter-GPU bandwidth than ordinary inter-node communication;
• Supporting multi-GPU parallel training and inference for very large models.

When a new model is released, if a cloud platform can complete adaptation on “Day 0,” users can run that model in cluster mode immediately without purchasing hardware themselves.

4.2 Key Adaptation Tasks

Day-0 adaptation typically includes:

1. Weight conversion: converting open-source weights into the format supported by the cloud platform;
2. Parallelism configuration: setting tensor parallelism and pipeline parallelism degrees according to the model structure and cluster topology;
3. Communication optimization: tuning NCCL parameters, enabling RDMA, and optimizing AllReduce paths;
4. Inference service integration: integrating the model into a unified inference framework with standard API support;
5. Accuracy verification: comparing outputs from the original implementation and the adapted version to ensure consistency.

These tasks may seem standard, but for a 1.5 TB model, each step can encounter GPU memory shortages, communication bottlenecks, or unsupported operators. Significant engineering experience is required.

4.3 Implications for Enterprises

For most enterprises, building three or more high-performance nodes involves not only procurement cost pressure, but also ongoing investment in data center load capacity, power and cooling, network upgrades, and operations staff. Cloud usage on demand, while not cheap per use, avoids idle assets and hardware iteration risk. When models are updated, the cloud can switch quickly, while local hardware may need to remain in service for a longer period.

5. How to Write a Budget Request: A Suggested Template

When requesting budget approval, it is advisable to highlight the following points:

1. Background
   Current business demands require adopting larger open-source models.
   The target model weights are too large for a single node.
2. Technical Option Comparison
   Option A: Local deployment
     - Minimum configuration: 3 nodes with 8 GPUs each, starting at approximately 8 million CNY
     - Recommended configuration: 8–12 nodes with 8 GPUs each, approximately 20–30 million CNY
     - Supporting costs: high-speed switches, optical modules, data center modifications, operations staff
   Option B: Cloud supernode
     - On-demand rental, no large upfront hardware investment
     - Rapid startup after Day-0 adaptation
     - Total cost may exceed local deployment when long-term usage is high
3. Risks and Recommendations
   - Local deployment carries hardware iteration risk; new models may require new hardware purchases
   - Cloud is more flexible and suitable for rapid validation and elastic scaling
   - Recommended approach: first validate the technology on the cloud, then evaluate long-term local deployment

This structured presentation is more persuasive than simply stating that “30 million CNY is needed.” The technical team needs to demonstrate that it has completed technical research, understands the boundaries of each option, and can provide comparable paths.

6. Summary

Local deployment of very large open-source models has moved from “just buy a GPU” to “seriously plan a cluster.” When weight size exceeds single-node GPU memory:

• Multi-node multi-GPU parallelism strategies are required;
• GPU memory planning becomes a more critical constraint than compute;
• Initial costs rise substantially, and 8 million CNY is only the minimum threshold at the physical limit;
• Cloud supernodes offer a more flexible alternative, especially for rapid validation and elastic scenarios.

For technical teams, the key task is not simply “how many GPUs to buy,” but finding an acceptable balance among GPU memory, precision, concurrency, latency, and cost, and translating that balance into a clear technical proposal and budget justification.