Deploying Large Open-Source LLMs On-Premises: Hardware Sizing and Cost Planning

    Deploying trillion-parameter open-source models requires more than enough GPU memory to load the weights. This guide explains how to estimate memory requirements, size multi-node GPU clusters, and budget for networking, storage, power, and operations. It also outlines practical validation steps and the trade-offs between hosted inference and infrastructure ownership.

    1. Why On-Premises Deployment Costs So Much

    As the open-source community releases weights for trillion-parameter sparse mixture-of-experts (MoE) models, more organizations are evaluating whether to run them in their own data centers.

    The benefits are clear: data stays within the organization’s environment, the inference pipeline remains under its control, and long-term usage does not incur per-token service charges. The trade-off is a hardware requirement far beyond what many teams initially expect.

    Consider a trillion-parameter-class sparse MoE model distributed in a low-precision format such as MXFP4-style 4-bit floating point. Even with heavily compressed weights, the files can occupy roughly 1.5 TB. This has several implications:

    • A single GPU cannot hold the model. Even with 141 GB of memory per GPU, loading the weights requires more than ten GPUs, before allowing for the KV cache, activations, and framework overhead.
    • Deployment becomes a multi-node infrastructure project. Inference frameworks must partition the model across GPUs and servers using techniques such as tensor parallelism and expert parallelism. Inter-node bandwidth, including RDMA networking, becomes a major throughput constraint.
    • Procurement is an infrastructure-scale expense. This is not a matter of adding a few graphics cards. The investment can reach tens of millions of Chinese yuan.

    The budget figures below retain the source estimates in CNY. They should be treated as planning references, not as current quotations for any particular market.

    2. Three Questions to Answer Before Sizing the Budget

    Before estimating costs, clarify three factors that determine cluster size and spending requirements.

    Model Size and Weight Format

    First, confirm the model’s total parameter count and the precision of its released weights. For native 4-bit weights, a rough planning formula is:

    1. Estimated weight-memory budget ≈ Total parameters × Bytes per parameter × Overhead factor (1.2–1.5)

    For a sparse MoE model with 2.8 trillion total parameters:

    1. total_params = 2.8e12
    2. bytes_per_param = 0.5 # MXFP4: 4-bit = 0.5 bytes
    3. weight_gb = total_params * bytes_per_param / (1024**3)
    4. # Approximately 1,304 GiB, or about 1.3 TiB, of raw weights
    5. # Allow 1.5x or more for runtime needs: KV cache, activations, and framework memory

    Context length also matters. Models supporting million-token contexts can require substantial KV-cache capacity when processing long sequences. Hybrid architectures combining linear and sparse attention can reduce this pressure, but cluster planning still needs headroom for peak workloads.

    GPU Memory and Cluster Size

    Using high-end data center GPUs with 141 GB of memory each as a reference:

    • Weights only: Approximately 1.6 TB of weights divided by 141 GB per GPU requires about 12 GPUs, with essentially no runtime headroom.
    • Runtime allowance: KV cache, activations, and communication buffers can require another 30%–50% of memory, bringing the practical estimate to roughly 16–20 GPUs.
    • Eight-GPU server configurations: This corresponds to approximately two to three servers for a minimum viable deployment.

    These are capacity estimates for getting the model running—not guarantees of production performance.

    To support concurrent users with acceptable time to first token and throughput, planning may call for four to eight eight-GPU nodes, or 32–64 GPUs, connected through a high-speed, lossless network such as 400G-class RDMA.

    Performance Targets and Budget Tiers

    A useful starting point is to separate requirements into three tiers:

    Deployment tier Cluster size, in eight-GPU nodes Typical use case Indicative budget
    Proof of concept 2–3 Functional validation and internal testing CNY 8–12 million
    Production deployment 5–8 Services with modest concurrency CNY 20–30 million
    High-throughput serving 10 or more High concurrency and low-latency SLAs More than CNY 30 million

    GPU prices can fluctuate significantly with supply and demand. Procurement prices for the same model can differ by more than 30% between purchase periods. Use current quotations and reserve an additional 10%–15% budget contingency.

    For an illustrative configuration budgeted at around CNY 30 million, GPUs are only part of the cost. A reasonable allocation is approximately:

    1. Compute nodes: 75%
      Multiple eight-GPU servers, including CPUs, system memory, and local NVMe storage. The source estimate places each complete server at approximately CNY 2.5–3.5 million.

    2. Networking: 10%
      The reference configuration calls for at least two 400G RDMA network adapters per node, along with suitable switches. Networking is essential to efficient multi-node parallelism; cutting this budget too aggressively can severely reduce throughput.

    3. Storage and software: 8%
      Weight files alone can exceed 1.5 TB. Checkpoints, logs, and evaluation datasets add to the requirement. A distributed parallel file system or high-performance object storage is recommended, with capacity planning starting at approximately 50 TB.

    4. Facilities and operations: 7%
      Servers in this class commonly consume more than 10 kW each. Data center power delivery, liquid or air cooling, and UPS capacity may require dedicated upgrades. These expenses are often underestimated.

    4. Practical Steps Before Committing to Deployment

    Before submitting a budget request, work through the following steps.

    Validate the Stack with a Smaller Quantized Model

    Use a smaller model with the same architecture to test the inference framework, parallelism strategy, and network configuration. This helps uncover engineering issues before the full cluster arrives.

    Compare Renting with Buying

    If average daily usage is low, hosted inference may have a substantially lower total cost of ownership than an on-premises deployment. Owning the infrastructure is more likely to offer clear economic advantages under sustained, high-utilization workloads.

    Evaluate Throughput Optimization Options

    Techniques such as multi-token prediction (MTP), sparse attention, and quantization-aware scheduling can improve the useful output of the hardware. Inference-framework selection and tuning therefore deserve dedicated engineering effort.

    Define the Benchmark Methodology

    Do not rely solely on published benchmark scores. Run end-to-end load tests using your own business data and workloads. Evaluation scores and real-world user experience can differ substantially.

    5. Conclusion

    Deploying a trillion-parameter open-source model on-premises is fundamentally a data center infrastructure project.

    The source estimates place a minimum viable configuration at around CNY 8 million, with a recommended production configuration near CNY 30 million. Networking, storage, facilities, and operations must all be included alongside compute hardware.

    For most organizations, the more measured approach is to validate business value through hosted inference first, then assess the return on an on-premises investment. A budget proposal should include both the sizing assumptions and the full cost breakdown so that the decision rests on a complete infrastructure cost model.