Deploying Large Language Models On-Premises: A Guide to Hardware Planning and Cost Estimation

    This guide explains how to plan hardware and estimate costs for on-premises large language model deployment. It covers how parameter precision, context length, and concurrency affect GPU memory, how to estimate weight and KV cache requirements, and how parallelism strategy and redundancy determine the number of nodes.

    1. Define the Requirements First

    The same model can cost several times more or less to deploy, depending on the setup. Four main variables drive the result: parameter count and weight precision, target context length, concurrent requests and latency requirements, and the availability target over the deployment period. Until these inputs are fixed, any budget figure is only a guess. Cost estimates should therefore start from business requirements and work backward to hardware, rather than selecting a machine first and checking what it can run.

    Before project approval, prepare a requirements sheet that includes at least the following: average daily request volume, peak concurrency, input and output length per request, upper limits for time to first token and total response time, and acceptable degradation strategies. This sheet feeds all later calculations and serves as the basis for explaining the budget to management.

    2. Estimating Memory: Model Weights

    Model weights are the first and largest memory consumer. The estimate is straightforward:

    Weight memory ≈ parameter count × bytes per parameter

    Common precisions map to these byte sizes: FP16 or BF16 uses 2 bytes per parameter, INT8 uses 1 byte, and 4-bit quantization uses about 0.5 bytes. Low-bit quantization adds scaling factors and group metadata, so actual usage is usually somewhat higher than the theoretical value. Confirm it through measurement.

    Take a hypothetical 1000B (1 trillion) parameter model as an example. With 4-bit weights, the weights alone need on the order of 500 GB of GPU memory. With 16-bit weights, the same model reaches roughly 2 TB. Precision choice directly determines how many GPUs are needed, so the trade-off between precision and output quality should be settled before the estimate is made.

    For Mixture-of-Experts (MoE) architectures, distinguish two concepts. Total parameter count determines the storage and loading space that the weights require. Active parameter count determines the computation each token actually uses. The former affects memory capacity, while the latter mainly affects throughput and latency. Estimating memory from active parameters alone would seriously underestimate the requirement.

    3. Estimating Memory: KV Cache and Runtime Overhead

    Beyond the weights, the KV cache is often the second-largest memory consumer during inference. It grows linearly with concurrency and context length. The approximate formula is:

    KV cache ≈ 2 × layers × KV heads × head dimension × sequence length × concurrency × bytes per element

    The factor of 2 accounts for separate Key and Value caches. In architectures such as grouped-query attention, the number of KV heads is much smaller than the number of attention heads, which reduces memory pressure. In long-context scenarios, this term can exceed the weights themselves, so it must be calculated separately.

    In addition, reserve memory for activations, communication buffers, and the inference framework itself. A margin is typically reserved, and its exact size should be determined through load testing rather than set by intuition.

    The following example code can be used for a preliminary estimate. The model structure parameters are illustrative values and do not correspond to any real model’s configuration:

    1. def weight_gib(params_billion: float, bytes_per_param: float) -> float:
    2. """Weight memory in GiB."""
    3. return params_billion * 1e9 * bytes_per_param / 1024**3
    4. def kv_cache_gib(layers: int, kv_heads: int, head_dim: int,
    5. seq_len: int, batch: int, bytes_per_elem: int = 2) -> float:
    6. """KV cache memory in GiB."""
    7. total = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    8. return total / 1024**3
    9. if __name__ == "__main__":
    10. w = weight_gib(1000, 0.5)
    11. kv = kv_cache_gib(layers=60, kv_heads=8, head_dim=128,
    12. seq_len=32768, batch=16)
    13. print(f"Weights: ~{w:.0f} GiB, KV cache: ~{kv:.0f} GiB")

    4. Parallelism Strategy and Node Count

    When a model does not fit on a single GPU, it must be split across multiple GPUs. Common approaches include the following:

    • Tensor parallelism: Splits the matrices within each layer across GPUs. It requires frequent communication and suits high-bandwidth interconnects within a single node.
    • Pipeline parallelism: Splits the model by layers across devices. Communication volume is smaller across nodes, but pipeline bubbles reduce utilization.
    • Expert parallelism: In MoE models, distributes different experts across devices. It requires heavy all-to-all communication.
    • Data-parallel replicas: When throughput demand is high, replicates the complete instance to increase concurrency, at the cost of additional resources.

    A rough node count comes from dividing the total required memory by the usable memory per node, rounding up, and then adding redundant nodes for availability. Cross-node communication lowers effective utilization, so the actual node count is usually higher than the theoretical minimum. This effect is especially pronounced for MoE models.