Deploying Trillion-Parameter LLMs: VRAM, Concurrency, and Cost Estimation

    A practical guide to deploying trillion-parameter large language models: how to define deployment goals, estimate VRAM requirements from parameter counts and quantization precision, choose hardware configurations for multi-GPU setups, and distinguish capacity from throughput when planning for concurrency.

    1. Start by Defining Your Deployment Goal

    The first step in deploying a large language model is specifying what “deployed” actually means. It can refer to three very different states:

    • Load and run: The weights fit into VRAM and inference works, with no speed requirement. This is sufficient for functionality checks and offline batch processing.
    • Single-user usable: A single request meets the minimum speed for interactive use. In typical chat scenarios, tens of tokens per second is a reasonable starting point; the exact target should account for time-to-first-token and overall user experience.
    • Multi-user serving: At a given concurrency level, every user still achieves the target speed. This requires throughput and scheduling capability, not just VRAM capacity.

    These goals differ dramatically in hardware scale and cost—often by several times. Before evaluating any solution, clarify three metrics: single-request speed, number of concurrent users, and the average context length per request.

    2. VRAM Estimation: From Parameter Count to Deployment Capacity

    2.1 Weight Footprint

    Weight VRAM can be estimated with this formula:

    Weight VRAM (GB) = parameter count (billions) × bits per parameter ÷ 8

    One billion parameters at 1 byte each is exactly 1 GB. For a model with 2.8 trillion parameters (2,800 billion) stored in 4-bit quantization, the weights require:

    2800 × 4 ÷ 8 = 1400 GB ≈ 1.4 TB

    Mixture-of-Experts architectures deserve special attention. Each token activates only a subset of parameters, so compute cost is determined by the activated parameter count. However, all weights must remain resident in memory—frequent swapping in and out will noticeably degrade speed. Therefore, plan VRAM based on total parameters, but evaluate compute speed based on activated parameters. Keep the two separate.

    2.2 Runtime Overhead

    Beyond weights, several other sources of overhead exist:

    • Activations: Intermediate tensors produced during the forward pass, scaling with batch size and sequence length.
    • KV cache: Key-value vectors that attention mechanisms store for already-generated content. This grows fastest in long-context scenarios.
    • Runtime and framework: Computation graphs, communication buffers, and VRAM fragmentation.

    These overheads cannot be covered by a single fixed ratio. In practice, start from the weight footprint, estimate separately based on context length and concurrency, then leave headroom. Any estimate must be validated by actually loading the model and running load tests.

    2.3 Impact of Precision on Capacity

    Quantization bit width directly determines weight size. For the same model:

    Precision Relative Weight Footprint
    16-bit float 4
    8-bit 2
    4-bit 1

    Lower precision introduces quality risks that should be verified with an evaluation set. Additionally, not all hardware natively supports every low-bit format. If the hardware must upconvert weights to higher precision at runtime, sufficient VRAM does not guarantee sufficient speed. When selecting hardware, confirm its native support for your target format.

    3. Hardware Selection: From Single GPU to Multi-GPU

    When weights exceed a single GPU’s VRAM, multi-GPU parallelism is required. Common strategies include tensor parallelism, pipeline parallelism, and expert parallelism. For single-node multi-GPU setups, the GPU count is usually a power of two—4 or 8, for example—because the split is more even and better supported by inference frameworks.

    The following function estimates the minimum number of GPUs needed to hold the weights plus runtime overhead:

    1. import math
    2. def min_gpu_count(weight_gb: float, gpu_mem_gb: float,
    3. overhead_ratio: float = 1.2) -> int:
    4. """Estimate the minimum GPU count for weights plus runtime overhead.
    5. overhead_ratio is a placeholder; replace it with measured values."""
    6. need = weight_gb * overhead_ratio
    7. return math.ceil(need / gpu_mem_gb)

    The result is only a lower bound. Real deployments must also fit the parallel configurations supported by the framework and reserve space for the KV cache, so the final GPU count is typically higher.

    Consider the following factors when selecting hardware:

    1. Per-GPU VRAM: Determines the minimum number of GPUs needed to hold the weights.
    2. Inter-GPU interconnect bandwidth: Multi-GPU communication is frequent; insufficient bandwidth slows every generation step.
    3. Framework support: Whether the hardware-plus-precision combination is supported by mainstream inference frameworks determines whether you can deploy directly.
    4. Resource model: Dedicated nodes, per-GPU partitioning, or on-demand instances—choose based on how stable your workload is.

    4. Throughput and Concurrency: Capacity vs. Speed

    Concurrency has two distinct layers.

    The first layer is how many requests can be resident simultaneously in VRAM. Every request consumes KV cache, so the longer the context, the fewer requests can be held at once.

    The second layer is whether the runtime can keep multiple requests at the target speed simultaneously. Aggregate output speed can be estimated as follows: