Compute Estimation for 3T–5T LLM Training and Inference

    This article presents a structured method for estimating the compute, GPU memory, parallelism, checkpointing, KV cache, and latency requirements of large language models in the 3T–5T parameter range. It uses explicit assumptions, sample configurations, formulas, and a troubleshooting checklist, with all figures intended for rough planning and calibration through measurements.

    Compute Estimation for 3T–5T LLM Training and Inference

    In recent years, the parameter scale of large language models has continued to grow. Some teams have disclosed plans for base models at the 3T–5T scale, but a plan is a target, not a training result. This article treats 3T–5T as a parameter-scale assumption and does not cite unverified specific data. Increasing parameter scale usually improves complex reasoning and multimodal understanding, but training time, memory footprint, and inference cost also rise significantly. The estimates below use a consistent accounting basis and state the assumptions at each step.

    1. Establish a Consistent Parameter and Memory Basis

    Assume the model has 5T total parameters (5 × 10^12). Weights can be stored at different precisions:

    Weight precision Bytes per parameter Memory for 5T parameters
    FP16 / BF16 2 10 TB
    FP8 / INT8 1 5 TB
    INT4 0.5 2.5 TB

    Training requires far more state than the weights alone. With mixed precision and the Adam optimizer, the per-parameter footprint breaks down as follows:

    • FP16 weights: 2 bytes
    • FP16 gradients: 2 bytes
    • FP32 master weights: 4 bytes
    • FP32 first moment: 4 bytes
    • FP32 second moment: 4 bytes

    The total is 2 + 2 + 4 + 4 + 4 = 16 bytes. For 5T parameters, the training state is approximately 5 × 10^12 × 16 bytes = 80 TB, before accounting for activations, communication buffers, and framework overhead. A single 80GB GPU cannot hold this, so training must shard state across a large number of devices.

    2. Three Major Training Challenges

    2.1 Memory Capacity and Hybrid Parallelism

    A common approach is 3D hybrid parallelism, combining data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP). The following is an illustrative configuration assuming a Megatron-LM-style framework, where total GPUs equal the product of the parallelism degrees:

    1. # Illustrative configuration: 3D hybrid parallelism, total GPUs = TP × PP × DP
    2. config = {
    3. "tensor_model_parallel_size": 8, # Tensor parallelism within an 8-GPU node, relying on intra-node high-speed interconnect (e.g., NVLink)
    4. "pipeline_model_parallel_size": 32, # 32 pipeline stages, with inter-stage communication over the inter-node network
    5. "data_parallel_size": 64, # 64 data-parallel replicas
    6. }
    7. total_gpus = 8 * 32 * 64 # 16,384 GPUs in total

    The three forms of parallelism have different costs:

    • TP splits individual matrix multiplications. It has the most frequent communication and requires intra-node high-speed interconnect, so it is usually limited to 8 GPUs within a node.
    • PP places different layers on different devices. It passes only activations between stages, but it introduces pipeline bubbles.
    • DP replicates the full model and synchronizes gradients at every step, making it sensitive to network bandwidth. Pure DP replicates the full training state, so it must be combined with sharding methods such as ZeRO.

    The pipeline bubble fraction can be estimated as: bubble fraction = (p - 1) / (m + p - 1), where p is the number of pipeline stages and m is the number of micro-batches. With p = 32 and m = 64, the fraction is 31 / 95, or about 32.6%—close to one third. This means devices are idle for a substantial portion of time. Increasing m reduces the bubble fraction, but each in-flight micro-batch consumes activation memory, so m is constrained by memory.

    The table below summarizes the trade-offs:

    Dimension TP PP DP
    Primary sharding target Matrices within a layer Different layers Data batches
    Communication pattern Collective communication before and after each layer, most frequent Point-to-point activation transfer between adjacent stages Gradient all-reduce at every step
    Bandwidth requirement Highest; requires intra-node interconnect Moderate; can span nodes Related to model size; requires cross-node communication
    Main cost Higher communication share Pipeline bubbles High memory footprint when replicating state; requires sharding

    There is no universal optimum for these three parameters. They must be tested against node network bandwidth, memory headroom, and throughput targets.

    2.2 MoE Routing and Communication Overhead

    A mixture-of-experts (MoE) model is a common way to control per-token compute as total parameter count grows. The model activates only a subset of experts at a time, so per-token compute does not grow linearly with total parameters. However, all parameters must still reside in GPU memory or host memory, so MoE faces the same storage pressure described above.

    The main risk in MoE is the routing layer. The gating network assigns tokens to experts. If the assignment is uneven, devices hosting a few popular experts become overloaded while other devices wait, creating inter-node latency differences. During tuning, check the following four items:

    • Expert load ratio: the number of tokens each expert receives in a batch, focusing on the ratio of maximum to average. Example threshold: a ratio below 1.2 is relatively healthy, and a ratio above 1.5 should trigger routing adjustments.
    • Auxiliary load-balancing loss weight: too large a weight can hurt model quality, while too small a weight can worsen load imbalance. Monitor both quality metrics and load ratio on the validation set.
    • Expert capacity factor: whether tokens exceeding capacity are dropped and whether the drop rate is acceptable. Example threshold: if the drop rate remains above 1%, check whether routing is imbalanced.
    • All-to-All communication share: communication time as a fraction of total step time. Example threshold: if the share exceeds 30%, the network topology or parallel partitioning usually needs adjustment.

    All-to-All communication requires sufficient bisection bandwidth. If the cluster has too many tiers or suffers from congestion, this communication can become a major cost. The thresholds above are examples and must be calibrated for the actual model and cluster.

    2.3 Training Stability and Checkpoint Resume

    In clusters with thousands of GPUs, hardware failures are relatively common. If a training run lasts weeks or months, a single failure can interrupt the job. Checkpoint-based resume depends on periodic checkpointing. A full checkpoint includes optimizer state, so a 5T model requires about 80 TB, far more than the 10 TB weight file.

    To complete a save within 10 minutes, the required write throughput is:

    Write throughput = total state size ÷ save window = 80 TB ÷ 600 s ≈ 133 GB/s

    This throughput usually requires a high-performance parallel file system or high-throughput object storage.

    Checkpoint frequency must balance storage overhead against rollback loss. Saving too often consumes training time with writes; saving too infrequently increases the number of steps that must be recomputed after a failure. The optimal interval can be approximated with Young’s formula: T_opt ≈ √(2 × C × M), where C is the time for one save and M is the cluster mean time between failures (MTBF). For example, if C is 10 minutes and M is 4 hours (240 minutes), T_opt ≈ √4800 ≈ 69 minutes. This approximation assumes failures follow an exponential distribution; actual values should be based on historical cluster failure data.

    Common engineering practices include:

    • Asynchronous saving: copy state to memory first, then write to storage from a background thread to reduce blocking of training steps.
    • Checksums and atomic commit: record a checksum for each save and update the latest-version pointer only after the write completes, preventing reads of incomplete files.
    • Retain multiple historical versions so that a corrupted latest version can be rolled back.
    • Periodic restore drills: actually resume training from a checkpoint to confirm that the loading path, shard mapping, and random seeds are restored correctly.

    3. Main Inference Bottlenecks

    3.1 Weight Memory and GPU Count

    For weights alone, the minimum number of 80GB GPUs is:

    • FP16: 10,000GB ÷ 80GB = 125 GPUs
    • INT8: 5,000GB ÷ 80GB ≈ 62.5, rounded up to 63 GPUs
    • INT4: 2,500GB ÷ 80GB ≈ 31.25, rounded up to 32 GPUs

    Actual deployment should reserve 10%–20% headroom for activations, communication buffers, and framework overhead. KV cache also consumes GPU memory, as described in the next section.

    3.2 KV Cache Growth

    During autoregressive decoding, the model reads the keys and values of all previous tokens for every generated token. To avoid recomputation, the system caches them in GPU memory; this cache is the KV cache.

    The formula is:

    KV cache bytes per token = 2 × L × n_kv × d_h × b

    Total footprint = bytes per token × sequence length × concurrent requests

    Here, L is the number of layers, n_kv is the number of KV heads (smaller than the number of attention heads when grouped-query attention, GQA, is used), d_h is the dimension per head, and b is the number of bytes per element (2 for FP16, 1 for FP8 or INT8). The factor 2 accounts for storing both K and V.

    Example: assume L = 120, n_kv = 8, d_h = 128, and b = 2 (FP16).

    • Per token: 2 × 120 × 8 × 128 × 2 = 491,520 bytes, about 480 KiB.
    • 32K context (32,768 tokens): about 15 GiB per request.
    • 32 concurrent requests: about 480 GiB.
    • 128K context: about 60 GiB per request, and about 1,920 GiB for 32 concurrent requests.

    If every attention head stores its own KV (n_kv = 128), the cache grows by 16× under the same configuration, and a single 128K-context request requires about 960 GiB. Therefore, KV head design has a major impact on long-context serving. Using FP8 for KV cache can halve the footprint, but quality must be validated on tasks such as long-text retrieval. The architecture parameters above are illustrative; actual values should follow the model configuration.

    3.3 Weight Quantization

    Weight memory equals parameter count multiplied by bit width and divided by 8. INT4 quantization typically uses group-wise scaling; for example, every 128 parameters share one FP16 scale factor, adding 16 ÷ 128 = 0.125 bits of overhead, for an effective rate of about 4.125 bits per parameter.

    Quantization introduces accuracy loss and should be checked on evaluation sets relevant to the business, not only on aggregate leaderboards. Focus on three task types: long-context question answering, format accuracy for tool calls, and error-sensitive tasks such as math and code. If the relative drop on critical tasks exceeds 1 percentage point, consider keeping sensitive layers at higher precision or adjusting the group size.

    Decoding also depends on bandwidth. For each generated token, the system usually reads the activated weights and the corresponding KV cache, so decode speed is often limited by GPU memory bandwidth. The theoretical lower bound for per-step latency can be estimated as:

    Per-step lower bound = bytes read per step ÷ total memory bandwidth participating in computation

    For a dense 5T FP16 model, each step reads about 10 TB of weights. If distributed across 128 GPUs with about 3 TB/s each, each GPU reads about 78GB, giving a lower bound of about 26 ms per step. MoE models read only activated experts, so actual cost is lower than this estimate. This lower bound excludes inter-GPU communication; actual latency will only be higher.

    4. Estimation Template and Troubleshooting Checklist

    Use the following steps for estimation:

    1. Determine parameter count, weight precision, KV cache precision, target context length, and concurrency.
    2. On the training side, compute training state: P × 16 bytes (mixed precision plus Adam), then divide by GPU count and check whether it exceeds available memory per GPU.
    3. On the inference side, compute the number of GPUs for weights by dividing weight bytes by available memory per GPU, then add KV cache footprint.
    4. Determine parallelism and estimate pipeline bubble rate and communication share.
    5. Reserve 10%–20% headroom and calibrate results with small-scale measurements.

    Use the table below for troubleshooting. All thresholds are examples and must be calibrated for the actual model and cluster.

    Check item Metric Example abnormal signal Common action
    Pipeline bubble (p - 1) / (m + p - 1) Above 20% Increase micro-batch count or reduce PP stages
    MoE load Max load ÷ average load Above 1.5 Adjust auxiliary loss weight or capacity factor
    All-to-All share Communication time ÷ step time Above 30% Check network topology and reduce cross-node expert placement
    Checkpoint write State size ÷ save window Save duration exceeds 10% of interval Use asynchronous saving and parallel file systems
    KV cache footprint Formula result ÷ available memory Above 60% and concurrency cannot increase Use GQA, KV quantization, or limit concurrency and context length
    Quantization accuracy Relative drop on benchmarks Critical task drop exceeds 1 percentage point Keep sensitive layers at higher precision and adjust group size

    5. Conclusion

    Compute estimation can be broken down into three decisions.

    1. How much compute the project needs: on the training side, start with parameter count × 16 bytes, then determine GPU count with parallel partitioning; on the inference side, start with GPU count for weights, add KV cache footprint, and estimate latency with the bandwidth lower bound. Both sides need headroom.
    2. Which areas are most likely to fail: inter-node communication (TP, pipeline bubbles, and MoE All-to-All), checkpoint writing and restore, and KV cache footprint in long-context scenarios.
    3. Decision order: first use formulas for a rough estimate and establish the hardware lower bound; then use small-scale measurements to calibrate parallelism, routing parameters, and quantization; finally commit to full-scale training or deployment.

    The numbers in this article are based on assumed parameters and illustrative configurations. Actual results should be validated with measurements.