Large-Scale Model Training and Inference Optimization Under Compute Constraints

    This article explains how to optimize large-scale model training and inference when compute is constrained. It covers the real bottleneck of cost per effective token, GPU memory accounting, parallelism strategies and communication costs, numerical stability and failure recovery, long-context optimization, inference serving improvements, observability, cloud engineering, and evaluation before production. It provides practical principles, formulas, code sketches, and engineering trade-offs for teams building stable, cost-efficient training and inference pipelines.

    Large-Scale Model Training and Inference Optimization Under Compute Constraints

    1. The Real Problem: Not Whether to Train, but How Long It Takes and How Much It Costs

    When discussing very large models, public attention often focuses on parameter count. From an engineering perspective, however, the first constraint is usually the cost per effective token. Once the cluster size is fixed, there are only three main areas left to optimize:

    • memory efficiency;
    • communication efficiency;
    • effective cluster utilization, excluding time lost to failures, slow nodes, and idle periods.

    This is fundamentally different from optimizing ordinary backend services. In ordinary services, resources can often be obtained elastically, and performance issues can be mitigated by adding machines. In large-scale training, accelerators are tightly coupled by the physical interconnect topology. Scaling across machines has quickly diminishing marginal returns. Adding more machines can even reduce overall throughput because communication becomes a larger share of the work.

    Therefore, the core engineering question in large-scale training is: how to find the best parallelization split point under a given topology.

    This is also where the value of truly large-scale experiments lies. Many engineering boundaries only appear at large scale:

    • whether a combination of parallelism strategies degrades at a certain layer depth;
    • where the break-even point lies between optimizer state sharding and activation recomputation;
    • what triggers loss spikes.

    These issues may not reproduce in small-scale experiments. Only by running into them can teams develop reusable engineering experience.

    2. The GPU Memory Ledger: Understand It Before Choosing a Strategy

    Every parallelization strategy starts with understanding where memory is spent. For Adam-like optimizers with mixed-precision training, the approximate byte cost per parameter is:

    Component Precision Bytes per parameter
    Parameter copy for forward pass bf16 2
    Gradient bf16 2
    Master weights fp32 4
    First moment fp32 4
    Second moment fp32 4
    Total ≈16

    In other words, a 70B parameter model requires roughly 1.1TB of memory just for model states. It must be sharded across tens of GPUs to fit.

    Activations must also be added:

    1. per_layer_activation ≈ seq_len × batch × hidden × k × bytes
    2. # k is an empirical coefficient that depends on whether activation recomputation
    3. # and the attention implementation are used.
    4. # Without recomputation, k is usually much larger; it is a major driver of peak memory.

    After calculating this ledger, the strategy choice becomes much clearer:

    • Optimizer state sharding: shard optimizer states and gradients across the data parallel group. This is usually the most cost-effective first step because it adds little computation and only adds reduction communication for forward/backward passes.
    • Activation recomputation: trade computation for memory. In practice, recomputation is often applied only to some layers to balance throughput loss and memory savings.
    • Intra-layer tensor sharding: when a single GPU cannot hold one layer’s weights or activations, heavier sharding methods are required.

    3. Main Parallelism Dimensions and Their Communication Costs

    Large-scale training usually combines multiple forms of parallelism. Each dimension has different communication characteristics and must be placed according to interconnect bandwidth.

    Data parallelism / sharded data parallelism: gradients or parameter shards must be synchronized within the group. Communication volume grows with parameter count, so it is best suited to domains with sufficient bandwidth.

    Tensor parallelism: the weight matrices of a layer are split by rows or columns. The forward pass requires reduction/sum operations, and the backward pass does as well. This is the most communication-intensive approach because communication happens at every layer. A common rule is to keep the tensor-parallel degree within a single node’s high-speed interconnect domain. Using it across machines usually causes significant performance degradation.

    Pipeline parallelism: layers are divided into stages, and micro-batches are used to fill the pipeline. Its extra overhead is the bubble, approximately:

    1. bubble_ratio = (p - 1) / (m + p - 1)
    2. # p = number of stages, m = number of micro-batches per round
    3. # Increasing m reduces the bubble ratio but increases activation memory.

    Sequence parallelism / context parallelism: in long-context scenarios, the sequence dimension is split to reduce attention activation memory pressure.

    Expert parallelism: under sparse Mixture-of-Experts designs, experts are distributed across devices. This trades computation through routing, but introduces all-to-all communication and load imbalance problems.

    A practical configuration principle can be expressed as a function so teams can encode it:

    1. def choose_parallelism(nodes, gpus_per_node, layers, min_micro_batches=8):
    2. """
    3. Return a recommended parallelism configuration. Principles:
    4. 1) Tensor parallelism should not exceed the number of GPUs in one node,
    5. because cross-node TP communication is usually unacceptable.
    6. 2) The number of pipeline stages should divide the number of layers evenly.
    7. 3) The minimum number of micro-batches is derived from the bubble ratio.
    8. """
    9. tp = min(8, gpus_per_node) # Keep TP inside one node
    10. pp = next(d for d in range(2, nodes + 1) if layers % d == 0 and d <= nodes)
    11. dp = (nodes * gpus_per_node) // (tp * pp) # Remaining dimension for data parallelism
    12. # Derive micro-batch count from bubble ratio: require bubble_ratio <= 1/8
    13. pp_stages = pp
    14. micro_batches = max(min_micro_batches, pp_stages * 8 - pp_stages + 1)
    15. return dict(tp=tp, pp=pp_stages, dp=dp, micro_batches=micro_batches)

    4. Numerical Stability: What Really Breaks Training

    The failure modes of large-scale training are rarely just “it is too slow.” More often, they are “it suddenly crashes.” Three common issues stand out.

    Loss spikes. In the later stage of training over billions of tokens, the loss may suddenly jump by orders of magnitude and may not return to the original trajectory even if training continues. The usual troubleshooting order is:

    • whether the gradient norm is abnormally large;
    • whether attention logits overflow;
    • whether the learning rate and warmup are matched to the current batch size.

    Possible remedies include skipping the optimizer step, rolling back to the most recent stable checkpoint and lowering the peak learning rate, and applying normalization constraints to outputs of specific layers.

    Precision boundary choices. bf16 has a wide dynamic range but relatively lower precision, and it is a common default for current training. Lower-precision formats such as fp8 can provide clear throughput gains on some operators, but sensitive modules—such as normalization, residual accumulation, and optimizer master weights—need higher precision. Otherwise, errors can accumulate and amplify in deep networks.

    Hardware issues and slow nodes. At a scale of thousands of GPUs, single-GPU failures are almost inevitable. A bigger problem is slow nodes: they do not crash but degrade significantly in performance and can slow down the entire synchronization group. Therefore:

    1. - Record the latency distribution of every step and watch the ratio of P99 to median.
    2. - Add timeouts and segment-level timing for communication stages to locate degraded links.
    3. - Write checkpoints asynchronously to object storage and keep the most recent N healthy versions.
    4. - After isolating a faulty node, replace it automatically and resume from the latest checkpoint instead of rerunning everything.

    Checkpoint interval and rerun cost should be considered together. If the interval is too long, one failure can lose hours of work. If it is too short, write overhead erodes throughput. A reasonable approach is to overlap checkpoint writes with computation and decide whether to keep a checkpoint based on loss health.

    5. Long Context: Quadratic Complexity and Engineering Workarounds

    Attention computation has quadratic complexity with respect to sequence length, while KV cache memory usage grows linearly with length. Together, these determine the cost structure of long-context capability.

    There are three main optimization lines:

    1. Reduce cache size: let multiple query heads share key/value heads through grouped-query attention. This can compress cache usage several times over with little quality loss.
    2. Reduce peak memory: use chunked computation and online normalization in attention implementations. This avoids explicitly materializing the full attention matrix and reduces peak memory from O(L²) to nearly O(L).
    3. Reduce per-GPU sequence length: split the sequence dimension across multiple GPUs. Each GPU holds a chunk of keys/values and exchanges them through ring communication, enabling context lengths beyond a single GPU’s memory limit.

    In addition, variable-length sequence packing is often underestimated. Packing short samples into the same batch can significantly reduce wasted computation caused by padding. The throughput gain is often more noticeable than switching to a stronger attention implementation. Extrapolation ability of positional encodings and length curricula—training with gradually increasing sequence lengths—also need to be designed carefully. Otherwise, long-context support may only look good on evaluation sets.

    6. Inference: From Model Capability to Deliverable Service

    Training is only the starting point. The optimization goals for inference services are often two conflicting metrics:

    • Time to first token (TTFT): determines interactive experience and is mainly determined by the computation of the prefill stage;
    • Time per output token (TPOT): determines reading fluency. The decode stage is limited by memory bandwidth and generates one token at a time.

    Four high-value techniques:

    Continuous batching. Traditional static batching must wait for all requests in a batch to finish before admitting new ones, leaving GPUs idle. Continuous batching schedules at iteration granularity: finished requests exit immediately, and new requests can be inserted immediately.

    Paged cache management. Changing KV cache from “large contiguous preallocation” to “on-demand paging” can significantly reduce fragmentation and reserved waste. This can increase concurrency by an order of magnitude and naturally supports prefix sharing, so an identical system prompt only needs to be computed once.

    Separated deployment of prefill and decode. The two stages have opposite compute characteristics: prefill is compute-intensive, while decode is bandwidth-intensive. Running them on the same resource pool means one side will be underutilized. Separating them allows each stage to scale and configure precision independently according to its characteristics.

    Speculative decoding. A lightweight draft model or shallow layers of the model itself propose candidate tokens, and the main model verifies them in parallel in one step. If accepted, it advances multiple steps. This is effective when draft hit rate is high, but it is sensitive to distribution differences and requires tuning the draft strategy for specific tasks.

    A sketch of scheduling logic:

    1. def schedule_step(running, waiting, cache_pages, max_batch_tokens):
    2. """One scheduling step: advance running requests first, then admit new requests with remaining budget."""
    3. for req in running:
    4. if req.finished():
    5. release(req) # Release paged cache immediately
    6. else:
    7. step_decode(req)
    8. admitted = 0
    9. for req in waiting: # Ordered by wait time and priority
    10. need = req.prefill_token_estimate()
    11. if (admitted + 1 <= MAX_CONCURRENCY
    12. and need <= max_batch_tokens
    13. and cache_pages.alloc(req.kv_pages) is not None):
    14. step_prefill(req)
    15. admitted += 1
    16. else:
    17. break
    18. return running

    7. Observability and the Tuning Loop

    In large-scale systems, if it is not visible, it cannot be tuned. On the training side, the following metrics should be exposed and aligned across layers:

    • loss and gradient norms;
    • learning rate schedules;
    • per-step time broken down into computation, communication, and checkpoint writing;
    • memory utilization;
    • load balance across parallel groups.

    Only after breaking down time can teams decide whether to optimize recomputation or communication topology next.

    On the inference side, the focus should be on tail latency rather than averages:

    • P50/P95/P99 time to first token;
    • queue waiting time;
    • cache hit rate;
    • in-batch padding rate.

    Average latency may look good while P99 collapses. This is one of the most common mistakes in capacity planning.

    These metrics should be unified into logging and monitoring/alerting systems. With thresholds and anomaly detection rules, teams can move from reactive firefighting to proactive intervention.

    8. Cloud Engineering: Turning Idle Resources into Effective Compute

    On cloud platforms such as Baidu AI Cloud, a complete training and inference pipeline can be built by combining general capabilities:

    • container platforms for job orchestration and fault rescheduling;
    • high-speed interconnect networks for parallel communication;
    • object storage for datasets, model weights, and checkpoints;
    • image repositories for environment consistency;
    • message queues for decoupling offline evaluation and data preprocessing tasks;
    • monitoring, alerting, and logging services for global observability.

    Resource utilization is often the most overlooked cost item. Training workloads often have tidal characteristics. Interactive debugging and data preprocessing do not require full interconnect bandwidth and can be filled using low-priority queues or co-scheduling. Inference services also have clear time-based fluctuations. Combining elastic scaling and canary replicas can reduce always-on resources while preserving P99 latency.

    9. Evaluation and Launch: Avoiding Offline Gains and Online Failures

    Model iteration requires a closed-loop evaluation system. A practical structure has four layers:

    1. Capability benchmarks: cover general understanding, reasoning, code, and math. Fix versions and decoding parameters to ensure comparability across versions.
    2. Adversarial and long-tail cases: include boundary inputs, very long contexts, multi-turn references, and format-following requirements. These are the cases most likely to expose problems in real scenarios.
    3. Automated regression: run the full test suite automatically after every training artifact and compare against a baseline to prevent regressions.
    4. Online canary and human sampling: use small-traffic canaries plus human or model-assisted evaluation to observe behavior under real distributions.

    Evaluation results should feed back into training. Decisions such as data mix adjustments, curriculum ordering, and even trade-offs between parallelization strategy and throughput should be supported by data rather than intuition.

    10. Conclusion

    The engineering essence of large-scale models is to decompose a multi-objective optimization problem under three constraints—memory, communication, and stability—into a series of measurable, verifiable, and reusable decisions. Parallel strategies have clear trade-off principles. Memory has a calculable ledger. Loss spikes have established handling procedures. Inference has clear latency and throughput trade-off models.

    Making these parts into a standardized, observable, and automatically recoverable pipeline is the real meaning of pushing the limit forward. It is not about one training run being exceptionally large. It is about making this class of training stable, cost-efficient, and repeatable.