Why Large Model Training Is Hard: Parallelism, Fault Tolerance, and Convergence

    Large model training is hard not because of parameter counts or compute alone, but because it demands a working balance among distributed parallelism, fault tolerance, numerical stability, and convergence. This article breaks down the core challenges of large-scale parallel training—compute efficiency decay, communication overhead, MFU, operator-level optimization, hardware failures, checkpointing, elastic training, numerical instability, and hyperparameter sensitivity—and explains why engineering goals and algorithmic goals often conflict. It closes with practical guidance on cluster scheduling, observability, and the trade-offs teams must manage to turn expensive compute into usable model capability.

    Introduction: Why Large Model Training Is Hard

    When people talk about training large models, they often think first about parameter counts, the number of accelerators, cluster budgets, or even the idea that “throwing more compute at it” is enough to produce a model. But those who have actually run large-scale training tend to ask sharper questions: Why is large model training hard? Where exactly do the difficulties of large-scale parallel training lie? And how can fault tolerance and convergence be handled at the same time?

    On the surface, large model training looks like a process that consumes “compute multiplied by time.” Once you enter real engineering, however, it turns out to be more like a long-running, tightly coupled systems engineering effort that is continually disrupted by faults and numerical risk. Compute is just the entry ticket; what really determines whether training completes successfully is the combined balance among parallel efficiency, fault recovery, numerical stability, cluster scheduling, and algorithmic convergence.

    If training a small model is like debugging in a lab, then training a super-large model is more like keeping a fully loaded race car at sustained high speed for weeks or even months: it must not only run fast but also stay stable under full load, handle faults while running, and optimize efficiency on the fly. If any single detail slips out of control, the progress of the entire training run can be seriously affected.

    1. A Shift in Training Paradigm: From Retryable to Non-interruptible

    To understand the difficulty of large model training, you first have to understand the qualitative change brought about by training scale.

    Training medium-sized models early on often required just a few servers and a few dozen accelerators, running for a few days. At that time, engineering usually adopted a “retryable” strategy: if a task failed, you simply restarted it, and the loss was relatively limited. But once a training job scales to thousands of accelerators and runs continuously for weeks or even months, the rules change.

    This change shows up in three main ways:

    • Long job duration: A single complete training run may last a long time, and a mid-run failure brings significant wasted compute.
    • High failure probability: The more cards there are, the higher the chance that something goes wrong in hardware, networking, storage, and other components.
    • Strong system coupling: The model is split across many devices, so a failure in one node can affect the entire training job.

    So large model training is not simply “more machines, bigger model.” It is a shift from short-cycle trial and error to long-cycle continuous operation. This shift is the starting point for all the engineering challenges that follow.

    2. Core Challenge One: Compute Efficiency Decay at Large-Scale Parallelism

    2.1 Why Sharding Is Necessary

    A single accelerator has limited memory and compute. When the combined footprint of model parameters, activations, gradients, and optimizer states far exceeds the capacity of one card, the model must be split across multiple cards.

    Common parallel strategies include:

    • Data parallelism: Each card holds a full replica of the model, processes a different batch of data, and then synchronizes gradients.
    • Tensor parallelism: Matrix operations inside a single layer are split across cards by dimension, requiring frequent inter-card communication.
    • Pipeline parallelism: The model is split by layer into multiple stages, with different cards handling different stages or different batches.

    Real training is usually not a single strategy but a combination, forming “3D parallelism,” and it may further layer on strategies such as sequence parallelism and expert parallelism. The more complex the strategy, the harder the system tuning becomes.

    2.2 Communication Overhead: Why More Cards Can Actually Be Slower

    In theory, more cards means more compute, but in practice compute efficiency often grows non-linearly. The reason is that once the model is split, cards must continuously exchange parameters, gradients, or activations. As the cluster grows, communication pressure rises quickly.

    Take gradient synchronization under data parallelism as an example:

    1. for layer in model.layers:
    2. local_grad = compute_local_gradient(layer, local_batch)
    3. global_grad = all_reduce(local_grad, world_size=NUM_GPUS)
    4. layer.weights -= learning_rate * global_grad

    This pseudocode only expresses the core idea: after each card computes its local gradient, it must synchronize across all cards. When NUM_GPUS grows from dozens to thousands, the combined communication frequency and data volume can cause a large amount of time to be spent waiting on transfers rather than doing real matrix computation.

    Therefore, the key to large-scale parallel training is not only “how many cards” but also a whole set of engineering capabilities: inter-card interconnect bandwidth, network topology, communication overlap, and compute/communication scheduling.

    2.3 MFU: The Key Metric for Real Efficiency

    When evaluating training efficiency, MFU (Model FLOPs Utilization) is a commonly used metric. It reflects the ratio of the effective floating-point operations actually produced during training to the hardware’s theoretical peak.

    Many teams see very high accelerator utilization, but this does not necessarily mean high training efficiency, because monitored utilization can include waiting, communication, scheduling, and idle spinning. Once converted to MFU, the actual effective compute utilization may be only around 30%, depending on model structure, network topology, parallel strategy, operator efficiency, and data-loading efficiency; the measured MFU should be the basis.

    This means that if MFU is low, adding more cards does not necessarily deliver proportional training gains. Optimizing communication, reducing bubbles, improving operator efficiency, and enhancing the data pipeline are often more effective than simply scaling up.

    2.4 Operator-Level Optimization: Reducing Memory Reads and Writes

    Beyond parallel strategies, operator optimization is also a key factor in training efficiency. Take the attention mechanism as an example: a standard implementation may produce large intermediate matrices and read/write GPU memory frequently, increasing memory-access pressure.

    Some common attention optimization approaches use techniques such as block-wise computation, recomputation, and operator fusion to reduce the number of times intermediate results are explicitly written to GPU memory, thereby lowering memory-access overhead and memory footprint. These optimizations are critical in large model training. Without appropriate operator optimization, training cost may rise significantly, and the specific impact needs to be measured in practice against model structure, hardware type, input length, and other factors.

    In other words, operator optimization is not a nice-to-have; it is a key variable that determines training cost.

    3. Core Challenge Two: Hardware Stability and Fault Tolerance

    3.1 The Larger the Scale, the More Frequent the Failures

    In a training job spanning thousands of cards and running for months, hardware failures are not a rare event; they are closer to the norm. Accelerators, GPU memory, network links, switches, power supplies, and storage services—a problem in any one of these can affect training continuity.

    If a failure is not handled properly and the job has to restart from scratch, the loss can reach thousands to tens of thousands of card-hours, depending on cluster size, checkpoint interval, and failure type. For a large cluster, this loss affects not only cost but also the overall R&D timeline.

    Fault tolerance, then, is not optional; it is part of the infrastructure for large model training.

    3.2 How to Do Checkpointing for Large Model Training

    The basic method for handling failures is to save checkpoints. A checkpoint usually saves model weights, optimizer state, learning-rate scheduler state, the current training step, and other information. If training is interrupted, you can resume from the most recent checkpoint rather than starting from zero.

    The following is illustrative pseudocode that only shows the basic flow of saving and restoring checkpoints; it does not cover storage exceptions, write failures, or object-consistency handling. In real engineering, you should add atomic writes, verification, failure retries, and safety checks.

    1. if step % save_interval == 0:
    2. save_checkpoint(
    3. model=model,
    4. optimizer=optimizer,
    5. scheduler=scheduler,
    6. step=step,
    7. path="checkpoints/step_{}".format(step)
    8. )
    9. ckpt = load_latest_checkpoint("checkpoints/")
    10. model.load_state_dict(ckpt.model)
    11. optimizer.load_state_dict(ckpt.optimizer)
    12. start_step = ckpt.step

    The save interval should be neither as frequent as possible nor as sparse as possible; it should be set based on I/O throughput, cluster size, and fault-recovery goals. If you want to reduce the loss from failures, you can increase save frequency within the limits of storage throughput; if storage pressure is high, you can use asynchronous saving, sharded saving, snapshot verification, and other approaches to balance overhead and safety.

    In addition, checkpoint writes involve data-consistency concerns. In production, you typically need to verify whether a checkpoint is complete, readable, and corresponds to the correct training step, to avoid restoring a corrupted or stale state.

    3.3 Elastic Training and Self-Healing

    A more mature approach is to build elastic training capabilities that keep training going as much as possible when some nodes fail. Deployable mechanisms typically include:

    • Health checks: Continuously monitor the status of nodes, accelerators, networks, and storage.
    • Failure detection: Identify anomalies via logs, metrics, heartbeats, and hardware signals.
    • Node isolation: Remove faulty nodes from the current job to prevent the impact from spreading.
    • Hot-standby resources: Reserve available nodes or cards for quick replacement.
    • Task rescheduling: Reassign the training job to healthy resources.
    • Resume from checkpoint: Restore training state using the most recent checkpoint.
    • Topology reconstruction: Rebuild the communication topology in multi-card, multi-node training.
    • Alerts and auditing: Record failure causes, recovery times, and impact scope.

    Elastic training is not a single-point technology but a full set of operations and scheduling capabilities. It requires monitoring, storage, scheduling, and the training framework to work together.

    4. Core Challenge Three: Algorithmic Convergence and Numerical Instability

    4.1 Loss Spikes and Divergence

    Large-scale training frequently encounters loss spikes: the loss curve suddenly rises abnormally at some step, then may gradually recover or may diverge completely. Common triggers include abnormal activations, abnormal gradients, bad data, numerical-precision issues, and too-high learning rates.

    A disturbance that occurs occasionally in small-scale training can be continually amplified in large-scale, long-cycle training. Therefore, during training you need to monitor the loss, gradient norms, parameter norms, and activation distributions in a timely way to catch anomalies as early as possible.

    4.2 Numerical Precision and Mixed-Precision Training

    To save memory and improve throughput, training usually uses mixed precision, such as combining FP16, BF16, and FP32. But lower precision narrows the numerical dynamic range and can cause overflow or underflow in gradient accumulation, normalization, optimizer-state updates, and similar steps.

    Common mitigation measures include loss scaling, more stable normalization implementations, numerically robust maintenance of optimizer state, and keeping high precision on critical paths. Numerical stability is not a single switch; it is the combined result of the training framework, the optimizer, and the data distribution.

    4.3 Hyperparameters and Scheduling Sensitivity: A Common-Issue Checklist

    Large model training is extremely sensitive to hyperparameters. A seemingly tiny configuration problem can lead to an abnormal result that is only discovered after weeks of training. Common problems include:

    Common problem Possible symptoms Recommendation
    Learning rate too high Loss spikes, training diverges Lower the learning rate; check warmup and decay schedule
    Insufficient warmup Early instability, drastic gradient swings Increase warmup steps; smooth optimizer-state changes
    Missing gradient clipping Gradient explosion, sudden loss spikes Set a reasonable gradient-clipping threshold and monitor gradient norms
    Uncleaned anomalous data Abnormal loss, sample contamination Strengthen data filtering, deduplication, and quality checks
    Improper precision configuration Overflow, underflow, abnormal loss Check the mixed-precision strategy and loss-scaling configuration
    Batch too large Slower convergence or abnormal generalization Adjust batch size together with learning rate and optimizer state
    Improper weight initialization Difficulty converging early Use an initialization strategy suited to the model structure

    Hyperparameter tuning should not rely on random trial and error; it should be validated step by step by combining small-scale experiments, training-curve monitoring, and experience-based rules.

    5. The Real Difficulty: Balancing Stability and Convergence

    Large model training is complex because engineering goals and algorithmic goals are not always aligned. The engineering side pursues stability—fewer interruptions, fast recovery, higher throughput; the algorithmic side pursues convergence—numerical stability, smooth training, controllable loss. But the two often constrain each other.

    Trade-off Conflict Possible impact Recommendation
    Higher throughput May reduce communication waiting but increase parallelism complexity Harder fault recovery Prioritize validating whether communication optimizations and topology are sound
    Checkpoint frequency More frequent is safer but adds I/O overhead Slower training or more storage pressure Set the interval based on I/O throughput and recovery goals
    Numerical precision Higher precision is more stable but uses more memory Lower training throughput Keep high precision on critical paths, mixed precision elsewhere
    Rollback strategy Rolling back to an old checkpoint can resume training May lose some optimizer momentum state Keep key checkpoints and verify state consistency
    Elastic replacement Hot-standby resources can speed up recovery Higher resource cost Configure the hot-standby ratio based on business goals

    Real engineering capability is not about doing one thing well, but about reaching an acceptable balance among these conflicting goals within the same system.

    6. Cluster Scheduling and Observability: The Underrated Foundation

    Large model training also depends on cluster scheduling. How thousands of cards are orchestrated, how faulty nodes are isolated, how the task queue is allocated, and how logs and metrics are collected all affect training efficiency and stability. It is recommended to establish at least the following checklist:

    • Metrics monitoring: Collect metrics such as accelerator utilization, memory usage, temperature, power, network throughput, MFU, loss curves, and gradient norms.
    • Faulty-node isolation: Automatically mark, isolate, or replace abnormal nodes to avoid contaminating the training job.
    • Task queue management: Schedule training jobs by priority, resource requirements, and estimated duration.
    • Resource quota management: Set resource quotas for different teams or projects to avoid contention.
    • Centralized log collection: Gather training logs, system logs, and scheduling logs in a unified way to ease debugging.
    • Monitoring and alerts: Set alerts for abnormal loss, communication timeouts, node dropouts, storage latency, and more.
    • Storage performance guarantees: Ensure that checkpoint writes and data reads do not become training bottlenecks.
    • Security boundaries: Access control and backup verification mechanisms should be in place for checkpoints, data, and logs.

    Observability is not an auxiliary feature; it is the first line of defense in large model training. Many efficiency decays and early-stage failures can only be caught in time through metrics and logs.

    Conclusion: Large Model Training Is Systems Engineering

    Back to the original question: Why is large model training hard?

    It is hard not because any single-point technology is mysterious, but because it requires a team to simultaneously handle multiple domains—distributed parallelism, operator optimization, hardware fault tolerance, numerical convergence, hyperparameter tuning, and cluster scheduling—and to find a stable, operable balance under mutually conflicting constraints.

    For teams and individuals who want to enter this field, rather than focusing only on parameter counts and compute scale, it is better to settle down and understand the underlying mechanisms: master parallel strategies, understand MFU and communication overhead, build checkpoint and elastic-recovery capabilities, pay attention to numerical stability and hyperparameter sensitivity, and improve the monitoring and scheduling systems. Only in this way can expensive compute be truly turned into usable model capability in large-scale, long-cycle, tightly coupled training jobs.