LLM Optimizer Evolution and Selection: From Momentum to Orthogonalized Updates

    A practical guide to how large language model optimizers evolved and how to choose among them. Starting from the core problem optimizers solve in neural network training, it walks through gradient descent and SGD, Polyak's Heavy Ball method and momentum, adaptive methods like RMSProp and Adam, and orthogonalized updates such as Newton-Schulz for matrix parameters. It includes a comparison table, engineering considerations, an actionable selection checklist, and common mistakes to avoid.

    1. What Optimizers Actually Solve: How to Choose One for LLMs

    In neural network training, the optimizer plays the role of a navigator. It doesn’t sense the full terrain itself, but it decides which direction each parameter update takes and how large that step is. In recent discussions about the interface capabilities and applications of large models, the optimizer—the real protagonist of the training stage—is often underestimated by outside observers. To understand the technical breakthroughs of newer optimizers, you first have to return to the most basic question: what exactly does an optimizer solve?

    1.1 The Three Elements of the Classic Descent Metaphor

    Imagine you are blindfolded and placed in a rolling mountain range, with the goal of reaching the lowest point of the valley. At this point you have only two “sensing tools”:

    • Loss: tells you the current elevation you’re at;
    • Gradient (via backpropagation): tells you, through the local slope, which side looks more downhill.

    The optimizer is the “stepping strategy.” It usually answers two decisive questions.

    First, how big is this step—learning rate control. Too big a step may carry you across the valley floor and up the opposite slope, causing the loss to oscillate back and forth and training to fail to converge stably; too small a step converges slowly, and you may walk for ages without reaching a suitable region. A good optimizer is usually paired with a learning rate schedule that allows larger steps early in training to quickly enter an effective region, and gradually reduces the step size later to converge stably.

    Second, which way to step—alleviating saddle points, local minima, and training stagnation. Large model training is typically a non-convex, high-dimensional problem. There are many “small pits” (local minima) on the slopes, as well as many saddle points where the gradient is near zero but the curvature directions are mixed, along with plateau regions where the loss briefly stalls. If you rely only on naive gradient descent, the model may linger near a saddle point for a long time, or fall into a local solution and struggle to improve further. Note that an optimizer cannot guarantee escape from a local optimum or arrival at a global optimum; a more realistic goal is to reduce training risk, ease stagnation, and improve convergence efficiency. Randomness, learning rate annealing, batch size, gradient clipping, and parameter initialization all affect the model’s ability to escape difficult regions.

    This also explains why “how to choose an optimizer” is not an isolated question—it has to be considered together with batch size, learning rate, regularization, model architecture, and data quality.

    2. Gradient Descent and Stochastic Approximation: Separate the Historical Attribution

    The optimization idea behind standard gradient descent is usually traced back to early research on steepest descent by Cauchy and others. Its core is to update parameters along the locally steepest descent direction at the current point:

    1. θ_t = θ_{t-1} - lr * g_t

    where θ_t is the parameter, lr is the learning rate, and g_t is the current gradient.

    But in large model training, computing the exact gradient over the full dataset is extremely expensive. So actual training commonly uses mini-batch data to approximate the gradient; methods of this kind are usually called stochastic gradient descent, or SGD. The stochastic approximation idea generally should not simply be traced to Cauchy, but rather to the stochastic approximation work of Robbins and Monro in 1951 and the subsequent development of stochastic approximation and stochastic gradient methods. In other words:

    • Gradient descent: a deterministic optimization idea, traceable to early steepest descent research by Cauchy and others;
    • Stochastic gradient descent: introduces random sampling and stochastic approximation on top of gradient descent, and the usual historical attribution should point to Robbins-Monro and subsequent stochastic approximation work.

    The advantages of SGD are simple implementation, low memory usage, and gradient noise that sometimes has a regularizing effect; its drawbacks are sensitivity to the learning rate, and a tendency to oscillate in direction in canyon-shaped or ill-conditioned-curvature scenarios.

    3. Heavy Ball: Adding Mass and Inertia to Parameter Updates

    Standard gradient descent has a physically unintuitive implicit assumption: it behaves like a “massless particle” sliding on the loss surface. Such a particle can instantly start from rest to any speed, and can stop immediately upon hitting a change in slope—it lacks inertia.

    In a long, narrow “canyon” terrain, the problem with this massless particle is especially pronounced: the gradient direction bounces back and forth between the two walls, and the algorithm wastes most of its updates oscillating perpendicular to the valley floor, while actual progress toward the bottom is pitifully slow.

    The classic method to break out of this predicament is the Heavy Ball Method, proposed by the Soviet mathematician Boris Polyak around 1964. Its core is to add “momentum” to the update, giving parameter updates an inertial memory:

    1. # θ_t: current parameter, same shape as the model parameter
    2. # v_t: momentum/velocity, same shape as θ_t
    3. # g_t: current gradient, same shape as θ_t
    4. # lr: learning rate, scalar
    5. # β: momentum coefficient, commonly around 0.9, usually in the range 0 to 1
    6. v_t = β * v_{t-1} + g_t
    7. θ_t = θ_{t-1} - lr * v_t

    Here β controls how much of the historical gradient is retained:

    • When the gradient direction is consistent over several consecutive steps, momentum keeps accumulating, and the parameters accelerate like a heavy ball rolling downhill;
    • When the gradient direction oscillates back and forth, positive and negative gradients cancel each other out, momentum is weakened, thereby suppressing perpendicular oscillation;
    • If β is too large, inertia is too strong, which may intensify oscillation or even cause divergence;
    • If β is too small, the smoothing effect is insufficient and it’s hard to suppress canyon oscillation.

    The key breakthrough of the Heavy Ball Method is: using historical information across the time dimension to smooth noisy directions and strengthen effective directions. This idea laid the foundation for nearly all subsequent modern optimizers.

    4. From Momentum to Adaptive Optimizers: What’s the Difference Between Adam and SGD?

    After Heavy Ball, several technical routes evolved around “how to decide steps more intelligently.” They can be understood as a full toolkit for solving the gradient update problem.

    Method Core update intuition Advantages Drawbacks Typical tuning Applicable scenarios
    SGD Update along current gradient Simple, low memory, noise may regularize Sensitive to learning rate, oscillates in canyon terrain Learning rate, momentum, weight decay Stable baseline, resource-constrained, some vision tasks
    Momentum Accumulate gradient history into momentum Smooths oscillation, accelerates consistent directions Excessive inertia may diverge β commonly 0.9 Canyon-shaped loss surfaces
    NAG Predict a step by momentum first, then compute gradient More timely direction correction Slightly more complex implementation Learning rate, momentum Tasks with high convergence speed requirements
    RMSProp Scale learning rate with a moving average of the squared gradient Adaptive to changes in gradient scale May accumulate too large a variance Decay rate, learning rate, eps Non-stationary gradients, recurrent networks, etc.
    Adam First moment + second moment + bias correction Robust, easy to use, good default performance May generalize slightly worse than a tuned SGD, slightly higher memory betas, eps, weight decay Common default optimizer for large model training
    Orthogonalized updates Approximately orthogonalize the momentum matrix before updating Balanced directions, matrix-friendly High implementation and numerical-conditioning requirements Number of NS iterations, learning rate scaling Large-scale 2D weight matrix experiments

    Nesterov Accelerated Gradient (NAG): Ordinary momentum is more like “sense first, then step.” Nesterov instead first predicts a step using momentum, then computes the gradient at the predicted point—equivalent to glancing at where you’re about to land while descending, correcting direction ahead of time, which usually converges more robustly.

    RMSProp-style adaptive methods: The core idea is to maintain a moving average of the gradient, or of the squared gradient, for each parameter, and normalize the current gradient using its square root. Directions where the gradient has been large for a long time automatically reduce their step size, while directions where it has been small get an appropriately larger step, achieving “per-parameter speed control.”

    Adam (Adaptive Moment Estimation): Combines the first moment (momentum) with the second moment (adaptive scaling), and introduces bias correction, balancing inertial smoothing with per-parameter adaptive learning rates. Because it is simple to implement, robust, and relatively friendly in terms of hyperparameters, Adam has long been widely adopted as the default optimizer.

    The common thread of these methods is: moving from a single global learning rate toward per-parameter, per-direction adaptive adjustment. But to further exploit the structural properties of matrix parameters in large models, another paradigm-level update is needed—orthogonalized updates.

    5. Orthogonalized Updates: A New Optimization Paradigm for Matrix Parameters

    Traditional optimizers treat every element of a parameter matrix “equally,” doing element-wise scaling. But in large models, many parameters exist in matrix form, such as attention weight matrices and feed-forward network weight matrices. Treating the gradient matrix as a whole and handling updates at the geometric-direction level may bring better convergence behavior. New optimizers characterized by orthogonalized updates develop along this line of thinking.

    5.1 Core Intuition: Making Updates “Direction-Balanced”

    A gradient matrix can be understood through singular value decomposition as a set of directions and their strengths. In actual training, gradients are often dominated by a few “principal directions,” with a highly uneven distribution of singular values. If you update directly along the raw gradient, parameters become overly biased toward these principal directions, while other directions are “underfed,” wasting efficiency.

    Orthogonalized updates usually work like this: first apply a polynomial approximation similar to Newton-Schulz iteration to the momentum matrix, pushing the singular values of the update matrix as close to 1 as possible, yielding an approximately orthogonal update matrix. The effect can be illustrated as:

    1. # M is the momentum matrix
    2. # NS denotes Newton-Schulz iteration
    3. O = NewtonSchulz(M) # approximately orthogonalize M, pushing singular values close to 1
    4. θ = θ - lr * O # advance parameters with the orthogonalized update

    Intuitively, orthogonalization makes the update matrix “even in strength” across all directions, as if redistributing scattered force evenly, avoiding being led astray by a few strong directions.

    But the scope of applicability must be emphasized: orthogonalized updates mainly apply to 2D weight matrix scenarios; for 1D parameters like embedding vectors, biases, and normalization parameters, mature adaptive methods are usually still needed alongside them. Their actual performance is also affected by the number of Newton-Schulz iterations, the spectral normalization approach, matrix shape, numerical stability, data type, and learning rate scaling strategy. This class of methods does not guarantee strict orthogonality, nor does it guarantee it will necessarily beat Adam on all tasks. Whether it helps must be measured empirically for the specific model, task, and hardware conditions.

    5.2 Engineering Value

    From an engineering perspective, the value of this class of methods is mainly reflected in:

    • Convergence efficiency: In some matrix-intensive large model trainings, it may approach the target loss in fewer training steps, but the conclusion depends on model scale and data quality;
    • Compute friendliness: The core computation involves a lot of matrix multiplication, naturally aligning with the parallel characteristics of mainstream accelerator hardware;
    • Hybrid use: Use orthogonalized updates for matrix parameters and adaptive methods for the rest, forming a combined strategy.

    To make the conclusions more verifiable, it’s recommended to record metrics such as step time, throughput, loss curve, gradient norm, update norm, and memory usage. Only with a clear model scale, hardware configuration, number of training steps, and data volume can you judge whether orthogonalized updates are truly better than conventional Adam.

    6. Practical Selection Advice: An Actionable Checklist

    After understanding this evolutionary thread, selection can follow this actionable checklist.

    1. Baseline first: Start with a mature Adam or AdamW. Common default recommendations are:

      • β1 = 0.9
      • β2 = 0.99; some implementations commonly use 0.999, adjust based on implementation and stability;
      • eps = 1e-8; needs adjustment alongside the implementation in low-precision or numerically unstable scenarios;
      • When using decoupled weight decay, be clear about the difference between weight decay and L2 regularization, and avoid double regularization.
    2. Learning rate scheduling must be paired with the optimizer. The optimizer decides “how to step,” and the learning rate schedule decides “how big a step.” Common strategies include warmup + cosine decay or warmup + linear decay. Without warmup, large model training tends to oscillate early on due to unstable gradients.

    3. Gradient clipping is used to stabilize training. Clipping the gradient norm, for example by global norm, can prevent training divergence caused by anomalous batches, numerical overflow, or a too-large learning rate.

    4. Validate matrix methods on a small scale first. If considering orthogonalized updates, prioritize evaluating them on 2D weight matrices. You need to compare:

      • Whether memory usage is acceptable;
      • Whether step time decreases;
      • Whether throughput improves;
      • Whether the loss curve is more stable;
      • Whether the gradient norm is abnormal;
      • Whether better results are obtained under the same compute budget.
    5. Don’t just look at loss; also look at gradient and update scale. A common mistake is observing only the loss curve while ignoring gradient norm, update norm, and the magnitude of parameter changes. Gradient explosions, overly large updates, and effective step-size drift may all show up as lagging signals in the loss.

    6. Monitoring, checkpoints, and reproducibility should be implemented neutrally. You can rely on standard engineering systems for training management, but should not depend on generic templated descriptions. Checkpoint storage needs attention to:

      • The least-privilege principle and access control;
      • Regular backups and recovery drills;
      • Availability and lifecycle management of storage media;
      • Log desensitization to avoid recording sensitive data;
      • Traceability of training configurations, random seeds, data versions, and code versions.

    Common mistakes checklist:

    • Learning rate too large, causing the loss to oscillate or diverge;
    • Momentum coefficient too high, making updates too inertial;
    • Missing warmup, causing instability early in training;
    • Looking only at loss, not at the gradient norm or update norm;
    • Matrix methods ignoring dtype, memory, and numerical stability;
    • Simply equating Adam’s weight decay with L2 regularization;
    • Batch size too large, leading to too little gradient noise and degrading generalization;
    • Forcing matrix orthogonalization methods onto 1D parameters like embedding layers and biases.

    7. Conclusion

    From Cauchy’s steepest descent idea, to stochastic gradient descent supported by stochastic approximation, then to Polyak’s Heavy Ball method and momentum, followed by adaptive scaling methods like Adam, and finally to the orthogonalized update paradigm for 2D weight matrices—the evolution of optimizers has always revolved around one theme: how to find better parameter solutions more intelligently and efficiently under limited perceptual information.

    Understanding this thread not only helps readers grasp the technical breakthroughs behind large model training, but also helps them more rationally make optimizer selection, tuning, and engineering decisions in real projects.