Optimizer Evolution in LLM Training: From SGD to Adaptive and Orthogonalized Methods

    An overview of how training optimizers have evolved from plain stochastic gradient descent to momentum, to adaptive methods like Adam and AdamW, and finally to matrix-aware orthogonalized optimizers. The article explains what an optimizer actually controls, the motivation behind each generation of methods, and the engineering practices—learning rate scheduling, gradient clipping, weight decay, resource tradeoffs, and monitoring—that matter when training large models. Readers will understand why each method emerged, how the newer orthogonalized approaches relate to AdamW, and how to choose a reliable baseline for real training workloads.

    Introduction: The Optimizer as Deep Learning’s Navigation System

    In neural network training, what really determines whether a model converges efficiently is often not the network architecture itself, but the quietly running optimizer. Picture it as a guide leading you across a mountain range: the parameters are where you stand, the loss tells you your current altitude, and the gradient is like a probe under your feet, hinting at which side slopes downhill. The optimizer is the policy in your head that decides “which way to step, and how far.”

    1. What an Optimizer Actually Controls

    An optimizer typically answers two key questions:

    • Step size (learning rate control): A step that is too large jumps straight over the valley floor and oscillates back and forth, causing training to diverge; a step that is too small may never reach the bottom, making training painfully slow. A good optimizer speeds up in flat regions and slows down where the terrain is steep and close to the minimum.
    • Escaping dead ends (avoiding local optima and saddle points): High-dimensional loss surfaces are dotted with shallow pits along the slopes. If you descend only along the steepest direction, you can easily get trapped in a local minimum or a saddle point, mistaking it for the true valley floor.

    Understanding these two points is the key to reading the motivation behind every step of optimizer evolution.

    2. First Generation: Naive Gradient Descent and Its Limits

    The idea of gradient descent (SGD) traces back to the work of the French mathematician Cauchy in 1847. It is the simplest “slide along the steepest direction” strategy, with an update rule that could hardly be more plain:

    1. g_t = ∇L(θ_t) # current gradient
    2. θ_t+1 = θ_t - η · g_t # take one step along the negative gradient

    Here η is the learning rate. The problem is that standard gradient descent is equivalent to simulating a massless particle: it can accelerate to top speed instantly and brake to a stop instantly, which defies physical intuition. In a long, narrow “valley” terrain, the particle oscillates back and forth along the steep direction while barely moving along the gentle valley floor, so convergence is very inefficient.

    3. Second Generation: Momentum—Giving the Particle “Inertia”

    The one who broke this setup was the Soviet mathematician Boris Polyak. In 1964, in his work on optimization theory, he asked what would happen if you forced this sliding particle to carry “mass” and “inertia.” That gave rise to the famous Heavy Ball Method, which is today’s momentum.

    Its core idea is to accumulate past gradients to build up inertia:

    1. v_t = β · v_{t-1} + g_t # inertia: retain part of the previous velocity
    2. θ_t+1 = θ_t - η · v_t # move along the direction that carries inertia

    β is the momentum coefficient (commonly 0.9). The physical intuition is direct: the lateral components that oscillate back and forth across the valley cancel each other out, while the consistent component along the valley floor keeps accumulating, so the particle rolls straight toward the bottom like a heavy ball. Building on this, Nesterov Accelerated Gradient (NAG) adds “lookahead”—it first projects one step ahead using inertia, then computes the gradient at that projected position to obtain a more precise correction direction.

    4. Third Generation: Adaptive Learning Rates and Adam

    A fixed learning rate has another weakness: different parameters have wildly different “temperaments” during training. Sparse features, embeddings, and the backbone network have vastly different needs when it comes to step size. Adaptive methods were born precisely to answer “how fast should each parameter move.”

    • AdaGrad: Accumulates the sum of squared historical gradients for each parameter, so parameters that are updated frequently automatically take smaller steps. Its drawback is that the learning rate decreases monotonically and can stall prematurely in the later stage of training.
    • RMSProp: Replaces unbounded accumulation with an exponential moving average, easing AdaGrad’s learning-rate decay problem.
    • Adam: Fuses momentum (first moment) with adaptivity (second moment) and adds bias correction, making it the de facto default choice in large model training.
    1. m_t = β1 · m_{t-1} + (1-β1) · g_t # first moment: moving average of the gradient
    2. v_t = β2 · v_{t-1} + (1-β2) · g_t² # second moment: moving average of the squared gradient
    3. m̂ = m_t / (1 - β1^t) # bias correction
    4. v̂ = v_t / (1 - β2^t)
    5. θ_t+1 = θ_t - η · m̂ / (√v̂ + ε) # per-parameter adaptive update

    Typical hyperparameters are β1=0.9, β2=0.999, η=1e-3, and ε=1e-8. Going further, AdamW decouples weight decay from the gradient and applies it directly to the parameters, making the regularization behavior clearer; it has become a routine configuration when training large models.

    5. Fourth Generation: Orthogonalized Optimizers for Matrix Structure

    All of the earlier methods essentially “flatten” the weights into a one-dimensional vector and update them element by element. But the hidden-layer weights of a neural network are naturally two-dimensional matrices that carry singular-value structure. The new generation of optimizers—represented by orthogonalized momentum methods—is built precisely around this fact.

    The core idea is: instead of simply scaling the momentum element by element, perform orthogonalization on the momentum matrix—approximately computing the orthogonal factor in its polar decomposition (equivalent to a “matrix sign” operation on the gradient matrix). In pseudocode:

    1. B = β · B + ∇L(W) # 1. accumulate the momentum matrix
    2. O = Orthogonalize(B) # 2. orthogonalize (usually approximated with Newton-Schulz iterations)
    3. W = W - η · O # 3. update the weights with the orthogonalized direction

    The Orthogonalize step is usually carried out with Newton-Schulz iterations, approximating the result in just a few steps without the cost of a full SVD. The theoretical motivation is that this approximates steepest descent under the spectral norm: it balances the update magnitude across different singular directions and improves the conditioning of the update, yielding more robust convergence on matrix-shaped parameters. In practice, methods of this kind are often hybridized with AdamW—hidden-layer matrices take the orthogonalization path, while non-matrix parameters such as embeddings, biases, and output heads are still handled by AdamW.

    6. Engineering Practices in Large Model Training

    No matter how good an optimizer is, it cannot do without supporting engineering practices:

    • Learning rate scheduling: “Warmup + cosine decay” is widely used. Warmup at the start prevents large gradients from overwhelming the parameters, and decay in the later stage helps stabilize convergence.
    • Gradient clipping: Clip gradients by their global norm to prevent an extreme batch from driving training into divergence.
    • Weight decay: Suppresses overfitting; note that the decoupled (AdamW-style) approach is more controllable.
    • Resource tradeoffs: Momentum-based methods need to store extra first- and second-moment state, which carries a significant memory cost; orthogonalization methods introduce iterative matrix operations that increase both compute and communication cost, so they must be evaluated against your cluster conditions.
    • Monitoring and observability: Combine logging and alerting to continuously track the loss, gradient norms, and learning rate curves so anomalies can be caught early.

    Conclusion

    The history of optimizer evolution is a history of “continually giving the particle more physical intuition”: from disorderly sliding, to inertia (momentum), to adaptivity that adjusts to local conditions (Adam), and finally to respecting matrix structure (orthogonalization). Understanding the motivation behind each step matters far more than memorizing the formulas. For real training tasks, use AdamW as a reliable baseline, then try new methods carefully in light of your parameter structure and hardware conditions—there is no universally optimal optimizer, only the strategy best suited to the problem at hand.