Optimizer Evolution in Large-Model Training: From Momentum Methods to Orthogonalized Updates

    This article provides a comprehensive technical review of optimizer design in deep learning training, tracing the evolution from classical gradient descent to modern adaptive methods and emerging orthogonalized update strategies for large-scale models. It explains how optimizers address three core training challenges—step-size control, direction correction, and escape from local minima—and offers practical guidance for optimizer selection and hyperparameter tuning in real-world engineering scenarios. The discussion covers classic SGD limitations, the Heavy Ball method's momentum mechanism, adaptive learning-rate algorithms like Adam, and the rationale behind matrix-level orthogonalization for billion-parameter models.

    The Optimizer: An Invisible Engine That Determines Training Efficiency

    In the full training lifecycle of a deep learning model, the factors that determine final performance extend far beyond network architecture and data scale. The parameter update strategy at each step is equally critical. If we picture the loss function as a rugged mountain range and model parameters as a hiker in the valley, the optimizer is the navigational strategy guiding that hiker. Each time gradient backpropagation signals the local downhill direction, the optimizer must answer three questions: How large should the next step be? In which direction should we move? And how should we adjust our stride based on the terrain?

    These three questions correspond to three core challenges in deep learning training:

    Step-size control. If the learning rate is too high, the model oscillates around the optimum or even diverges. If it is too low, convergence slows dramatically and computational costs multiply. Adaptive step-size adjustment is therefore the first priority in optimizer design.

    Direction correction. In narrow canyon-like terrains, gradient directions can shift frequently. If the optimizer fully trusts the current gradient, the model will bounce back and forth between canyon walls, wasting countless iterations. A well-designed optimizer needs to “remember” prior motion trends and use historical information to smooth the current direction.

    Escaping local minima. Real loss surfaces are far from ideal paraboloids—they contain numerous local minima. Without the ability to overcome these obstacles, the model can become stuck in a basin halfway up the mountain, mistakenly believing it has reached the valley floor.

    From classic first-order methods to today’s adaptive approaches, optimizer development has consistently revolved around these challenges. A recent training strategy disclosed in a large-model API documentation release has once again brought fine-grained optimizer design into the spotlight. This article traces the technical evolution of optimizers, offering an in-depth analysis of their design logic and engineering performance.

    Classic Gradient Descent: The Plight of a “Massless” Slider

    The most basic form of the optimizer traces back to classical ideas from the nineteenth century. In 1847, French mathematician Augustin-Louis Cauchy proposed the method of steepest descent based on gradient information: since the gradient points in the direction of fastest local increase, the negative gradient naturally points in the direction of fastest decrease. Iteratively updating along the negative gradient allows us to progressively approach a minimum. This idea laid the first cornerstone of modern optimization theory.

    However, this simple strategy has significant drawbacks in real training scenarios. The standard gradient descent update rule can be written as:

    θ_{t+1} = θ_t - η · ∇L(θ_t)

    It simulates a “massless particle” sliding across the loss surface—the particle can reach maximum velocity instantly and come to a halt just as quickly. This model is physically unrealistic: objects in real motion possess mass and inertia, and velocity changes continuously. Past motion direction and speed influence current decisions.

    This limitation becomes glaringly apparent in narrow “canyon” terrains. When the loss surface has steep gradient changes in one direction but gentle variation in another, standard gradient descent causes the particle to repeatedly collide with the canyon walls in the steep direction while crawling slowly along the gentle direction. The result is severe oscillation in the training curve, extremely low iteration efficiency, and massive waste of computational resources. This is the fundamental reason why classic gradient descent performs poorly on real high-dimensional optimization problems.

    The Heavy Ball Method: Adding Inertia to the Slider

    In 1964, Soviet mathematician Boris Polyak proposed a physically intuitive improvement while researching optimization theory: if we forcibly add “mass” and “inertia” to the sliding particle, would the trajectory change? The answer is yes. When the particle rushes down a slope, even upon reaching a local minimum, its inertia carries it forward, providing an opportunity to cross the basin and continue searching for the true valley floor.

    This is the famous Heavy Ball Method, with the following update rule:

    vt = μ · v{t-1} + ∇L(θt)
    θ
    {t+1} = θ_t - η · v_t

    Here, v_t is the accumulated velocity vector, and μ is the momentum coefficient (typically around 0.9). This mechanism of “accumulating momentum” delivers two key benefits:

    Accelerated traversal of flat regions. In regions where the gradient direction remains consistent, momentum accumulates, gradually increasing the effective step size and allowing the model to cross flat areas more quickly, shortening the overall training cycle.

    Smoothing of high-frequency oscillation. In regions where the gradient direction frequently alternates, the momentum term offsets the current gradient, dampening high-frequency fluctuations and making the update trajectory more stable, thereby improving training stability.

    The Heavy Ball method stands as a milestone in optimizer development, establishing the “inertia + gradient” paradigm that defines modern optimizers. Subsequent improvements have largely revolved around this idea—for example, computing gradients ahead of time in the direction of momentum prediction, using “look-ahead” to correct the update direction, and achieving a better balance between convergence speed and oscillation suppression.

    Adaptive Learning Rates: Intelligent Step Sizes for High-Dimensional Surfaces

    As optimizers moved from low-dimensional problems to high-dimensional neural networks, a new challenge emerged. Gradient magnitudes and variation frequencies can differ dramatically across parameters: some weights update rapidly, while others remain in a state of small-amplitude fluctuation for long periods. If all parameters share a single global learning rate, it becomes impossible to accommodate all dimensions simultaneously.

    This is the context in which adaptive learning-rate algorithms emerged. The core idea is to track the gradient history of each parameter and dynamically allocate a distinct learning rate for each dimension. The general formulation can be summarized as:

    mt = β₁ · m{t-1} + (1 - β₁) · ∇L(θt) # First-order momentum: gradient direction trend
    v_t = β₂ · v
    {t-1} + (1 - β₂) · (∇L(θt))² # Second-order momentum: gradient fluctuation magnitude
    θ
    {t+1} = θ_t - η · m_t / (√v_t + ε) # Parameter update

    Here, the first-order momentum captures gradient direction and trend, while the second-order momentum captures gradient variance. This design enables the optimizer to:

    • Take larger steps in dimensions where gradients are relatively stable;
    • Automatically reduce step size in dimensions with dramatic gradient fluctuations, preventing oscillation;
    • Provide more update opportunities to sparse gradient dimensions, avoiding long periods of no learning.

    The combination of first-order and second-order momentum gives the optimizer a powerful “intelligent” character on highly rugged terrains, making it a de facto standard for training large models. However, these methods are not without flaws: the initial learning rate still requires careful manual tuning; the long-term accumulation of second-order momentum can cause step sizes to shrink in later training stages, reducing convergence accuracy; and in ultra-large model training, the additional memory overhead and communication costs of storing both momentum terms are significant concerns.

    These shortcomings have driven researchers to continue exploring a new generation of optimizer designs.

    New Optimizer Directions for Ultra-Large-Scale Models

    In ultra-large-scale model training, optimizers must not only be effective but also efficient. A recent large-model API documentation release disclosed a technical direction worth close attention—a novel optimizer design based on momentum accumulation combined with matrix orthogonalization.

    Core idea. Traditional optimizers perform element-wise adaptive scaling on each parameter, ignoring the spectral structure within weight matrices. The new optimizer approaches the problem at the matrix level, imposing orthogonality constraints on weight update matrices during parameter updates. This makes the update direction more uniform in spectral space, preventing certain feature directions from being over-updated while others remain nearly stagnant.

    Mathematically, this design can be understood as a low-cost approximation of second-order information. Classic second-order optimization methods (such as Newton’s method) require computing and inverting the full Hessian matrix, which is computationally prohibitive. The orthogonalization approach attempts to capture similar direction-correction capability with modest additional computation.

    Technical advantages manifest in three main areas:

    • Higher-quality update directions: Orthogonalization keeps update step sizes balanced across different singular directions of the weight matrix, which is particularly beneficial for deep network layers and helps mitigate gradient vanishing or explosion issues in deep architectures.
    • Natural compatibility with momentum: The new optimizer retains the momentum accumulation framework, merging historical velocity information with orthogonalized update directions—combining the benefits of inertia with improved directional accuracy.
    • Greater training-efficiency potential: By avoiding the extra computation of element-wise adaptive scaling and reducing the interference of gradient variance in update directions, this approach may deliver improved throughput and numerical stability in large-scale parallel training environments.

    It is important to note that orthogonalized updates are not a concept that emerged from nowhere—they build on the long-standing recognition that matrix spectral structure plays a crucial role in deep network training. When model parameter counts reach the hundreds of billions, even minor algorithmic improvements in the optimizer yield substantial returns in computational cost and training duration. This explains why optimizer design has become a core highlight in recent large-model technology releases.

    Engineering Practice: General Strategies for Optimizer Selection and Tuning

    For developers, the ultimate purpose of understanding optimizer principles is to guide practical work. The following are several general strategies that have proven effective in engineering practice.

    Selection: Evaluate based on task scale. For small-to-medium models (parameter counts below the tens of millions), adaptive learning-rate methods remain a reliable choice that balances stability and effectiveness. Their built-in gradient-variance normalization significantly reduces the difficulty of learning-rate tuning. For large-scale model training, it is worth testing new optimizers with momentum accumulation and orthogonality constraints, comparing convergence curves and training throughput to assess their practical benefits.

    Tuning order: Control variables, coarse to fine. A general hyperparameter search strategy: first fix all other conditions, use a short segment of the training curve to estimate loss descent speed, and roughly determine the order of magnitude of the initial learning rate. Then adjust the momentum coefficient or second-order decay coefficient to observe mid-to-late training stability. Finally, combine with regularization terms to determine whether the model is overfitting or underfitting.

    Stability safeguards: Gradient clipping and learning-rate schedules. When training shows signs of divergence or loss oscillation, gradient clipping is the most direct and effective remedy. It is recommended to adopt a “warmup + decay” learning-rate schedule in the early stages of training: linearly increase the learning rate from a very small value to the target over the first few thousand steps, then gradually decay following a cosine or polynomial curve. This strategy has been widely validated to significantly improve training stability in the early phases of large-model training.

    Diagnostic tips: Let the training curve tell you the problem. If the loss oscillates persistently while descending slowly, this typically indicates a learning rate that is too high or insufficient momentum. If the loss drops sharply and then plateaus for an extended period, the model may be stuck in a local optimum—consider introducing noise perturbation, lowering the learning rate, or adjusting batch size to attempt an escape. If validation loss rises while training loss remains stable, this suggests overfitting—strengthen regularization or implement early stopping.

    Conclusion

    The optimizer is a seemingly simple yet profoundly influential component of deep learning training. From Cauchy’s classical ideas to Polyak’s Heavy Ball method, from adaptive learning rates to orthogonalized update strategies for ultra-large-scale models, each evolution embodies deep research into the dynamics of training. For developers, mastering the core design logic of optimizers is not just essential for tuning capability—it is a key step toward understanding the full picture of deep learning training. As model scales continue to grow, optimizer technology will remain an important engine for improving training efficiency and model quality.