A Practical Guide to Optimizer Evolution: From Gradient Descent to Muon

    This article walks through how deep learning optimizers evolved from plain gradient descent to modern adaptive methods and the newer Muon optimizer. It explains the three core problems any optimizer must solve—learning rate control, escaping local minima, and handling ill-conditioned curvature—then traces the progression from SGD to momentum (Heavy Ball), Nesterov acceleration, AdaGrad, RMSProp, Adam, and AdamW. It also breaks down Muon's central idea of orthogonalizing momentum updates via Newton-Schulz iteration, along with its practical constraints, and closes with concrete guidance on optimizer selection and tuning for real-world training.

    1. Optimizers: The “Navigation System” of Deep Learning

    In deep learning training, updating parameters is fundamentally a high-dimensional, non-convex numerical optimization problem. We are navigating a high-dimensional loss surface made up of parameters, and the goal is to find the combination of values that minimizes the loss across an enormous number of dimensions. Picture the process as “descending a mountain blindfolded”—you are placed into a rugged mountain range with your eyes covered, and your goal is to reach the deepest point in the valley. The only information you have is:

    • Loss (the loss function): tells you the altitude you are currently at;
    • Gradient (via backpropagation): like probing the slope of the ground with your feet, it tells you which direction leads downhill.

    The optimizer is the strategy in your head that decides “which way to go, and how big a step to take.” It provides no new information, yet it determines how efficiently you can use the information you already have to reach the valley floor. For this reason, optimizer design directly determines the convergence speed, stability, and final accuracy of training. In recent years, cutting-edge large model training has increasingly adopted non-conventional optimizer approaches, bringing this low-level component back into the technical spotlight.

    2. The Three Things an Optimizer Must Solve

    Judging whether an optimizer is good ultimately comes down to how it handles the following three classes of problems.

    2.1 Learning Rate Control—Determining Step Size

    This corresponds to learning rate adjustment in deep learning. If your steps are too large, you may leap straight over the valley floor and onto the opposite slope, causing the loss to oscillate and never converge. If your steps are too small, you may take forever to reach the bottom, making training extremely slow. A core capability of a good optimizer is the ability to adjust step size adaptively: speeding up in flat regions and slowing down in steep regions near the valley floor.

    2.2 Escaping Local Minima—Avoiding Dead Ends

    The mountain range is full of small pits partway up the slopes, corresponding to local minima on the loss surface. If you descend greedily using only the gradient direction, the model can easily fall into one of these pits and get stuck, mistaking a mid-slope depression for the true valley floor. An optimizer needs to use “inertia” or “historical information” to help the parameters push through these small pits and find flatter, better-generalizing solutions.

    2.3 Handling Ill-Conditioned Curvature—Dealing with Ravine Terrain

    When different directions on the loss surface have vastly different degrees of curvature (a “narrow, elongated ravine” with a large condition number), naive gradient descent oscillates back and forth along the steep directions while barely progressing along the flat ones. This requires the optimizer to treat different parameter dimensions differently, rather than applying the same step size to every direction.

    3. The Evolution: From SGD to Adaptive Methods

    3.1 SGD: Where Everything Began

    The idea of stochastic gradient descent (SGD) can be traced back to the 19th-century mathematician Cauchy and his work on optimization methods. Its update is extremely simple: take one step in the negative gradient direction from the current point.

    1. θ_{t+1} = θ_t - η · g_t # η is the learning rate, g_t is the current gradient

    The problem is that standard gradient descent simulates a “massless particle” sliding over a surface: it can instantly accelerate to top speed and instantly brake to a stop. This defies physical intuition and makes it inefficient in ravine terrain.

    3.2 Heavy Ball and Momentum: Introducing “Inertia”

    The one who broke this assumption was the Soviet mathematician Boris Polyak. In 1964, while studying optimization theory, he proposed the Heavy Ball Method: forcibly giving the sliding particle “mass” and “inertia.” The core idea is to accumulate the gradients from the past several steps into a “velocity,” so that updates carry historical direction.

    1. v_t = β · v_{t-1} + g_t # accumulate historical gradients, β is the momentum coefficient
    2. θ_{t+1} = θ_t - η · v_t

    With momentum introduced, the particle behaves like a heavy ball in a ravine—carrying its inertia through the lateral oscillations while retaining the ability to accelerate in the longitudinal direction. This is highly intuitive from a physics standpoint and significantly improves convergence under ill-conditioned curvature. A later development, Nesterov Accelerated Gradient (NAG), uses a look-ahead trick of “taking one step along the momentum before computing the gradient” to further reduce oscillations.

    3.3 Adaptive Methods: Customizing Step Size per Dimension

    The Heavy Ball Method solves first-order inertia, but all parameters still share the same global learning rate. The breakthrough of adaptive methods is maintaining an independent step-size scale for each parameter dimension:

    • AdaGrad accumulates the squares of historical gradients, so dimensions updated more frequently get smaller steps;
    • RMSProp switches to an exponential moving average, avoiding the problem in AdaGrad where the step size decays monotonically to zero;
    • Adam combines momentum (a first-moment estimate) with adaptive step size (a second-moment estimate) and applies bias correction to both, making it the most mainstream choice for a long time:
    1. m_t = β1·m_{t-1} + (1-β1)·g_t # first moment (direction)
    2. v_t = β2·v_{t-1} + (1-β2)·g_t² # second moment (scale)
    3. m̂ = m_t / (1-β1^t), v̂ = v_t / (1-β2^t) # bias correction
    4. θ_{t+1} = θ_t - η · m̂ / (√v̂ + ε)

    AdamW further decouples weight decay from the gradient, avoiding the bias caused by coupling L2 regularization with adaptive step size, and it is now the de facto standard for training Transformer-based models.

    4. A New Paradigm: The Muon Matrix-Orthogonalization Optimizer

    Although Adam is already powerful enough, a new class of optimizers has emerged in recent years that approaches the problem from a completely different angle, with Muon being its representative. Its central insight is: a weight matrix is a whole and should be updated as a matrix structure, rather than treating each element as an isolated scalar handled separately.

    4.1 Core Idea: Making Updates “Balanced Across Directions”

    Like momentum methods, Muon first maintains a momentum matrix M for the parameter matrix. The key difference lies in the processing before the update: it wants all the singular directions of this momentum matrix to be used in a “balanced” way—that is, to approximately orthogonalize M into UVᵀ (the result of replacing all singular values in the singular value decomposition of M with 1).

    Intuitively, this means the update advances through every singular direction of the parameter matrix with roughly the same force, preventing certain dominant directions from “dominating everything,” and thus more fully leveraging the entire expressive capacity of the matrix. This echoes the earlier point about “handling ill-conditioned curvature”—except that here it operates on the spectral structure of the matrix.

    4.2 Efficient Orthogonalization: Newton-Schulz Iteration

    Directly performing an SVD on a matrix is expensive. Muon’s cleverness lies in using Newton-Schulz iteration to approximate the orthogonalization at low cost, requiring only a few matrix multiplications to approach UVᵀ:

    1. # Pseudocode: approximately orthogonalizing the momentum matrix (illustration of Newton-Schulz iteration)
    2. def orthogonalize(G, steps=5):
    3. X = G / (G.norm() + 1e-7) # normalize first to keep the iteration numerically stable
    4. # Apply a polynomial iteration with carefully chosen coefficients so that singular values gradually approach 1
    5. a, b, c = 3.4445, -4.7750, 2.0315
    6. for _ in range(steps):
    7. A = X @ X.T
    8. B = b * A + c * (A @ A)
    9. X = a * X + B @ X
    10. return X

    4.3 Applicability and Engineering Constraints

    A few engineering realities must be made clear to avoid misuse:

    • It mainly targets the 2D weight matrices of hidden layers. For embeddings, output heads, biases, normalization parameters, and other 1D or non-matrix structures, it usually still falls back to Adam/AdamW, forming a “hybrid optimizer” strategy.
    • Higher per-step computational cost. Compared to purely element-wise Adam, Muon adds several matrix multiplications per step, requiring a trade-off between throughput and convergence quality.
    • Its sensitivity to learning rate and momentum coefficient differs from traditional methods, so it needs separate tuning and Adam’s hyperparameter experience cannot be applied directly.

    5. Practical Advice: How to Choose and Tune

    1. Start with AdamW by default. It is a well-validated, robust baseline that produces reliable results in the vast majority of scenarios. Always get the baseline working before considering a replacement.

    2. Choose optimizers in layers based on parameter structure. For large-scale hidden-layer weights, you can try matrix-based optimizers; for embeddings, outputs, normalization, and similar parameters, keep the Adam family. This is currently the more mainstream hybrid approach.

    3. Pay attention to coordination between learning rate and scheduling strategies. No matter which optimizer you use, it is hard to work well in isolation from warmup and decay schedules such as cosine or linear decay; the two must be designed together.

    4. Let metrics drive decisions, not novelty. Replacing an optimizer should be based on comprehensive metrics such as validation loss, convergence-curve stability, and throughput cost, rather than introducing unnecessary complexity in pursuit of novelty.

    5. Ensure numerical stability. When matrix operations and second-moment estimates are involved, always set a reasonable epsilon and normalization, and combine them with gradient clipping to prevent numerical blow-up in early training.

    6. Summary

    The history of optimizer evolution is essentially a history of “how to use gradient information more intelligently”: from SGD’s plain descent, to Heavy Ball’s introduction of inertia, to Adam’s adaptive step size, to the new matrix-orthogonalization paradigm represented by Muon, each step attempts to more fully capture the geometric structure of the loss surface. Understanding this main thread is more important than memorizing any specific algorithm—because only by understanding “what problem an optimizer solves” can you make the right selection and tuning decisions in real engineering.