Training Optimizers for Large Language Models: The Evolution from Momentum to Muon
This article traces how deep learning optimizers evolved, from the inertia-free assumptions of gradient descent to Polyak's heavy ball method, momentum and Nesterov acceleration, adaptive per-parameter learning rates, and finally Muon's approximate orthogonalization of the momentum matrix. It explains the three core problems an optimizer must solve, how each generation of methods addresses them, and how to choose and tune optimizers in practice for Transformer-scale training.
1. Optimizers: The Navigation System of Deep Learning
Training a model is a long journey. The parameter space is a landscape of rolling hills, and the goal is to move the model from a randomly initialized point on some high ground down into the low valleys of the loss function. The component responsible for navigation is the optimizer.
Think of the process as descending a mountain blindfolded. Only two kinds of information are available: the loss function, which tells you the altitude of the terrain under your feet, and the gradient (computed by backpropagation), which works like feeling the slope with your foot, pointing out which direction is steeper and more downhill. Everything else — which direction to step in, how large the step should be, and whether to borrow momentum from previous steps — is decided by the optimizer.
In other words, the loss and the gradient only supply terrain information; the policy behind the optimizer determines how you actually walk. The quality of that policy decides whether the model can converge stably to a sufficiently good solution within an acceptable amount of time. Any discussion of large-model training efficiency and technical breakthroughs therefore has to run through the optimizer.
2. Three Core Problems an Optimizer Must Solve
An optimizer makes a sequence of decisions in parameter space, and at minimum it must handle three things at once:
Step size (learning rate) control. Too large a step can leap over the valley and up the opposite slope, causing the loss to oscillate violently or even diverge. Too small a step means the valley takes forever to reach, making training extremely slow and wasting compute. The ideal behavior is to move quickly over flat terrain and slowly where the slope is steep and the valley is near — in other words, adaptively adjusting the effective step size.
Escaping local optima and saddle points. The landscape is dotted with small pits halfway up the slopes (local optima) and flat mountain saddles. Following only the current slope downwards easily traps you in a shallow pit, where you mistake it for the true valley floor. An optimizer needs the ability to climb over such traps.
Convergence speed and stability. In large-model settings with hundreds of billions of parameters and expensive training runs, whichever optimizer converges faster and more reliably on the same hardware saves substantial compute and time. Stability matters especially, because an interrupted or diverged run usually means a large waste of resources.
3. From Steepest Descent to the Heavy Ball Method: The Birth of Inertia
The idea of gradient descent dates back to the steepest descent method proposed by the French mathematician Cauchy in 1847, but it only became a standard, widely used training algorithm more than a century later. Standard gradient descent contains a hidden assumption that contradicts physical intuition: it treats the parameter update as a massless particle sliding on a surface — one that can instantly start at maximum speed and stop on a dime, entirely unconstrained by inertia.
On ill-conditioned “narrow canyon” terrain, this inertia-free particle performs poorly. It bounces back and forth between the two steep walls of the canyon, oscillating repeatedly, while struggling to travel quickly along the canyon toward the valley floor. The assumption was broken by the Soviet mathematician Boris Polyak. In 1964, while studying optimization theory, he proposed an improvement grounded in strong physical intuition: give the particle “mass” and “inertia” so that it rolls like a real ball. This became the well-known heavy ball method, and the starting point of modern momentum.
The intuition behind the heavy ball method is straightforward. A real rolling ball accumulates momentum, which lets it keep accelerating along the main direction and coast through flat regions, while its inertia carries it across the narrow direction where it would otherwise oscillate. The result is suppressed lateral jitter and larger effective displacement.
4. Momentum: Bringing “Velocity” into Parameter Updates
Momentum turns the physical intuition of the heavy ball method into a computable update rule: maintain a “velocity” or momentum buffer that accumulates the direction of past gradients, and use it to update the parameters.
# Momentum (Heavy Ball) update, pseudocodev = 0 # velocity (momentum) bufferfor step in range(num_steps):g = compute_gradient(params) # current gradient from backpropagationv = beta * v + g # accumulate history, beta is typically around 0.9params = params - lr * v # update parameters along the momentum direction
Here beta controls how quickly historical information decays. The effect works in two directions:
- Acceleration along consistent directions: when consecutive gradients point the same way, momentum keeps accumulating, the effective step size grows, and the optimizer crosses flat regions quickly.
- Damping along oscillating directions: in directions that swing back and forth, consecutive gradients cancel each other out, momentum converges, and lateral jitter is noticeably reduced.
On top of this, Nesterov momentum adds a lookahead idea: predict the position one step ahead along the current momentum, then compute the gradient at that point. It is as if the optimizer looks where it is about to land, which often makes the final stage of convergence more stable and more precise. Momentum remains a cornerstone for understanding modern optimizers.
5. Adaptive Learning Rates: From Per-Parameter Scaling to Adam
A single global learning rate has an inherent flaw: different parameters and different layers of a model sit on radically different terrain, so applying one step size to all of them is unreasonable. Adaptive learning rate methods introduced the idea of estimating an effective step size per parameter. Typical representatives include AdaGrad, which scales by the sum of historical squared gradients, and RMSProp, which estimates the same quantity with a moving average.
The method that really popularized this idea is Adam. It combines “momentum” (the first moment, a moving average of gradients) with “adaptive step size” (the second moment, a moving average of squared gradients), and adds bias correction to reduce estimation bias early in training:
# Adam core update logic (simplified pseudocode)m = 0; v = 0beta1, beta2 = 0.9, 0.999eps = 1e-8for t in range(1, num_steps + 1):g = compute_gradient(params)m = beta1 * m + (1 - beta1) * g # first moment: directionv = beta2 * v + (1 - beta2) * (g * g) # second moment: per-parameter scalem_hat = m / (1 - beta1 ** t) # bias correctionv_hat = v / (1 - beta2 ** t)params = params - lr * m_hat / (sqrt(v_hat) + eps)
The strength of Adam is that it retains momentum’s resistance to oscillation and its acceleration, while automatically scaling the effective step size for each parameter, and it is relatively robust to hyperparameters. That is why it has long been the default choice for deep learning, especially Transformer-class large models. In engineering practice, AdamW decouples weight decay from the gradient update, making regularization more direct and controllable and avoiding contamination of the decay term by second-moment scaling. It usually achieves better generalization.
6. Muon: A New Optimizer for Large-Scale Training
While momentum and Adam primarily measure updates in the sense of the Euclidean norm, a newer class of optimizers that has drawn attention in the large-model training community — Muon, for example, which can be understood as momentum orthogonalized through Newton-Schulz iterations — approaches the update problem from a different geometric perspective.
Its core idea can be summarized as: first accumulate a momentum matrix as in momentum, then apply an approximate orthogonalization to that matrix, and finally update the parameters along the orthogonalized direction. Intuitively, orthogonalization equalizes the scale of the direction components in the update, bringing it closer to steepest descent in the sense of the spectral (operator) norm, which improves the conditioning of parameter matrices and the directional quality of the update.
# Muon core idea (pseudocode; newton_schulz is an approximate orthogonalization iteration)for W in weight_matrices:M = beta * M + G # maintain the momentum matrixO = newton_schulz_orthogonalize(M) # approximate orthogonalization via a few matrix iterationsW = W - lr * O # update weights along the orthogonalized direction
It is worth stressing that methods of this kind are usually designed for two-dimensional weight matrices, and in real training they are often combined with Adam-style methods — for example, using the new optimizer for matrix parameters and Adam for vector-type parameters. They offer a new direction for the question of how to train very large models more stably with less compute and less tuning effort. This remains a fast-moving research area, and the actual gains vary with model architecture, data, and training configuration, so they need experimental validation rather than fixed conclusions applied blindly.
7. Practical Engineering Recommendations
Once the evolution is clear, the following principles are useful for selection and tuning in real projects:
- Default baseline: for Transformer-class large models, Adam / AdamW with decoupled weight decay is a robust starting point.
- Learning rate schedule: use warmup followed by decay (cosine, linear, etc.). Warmup avoids divergence early in training when second-moment estimates are inaccurate; decay helps the final stage converge finely.
- Gradient clipping: clipping the gradient norm suppresses occasional gradient spikes and markedly improves stability in large-scale training.
- Monitor key metrics: continuously track the loss curve, gradient norm, and learning rate changes to catch oscillation or stagnation early and tune accordingly.
- Evaluate new optimizers rationally: for methods such as momentum and Muon, run controlled experiments at small and medium scale first, confirm stability and convergence behavior, and only then scale up rather than experimenting directly in an expensive training run.
8. Summary
The optimizer is an indispensable navigation system for large-model training, and its evolution is essentially a history of how to descend a mountain more intelligently: from the inertia-free assumption of steepest descent, to the mass and inertia introduced by Polyak’s heavy ball method, to the widespread adoption of momentum, to Adam fusing momentum with adaptive step sizes, and finally to new explorations such as Muon that improve the geometry of the update through orthogonalization. Their shared goal never changes — reaching a better solution in less time and in a more stable way. For developers, understanding the physical intuition and the applicable boundaries of these methods is more valuable than memorizing a particular hyperparameter value, and it is what makes rational, reproducible technical decisions possible in expensive large-model training.
