Large Language Model Optimizers: From Gradient Descent to Muon
Optimizers are a core control mechanism in large language model training. They determine how gradients are translated into parameter updates, affecting training stability, convergence speed, memory use, and distributed efficiency. This article explains the evolution from SGD and momentum to Adam, then examines Muon’s matrix-aware optimization approach, its potential benefits, limitations, and practical evaluation considerations.
In large language model training, an optimizer is more than a simple parameter-update tool. It is the core control mechanism that connects gradient computation, model parameters, and training objectives.
The loss function measures the model’s current performance. Backpropagation calculates the direction in which parameters should change. The optimizer determines how parameters actually move, how far they move, and how historical information should influence the next update.
From this perspective, the optimizer has a direct impact on training stability, convergence speed, and how effectively a fixed compute budget can be converted into model quality. As models continue to scale, conventional optimization methods have shown growing limitations in memory consumption, communication overhead, matrix-parameter update efficiency, and training stability. This has made new optimizer designs an important area of large-model research. Muon has attracted attention in this context as a matrix-oriented optimization approach.
1. What Problems Does an Optimizer Solve?
Let the model parameters be (\theta), the training objective be the loss function (L(\theta)), and the gradient be:
[
gt=\nabla\theta L(\theta_t)
]
The most basic gradient descent update is:
[
\theta_{t+1}=\theta_t-\eta g_t
]
where (\eta) is the learning rate. Although this equation is simple, it contains three central optimization challenges.
1. How Should the Update Size Be Chosen?
If the learning rate is too large, parameters may repeatedly overshoot around an optimum or diverge entirely. If it is too small, training becomes slow and may fail to reach the desired result within the available compute budget.
The problem is more complex because different directions in the loss landscape often have different curvature. Some directions are steep, where even a small movement can significantly change the loss. Others are relatively flat and require larger steps to make meaningful progress. A single uniform step size is often unable to handle all directions effectively.
Optimizers therefore typically use gradient history, parameter states, or local geometric information to adjust update magnitudes.
2. How Can Oscillation Be Reduced?
In a narrow loss valley, gradient descent may oscillate from one side to the other. Even if each update locally reduces the loss, substantial computation can be wasted on lateral movement while progress along the valley floor remains slow.
This problem is common in deep networks and high-dimensional parameter spaces. An optimizer needs to distinguish between directions with sustained downward trends and directions dominated by short-term noise, reducing ineffective movement.
3. How Should Stochastic Gradient Noise Be Handled?
Large models are typically trained with mini-batches. The resulting gradient is only an approximation of the full-data gradient and therefore naturally contains noise. Some noise can help the model escape local regions, but excessive noise can make training unstable.
An effective optimizer must balance exploration with stable convergence.
2. From Stochastic Gradient Descent to Momentum
Stochastic Gradient Descent, or SGD, is the most fundamental training method. At each step, it samples a mini-batch from the dataset, computes the gradient for that batch, and updates the parameters.
Its advantages include simple implementation, low memory overhead, and broad applicability. However, it has two clear limitations:
- Gradient noise can be large, making update directions unstable.
- It can produce substantial oscillation in narrow loss regions.
To address these issues, researchers introduced momentum. Rather than relying only on the current gradient, momentum maintains an accumulated historical term:
[
vt=\beta v{t-1}+g_t
]
[
\theta_{t+1}=\theta_t-\eta v_t
]
where (\beta) controls how much historical information is retained.
From a physical perspective, momentum gives the optimization process a form of inertia. The current gradient provides a new force direction, while historical velocity preserves the broader movement trend. Gradients that consistently point in the same direction accumulate over time, while short-term opposing noise is less likely to immediately reverse the update direction.
Momentum provides two main benefits: it accelerates movement along stable descent directions and suppresses frequent oscillation in high-curvature directions. It also established the foundation for more sophisticated adaptive optimizers.
3. Why Adam Became a Long-Standing Default
Beyond momentum, another important idea is to track second-order gradient statistics. A typical approach maintains both the first moment and second moment of gradients:
[
mt=\beta_1m{t-1}+(1-\beta_1)g_t
]
[
vt=\beta_2v{t-1}+(1-\beta_2)g_t^2
]
After bias correction, parameters are updated using a rule similar to:
[
\theta_{t+1}
=
\theta_t-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}
]
The first moment can be understood as a smoothed estimate of gradient direction. The second moment reflects the scale of gradient variation across different parameter directions. For directions with consistently large gradients, the optimizer automatically reduces the effective step size. For directions with smaller or sparse gradients, it can preserve relatively more aggressive updates.
Adam offers several advantages:
- It is less sensitive to the initial learning rate.
- It adapts to gradient scales across different parameter dimensions.
- It handles sparse and noisy gradients effectively.
- It performs reliably across many model architectures and tasks.
However, Adam also introduces additional costs. Each parameter typically requires extra first-moment and second-moment states, which can make optimizer state several times larger than the model parameters themselves. In large-model training, this significantly increases memory consumption, parameter communication, and checkpoint storage requirements.
In addition, element-wise scaling is flexible but may not fully exploit the global structure of weight matrices. For neural network parameters with clear matrix structure, adjusting updates only at the individual-element level may fail to capture higher-level geometric relationships in parameter space.
4. The Core Idea Behind Muon
Muon can be viewed as an optimization approach designed for matrix parameters. Instead of simply assigning an independent step size to every parameter, it combines momentum with matrix structure and further processes the update direction.
Its basic workflow can be summarized as follows:
- Update momentum using the current gradient.
- Treat the momentum as a matrix-form update direction.
- Apply orthogonalization or approximate orthogonalization to the matrix.
- Update weight parameters using the processed direction.
Here, “orthogonalization” does not mean that all parameters become completely independent directions. Instead, through matrix decomposition, iterative approximation, or other numerical techniques, the update matrix is processed to make its singular value distribution more balanced and prevent a small number of directions from dominating the update.
Intuitively, ordinary momentum focuses on where to move. Matrix orthogonalization additionally considers whether movement across different directions is imbalanced. If some directions receive much larger updates than others, parameter changes may become overly concentrated, training may oscillate, or model capacity may be used inefficiently. Structured processing of the update matrix can make changes across different directions more balanced.
Muon is not a universal replacement for every parameter type. It is better suited to weight parameters with a clear two-dimensional matrix structure. Biases, normalization parameters, and other vector or scalar parameters generally require update strategies that better match their shapes.
In practice, this often leads to hybrid optimization: matrix weights use a matrix-oriented optimizer, while other parameters use a conventional optimizer or a specialized simplified update rule.
5. Potential Technical Benefits of Muon
1. Better Use of Matrix Structure
Traditional element-wise optimizers treat matrix parameters as many independent scalar values and ignore relationships between rows and columns. Muon treats weights directly as matrices, allowing update directions to be constructed from a more global perspective that better matches the structure of linear transformation layers.
2. Reduced Optimizer-State Overhead in Some Cases
Some matrix-based optimization approaches do not need to maintain complete second-order statistics for every parameter. Depending on the implementation, this may reduce optimizer-state memory pressure.
However, actual overhead still depends on how momentum is stored, the matrix-processing algorithm, mixed-precision settings, and the parallel implementation. It should not be assumed that memory use will decrease in every scenario.
3. More Balanced Update Directions
One objective of matrix orthogonalization is to prevent a small number of singular directions from accounting for an excessive share of the update. More balanced directions may improve training stability and reduce extreme changes in certain parameter subspaces.
4. Compatibility with Large-Scale Parallel Training
In distributed training, an optimizer must be evaluated not only by its mathematical behavior but also by its communication and computational efficiency. Matrix-level processing may be combined with tensor parallelism, data parallelism, and mixed-precision computation.
However, this requires efficient low-level batched matrix operations. Otherwise, the additional cost of orthogonalization can offset the expected gains.
6. Practical Considerations for New Optimizers
First, do not evaluate an optimizer only by how quickly training loss decreases. Validation metrics, gradient norms, parameter norms, learning-rate behavior, and the ratio of update magnitude to weight magnitude should also be monitored. Some optimizers may reduce loss faster in early training but may not produce better final generalization.
Second, distinguish between parameter types. Matrix weights, vector parameters, biases, and normalization parameters should not automatically use the same update rule. Effective parameter grouping is often essential for stable hybrid optimization.
Third, numerical stability matters. Matrix decompositions, iterative orthogonalization, and normalization operations can all be affected by low-precision computation. Engineering implementations should select appropriate data types based on hardware capabilities and include safeguards for very small values, abnormal gradients, and numerical overflow.
Finally, hyperparameters remain important. A new optimizer does not eliminate the need to tune learning rate, weight decay, momentum coefficients, or warmup schedules. Different model sizes, batch sizes, and training stages often require different configurations.
7. Building a Reliable Evaluation Process
In practical experiments, optimizers should be compared using controlled variables:
- Keep the model architecture, data order, and number of training steps fixed.
- Keep the effective batch size and learning-rate search range fixed.
- Record training loss and validation metrics.
- Measure memory usage, throughput, and per-step execution time.
- Monitor abnormal gradients, loss spikes, and training interruptions.
- Compare final quality against compute cost.
It is also useful to analyze the relative update magnitude:
[
r_t=\frac{|\Delta\theta_t|}{|\theta_t|+\epsilon}
]
If this value remains too large over time, the learning rate or matrix-processing procedure may be unstable. If it remains too small, the optimizer may be overly conservative, or training may have entered an inefficient stage.
Conclusion
The development of optimizers is fundamentally an ongoing effort to find workable solutions more efficiently in complex loss landscapes. SGD addressed the basic problem of directional search. Momentum introduced historical information. Adam improved step-size control across parameter dimensions through adaptive statistics. Muon goes further by attempting to use matrix structure to optimize update directions.
For large language model training, optimizer selection cannot be separated from model architecture, hardware environment, parallelization strategy, and training objectives. The matrix-level optimization ideas represented by Muon suggest that future optimizers may move beyond per-parameter adjustment and place greater emphasis on parameter blocks, matrix geometry, and the structure of global updates.
A meaningful technical breakthrough is not merely a new formula. Its value lies in whether it can achieve a better balance among stability, convergence efficiency, resource consumption, and final generalization performance.
