From SGD to Muon: Optimizer Evolution and Selection for Large Model Training
This article reviews optimizer choices for large model training, from SGD and momentum to AdamW and Muon. It explains the intuition behind each approach, compares computational and memory tradeoffs, presents a minimal Muon implementation, and outlines practical guidance for parameter grouping, learning-rate tuning, weight decay, numerical precision, distributed training, and selection with rollback planning.
1. Introduction: More Than Parameters: How Training Reaches the Optimum
New-generation large model previews increasingly emphasize long-context capability, inference efficiency, and training efficiency. Parameter scale sets an upper bound on model capability, but whether a model can approach that bound under limited compute and time depends heavily on the training process: data mixture, training strategy, parallelism, and optimizers.
Optimizers such as Muon are becoming a high-frequency topic in discussions of large model training. How does Muon differ from classical SGD and AdamW? Why is it attracting attention in large-model settings? What should engineering teams watch for when deploying it? To answer these questions, it helps to return to the classic metaphor of gradient descent: descending a mountain while blindfolded.
2. What Is an Optimizer: A Navigation Strategy While Descending Blindfolded
Imagine you are blindfolded and placed in a continuous mountain range. Your goal is to reach the lowest point in the valley. You have only two tools:
- Loss: like an altimeter, it tells you how “bad” your current position is.
- Gradient: obtained through backpropagation, it is like the slope you probe with your feet, telling you which direction is steeper and which way goes downhill.
The optimizer is the “action strategy”: it decides where to step next, how large the step should be, and whether to rely on past paths. Usually it must answer three questions:
- How should the step size be set? Too large a step may overshoot the valley bottom and cause loss oscillation; too small a step makes training too costly.
- How can it escape small pits and saddle points? In high-dimensional loss surfaces, local minima and saddle points are common. Relying only on the instantaneous gradient can easily trap the optimizer in non-optimal regions.
- Should different parameters be treated differently? Gradient scales differ greatly across parameters in a neural network, so a uniform learning rate is often not efficient enough.
Around these questions, optimizers have gone through several stages:
- SGD: a massless particle that takes one step along the current negative gradient.
- Momentum methods: heavy-ball and Nesterov methods add inertia to the particle.
- Adaptive methods: AdaGrad, RMSProp, and Adam give each parameter its own effective step size.
- Matrix-level methods: represented by Muon, shifting from element-wise scaling to matrix-structure handling.
3. SGD: A Massless Particle
The SGD update rule is very simple:
g_t = ∇L(θ_t)θ_{t+1} = θ_t - η * g_t
Here, θ is the parameter, η is the learning rate, and g_t is the current mini-batch gradient. SGD behaves like a particle without mass: its velocity is entirely determined by the current gradient, with no inertia and no memory.
SGD has several main issues:
- Oscillation in ill-conditioned ravines: when curvature differs greatly across directions, the steepest-descent direction tends to bounce repeatedly on steep directions.
- Lack of directional smoothing: historical direction is not retained, so “forward information” accumulated in a long, narrow valley is easily lost.
- Sensitivity to learning rate and momentum: vanilla SGD requires substantial tuning on complex tasks.
Still, SGD remains important: it is simple to implement, has low memory overhead, and mini-batch noise can help escape sharp minima in some scenarios. In mature tasks such as vision, a well-tuned SGD + Momentum can still be a strong baseline.
4. Heavy Ball and Nesterov: Giving the Particle Mass
In 1964, Polyak proposed the heavy-ball method, adding physical intuition to gradient descent: the particle should not be massless; it should have velocity, inertia, and friction.
v_t = β * v_{t-1} + g_tθ_{t+1} = θ_t - η * v_t
Equivalently, it can be written as:
θ_{t+1} = θ_t - η * g_t + β * (θ_t - θ_{t-1})
Here, β is the momentum coefficient, commonly around 0.9. Momentum has two key effects:
- Directional smoothing: when adjacent gradient directions are consistent, velocity accumulates; when directions fluctuate, part of the update cancels out. This is effectively an exponential moving average of historical gradients.
- Effective step-size amplification: if the gradient continues in the same direction, the steady-state velocity satisfies
v* = g / (1 - β), giving an effective learning rate of aboutη / (1 - β). Withβ=0.9, the effective step size is amplified by roughly 10x.
Nesterov accelerated gradient adds a “lookahead” idea: it first estimates one step along the momentum direction, then computes the gradient at that estimated position.
v_t = β * v_{t-1} + ∇L(θ_t - η * β * v_{t-1})θ_{t+1} = θ_t - η * v_t
Intuitively, instead of only looking at the current step, it first asks “where will inertia take me?” and then decides whether to brake early. Momentum methods significantly improve convergence efficiency on ill-conditioned terrain, but they still assume that all parameters share the same learning rate and do not address differences in gradient scale across parameters.
5. Adaptive Optimizers and Adam/AdamW: Each Parameter Gets Its Own “Shoe Size”
5.1 AdaGrad and RMSProp
AdaGrad accumulates squared historical gradients for each parameter:
G_t = G_{t-1} + g_t * g_tθ_{t+1} = θ_t - η * g_t / (sqrt(G_t) + ε)
The intuition is: parameters with large long-term gradients take small steps, while parameters with small long-term gradients take large steps. AdaGrad works well for sparse features, but its learning rate monotonically decreases, which may cause it to lose updating ability too early in late training.
RMSProp uses an exponential moving average instead of unlimited accumulation:
E_t = β2 * E_{t-1} + (1 - β2) * g_t * g_t
β2 is commonly 0.999, meaning it focuses on recent gradient scales.
5.2 Adam
Adam combines three elements:
- First-moment estimation: momentum;
- Second-moment estimation: adaptive scaling;
- Bias correction: fixing initialization bias.
m_t = β1 * m_{t-1} + (1 - β1) * g_tv_t = β2 * v_{t-1} + (1 - β2) * g_t * g_tm_hat = m_t / (1 - β1^t)v_hat = v_t / (1 - β2^t)θ_{t+1} = θ_t - η * m_hat / (sqrt(v_hat) + ε)
Bias correction is important. At the beginning of training, m and v start from zero, so uncorrected values are systematically too small. This is especially true for the second moment. If β2=0.999, the uncorrected value at step 1 is only 0.001*g^2; after taking the square root, it is about 0.0316*|g|. Without correction, early effective step sizes would be severely distorted.
Default Adam settings such as β1=0.9, β2=0.999, and ε=1e-8 are robust across many tasks and have had a profound impact on large language model training.
5.3 AdamW: Weight Decay Must Be Decoupled
In Adam, if L2 regularization is added directly into the gradient, the decay term will also be divided by sqrt(v_hat)+ε through adaptive scaling. This leads to inconsistent effective decay strength across parameters and makes control difficult.
AdamW decouples weight decay from the gradient:
θ_{t+1} = θ_t - η * (m_hat / (sqrt(v_hat) + ε) + λ * θ_t)
With this formulation, weight decay is applied consistently across parameters and its strength can be precisely controlled. Today, Adam-like optimizers used in large model training usually adopt AdamW-style decoupled weight decay.
5.4 The Cost: Optimizer State Memory
Adam needs to maintain two state tensors for every parameter: the first moment and the second moment. With mixed-precision training, there may also be a master-weight copy. As a result, optimizer state memory can be on the same order of magnitude as the weights themselves. For models with hundreds of billions of parameters, this is a cost that must be carefully managed.
6. Muon: From Element-wise to Matrix-level
6.1 Core Idea: Orthogonalizing the Momentum Matrix
The core step of Muon is to orthogonalize the momentum matrix.
M_t = β * M_{t-1} + g_tO_t = Orthogonalize(M_t)θ_{t+1} = θ_t - η * O_t
From an SVD perspective, if:
M = U * Σ * V^T
then orthogonalization can be intuitively understood as setting all non-zero singular values to 1, yielding:
O = U * V^T
Compared with Adam:
- Adam scales each gradient element to roughly the same magnitude;
- Muon scales the principal directions of the momentum matrix to comparable strength.
Therefore, Muon’s update is not “hijacked” by the one or two highest-energy directions in the momentum matrix. Instead, it acts more evenly across multiple directions.
6.2 Geometric Intuition: Balanced Updates Across Multiple Directions
If you move one step only along the strongest direction of the momentum matrix and remove that direction’s component, it is equivalent to subtracting:
σ₁ * u₁ * v₁^T
where σ₁ is the largest singular value, and u₁, v₁ are the corresponding singular vectors. Repeating this process can be intuitively understood as gradually “walking through” the multiple directions spanned by the momentum matrix until their singular values are reduced to similar levels.
One caveat: strictly speaking, the statement that an orthogonal matrix has rank equal to min(m, n) applies to non-degenerate, full-rank cases. For common two-dimensional weight matrices, min(m, n) is usually greater than 1, but degenerate matrices, low-rank matrices, or zero singular values need to be understood separately. Muon’s engineering value lies in making the update scale of two-dimensional matrix parameters more stable at every step, reducing the dominance of a single direction during training.
6.3 Why Muon Can Be Understood as a “Trust-Region-like” Update
Muon’s orthogonalized update can be viewed as analogous to an optimization problem with a spectral-norm constraint:
maximize <M, Δ>subject to ||Δ||₂ ≤ 1
where ||Δ||₂ denotes the spectral norm, i.e., the largest singular value. This problem is closely related to polar decomposition. If M = U Σ V^T, one optimal direction is:
Δ = U * V^T
A common misconception should be clarified here: under a spectral-norm constraint, the dual norm of the linear objective <M, Δ> is not the spectral norm, but the nuclear norm. That is, the maximum value of this optimization problem is:
||M||_* = Σ σ_i
rather than always being equal to 1. If one wants to normalize the objective value to 1, M should be scaled by its nuclear norm, not by simply normalizing the trace or the spectral norm to 1. In practice, Muon implementations usually do not rely on this normalization. Instead, they directly use the orthogonalized update matrix and then multiply it by the learning rate and shape scaling.
Thus, Muon can be understood as follows: within a “trust region” where the spectral norm is controlled at each step, the parameter update is prevented from being dominated by a single strong singular direction.
6.4 Muon vs. Shampoo
Shampoo is a family of second-order optimizers that approximate parameter-space curvature using left and right Kronecker factors. It maintains more complex second-order statistics, with higher computational and memory costs.
Muon only processes the momentum matrix itself and does not explicitly maintain second-order curvature statistics. Therefore, its state overhead is relatively closer to SGD + Momentum. The tradeoff is that it mainly applies to two-dimensional matrix layers and cannot naturally take effect on non-two-dimensional parameters.
6.5 Newton-Schulz Iteration: Approximating Orthogonalization with Matrix Multiplications
In theory, one could perform SVD directly. However, in large model training, doing an SVD for every layer at every step is too expensive, and numerical stability under low precision is not ideal. Muon therefore uses Newton-Schulz iteration to approximate orthogonalization with matrix multiplications.
For a singular value s, the iteration can be written as:
X ← a * X + b * (X * X^T) * X + c * (X * X^T)^2 * X
This is equivalent to applying a polynomial mapping to the singular values:
s ← a * s + b * s^3 + c * s^5
Different implementations may use different (a, b, c) coefficients. In engineering, it is essential to keep the formula, code, and coefficients consistent; otherwise, the orthogonalization effect may be distorted.
To keep the formula and code self-consistent, here is a common cubic approximation:
g(s) = 1.5 * s - 0.5 * s^3
The corresponding matrix update is:
X ← 1.5 * X - 0.5 * (X * X^T) * X
After normalizing with the Frobenius norm first, the singular values roughly fall in (0, 1]. For 0 < s < 1, g(s) > s, and g(1)=1, so the iteration gradually pushes singular values toward 1. In engineering practice, about 5 iterations are commonly used as a compromise between precision and cost. If the matrix rank is very low, singular values are extremely small, sharding makes the matrix incomplete, or low-precision errors are large, it may be necessary to increase the number of iterations, keep an FP32 state, or even use a more precise orthogonalization scheme.
6.6 Applicability Boundaries
Muon is not a complete replacement for Adam. It mainly applies to two-dimensional weight matrices and assumes that a complete matrix can be obtained at every step. If matrices are sharded, momentum buffers are not kept in FP32, or shape scaling does not match the actual dimensions, orthogonalization may fail, update scales may become abnormal, or training spikes may occur. When these issues arise, check parameter grouping, distributed aggregation, numerical precision, and gradient clipping configuration first.
7. Side-by-side Comparison
| Dimension | SGD + Momentum | AdamW | Muon |
|---|---|---|---|
| Core idea | Move along the negative gradient, using momentum to smooth direction | Element-wise adaptive scaling using first and second moments | Orthogonalize the momentum matrix and update from it |
| Normalization | None, or used with gradient clipping | Element-wise, based on each parameter’s gradient history | Matrix-level, approximating orthogonalization in the spectral-norm sense |
| Extra state | Optional: one momentum buffer | First moment + second moment | One momentum buffer |
| Main computation | Several times the gradient computation | Linear in parameter scale | Several additional matrix multiplications |
| Applicable parameters | Any parameter | Any parameter | Two-dimensional weight matrices |
| Advantages | Simple, memory-efficient, noise may have a regularizing effect | Robust, friendly hyperparameters | More consistent update scale across layers; exploits matrix structure |
| Watch-outs | Sensitive to learning rate/momentum; slow on ill-conditioned terrain | High memory overhead; regularization must be decoupled | Relatively new ecosystem; non-2D parameters need another optimizer |
It should be emphasized that Muon is usually not used alone, but in combination with AdamW. Two-dimensional matrix layers are assigned to Muon, while embedding layers, output heads, biases, normalization parameters, and others are assigned to AdamW.
8. Minimal Working Muon Example
Runtime note: the pseudocode below targets Python 3.10+ / PyTorch 2.x. Optimizer, no_grad, and zeros_like can be replaced with torch.optim.Optimizer, torch.no_grad, and torch.zeros_like. In a real framework, parameter groups and state management can be adjusted according to the API.
def orthogonalize(G, steps=5, eps=1e-7):"""Approximate orthogonalization of a momentum matrix using Newton-Schulz iteration.Here the singular-value polynomial is:g(s) = 1.5*s - 0.5*s^3The corresponding matrix update is:X <- 1.5*X - 0.5*(X@X.T)@X"""a, b = 1.5, -0.5# Normalize by the Frobenius norm so the spectral norm is roughly bounded by 1.X = G / (G.norm() + eps)# If the matrix is tall, transpose it so multiplications happen along the shorter dimension.transposed = X.shape[0] > X.shape[1]if transposed:X = X.Tfor _ in range(steps):A = X @ X.TX = a * X + b * (A @ X)if transposed:X = X.Treturn Xclass Muon(Optimizer):def __init__(self, params, lr=0.02, momentum=0.95, nesterov=True):super().__init__(params, dict(lr=lr, momentum=momentum, nesterov=nesterov))@no_grad()def step(self):for group in self.param_groups:lr = group["lr"]momentum = group["momentum"]for p in group["params"]:if p.grad is None:continuestate = self.state[p]buf = state.setdefault("momentum_buffer", zeros_like(p))# Momentum accumulationbuf.mul_(momentum).add_(p.grad)# For the Nesterov variant, combine the current gradient and momentum as the orthogonalization input.if group["nesterov"]:g = p.grad + momentum * bufelse:g = bufg = orthogonalize(g)# Shape scaling: compensate for how matrix shape affects update RMS.scale = max(1.0, p.shape[-2] / p.shape[-1]) ** 0.5p.add_(g, alpha=-lr * scale)
8.1 Understanding Shape Scaling
For an m x n matrix, the orthogonalized update satisfies:
||O||_F ≈ sqrt(min(m, n))
where ||O||_F is the Frobenius norm. The total number of matrix elements is m*n, so without scaling, the element-wise RMS is:
RMS ≈ 1 / sqrt(max(m, n))
After multiplying by:
scale = max(1, m / n)^0.5
the update RMS becomes approximately:
RMS ≈ 1 / sqrt(n)
where n is the matrix input dimension, i.e., p.shape[-1]. This allows matrices of different shapes to use a more similar effective step size and avoids scale differences between wide and narrow layers caused by matrix shape.
8.2 Parameter Grouping: Do Not Accidentally Apply Muon to Embeddings and Output Heads
Muon is usually only used for two-dimensional linear weight matrices. Embedding layers, output heads, normalization parameters, biases, and similar parameters should be assigned to AdamW. Example:
import reskip_pattern = re.compile(r"(embed|embedding|wte|lm_head|output|head|norm|bias)",re.IGNORECASE)muon_params, adamw_params = [], []for name, p in model.named_parameters():lname = name.lower()if p.ndim == 2 and not skip_pattern.search(lname):muon_params.append(p)else:adamw_params.append(p)opt_muon = Muon(muon_params, lr=0.02, momentum=0.95)opt_adamw = AdamW(adamw_params,lr=3e-4,betas=(0.9, 0.95),weight_decay=0.1)
If model names are not well-structured, two-dimensional matrices such as nn.Embedding, lm_head, and output heads must be manually excluded. Otherwise, this may cause dimension errors, failed orthogonalization, or unstable training.
9. Engineering Implementation Notes
Parameter grouping comes first
- Include only two-dimensional linear weight matrices in Muon.
- Keep embedding layers, output heads, biases, and normalization parameters on AdamW.
- Incorrect grouping can cause dimension errors at best and break training stability at worst.
The learning rate needs to be retuned
- Muon’s update scale differs from AdamW’s, so AdamW learning rates cannot be reused directly.
- After shape scaling, the update RMS is approximately
η / sqrt(n), wherenis the matrix input dimension. - Warmup is usually needed, and the learning rate should be validated on the target model size.
- Keeping a reasonable minimum learning rate near the end of training helps smooth convergence.
Set weight decay independently
- Muon itself does not include weight decay, so decoupled decay needs to be implemented separately, similar to AdamW.
- Decay must not be mixed in before orthogonalization, as this may distort the momentum direction.
Momentum and gradient clipping
- The momentum coefficient is commonly around
0.95. - Global gradient-norm clipping is still a necessary safety net and can effectively suppress loss spikes.
- The momentum coefficient is commonly around
Numerical precision
- Newton-Schulz iteration can run in low precision.
- Momentum buffers and parameter updates should keep high-precision copies to avoid long-range error accumulation.
Distributed training and state management
- Under sharding schemes such as ZeRO or FSDP, matrix parameters may be split.
- Orthogonalization requires the full matrix at each step. A common approach is to gather parameters one by one or complete orthogonalization before sharding.
- When resuming from a checkpoint, momentum buffers must be fully saved and restored.
- It is recommended to monitor metrics such as gradient clipping ratio, update norms, and loss spike frequency.
Validate at small scale before scaling up
- Run short ablation experiments on small models to compare convergence speed and downstream metrics.
- Once targets are met, scale to the target model size and prepare a one-click rollback plan.
10. Selection Decision Framework
How to Choose Between Muon and AdamW?
- Small and medium models, mature vision tasks: SGD + Momentum or AdamW is usually sufficient. Prioritize reproducibility and stability.
- Large language model pretraining with goals of training efficiency or tight memory budgets: evaluate a Muon + AdamW hybrid approach.
- Sparse recommendation scenarios or feature-distribution drift: AdamW is usually safer.
- For any new optimizer: before deployment, compare it strictly with the baseline under the same conditions and prepare a rollback plan.
What Are Common Muon Mistakes?
- Assigning embedding layers or output heads to Muon.
- Applying Muon to non-two-dimensional parameters.
- Using bf16 momentum accumulation, causing long-range error amplification.
- Failing to gather matrices under distributed sharding.
- Shape scaling not aligned, causing inconsistent update scales across layers.
- Gradient clipping not working, or weight decay mixed in before orthogonalization.
How to Tune the Muon Learning Rate?
- Do not directly copy the AdamW learning rate.
- First validate on a small model whether update norms are reasonable.
- Note that the update RMS is approximately
η / sqrt(n), wherenis the matrix input dimension. - Use warmup and keep a minimum learning rate in the later stage of training.
11. Conclusion
From Cauchy’s steepest descent, to Polyak’s heavy-ball method, then to adaptive optimizers and matrix-level orthogonalization, the evolution of optimizers is essentially a history of continuously giving the downhill traveler more senses:
- SGD is like a particle without mass or memory;
- Heavy ball gives it mass and inertia;
- Adam gives each parameter its own shoe size;
- Muon lets matrix layers move more evenly at every step.
Optimizer selection is not about “newer is better,” but about “more suitable is better.” Understanding the mathematical intuition, engineering boundaries, and tuning details behind each method is what enables rational and reproducible decisions for a specific task.
