LLM Training Optimizers Explained: From SGD to Matrix-Based Methods
This article walks through the evolution of deep learning optimizers, framed by three core problems every optimizer must solve: how large a step to take (learning rate), how to escape local minima and saddle points, and which direction to update. It traces the progression from vanilla SGD to momentum methods (Heavy Ball, Nesterov), then to adaptive methods like Adam, and finally to a new class of matrix-based optimizers that respect the structure of weight matrices through orthogonalization. It also covers the demands that long-context, large-scale training place on optimizers, along with practical engineering recommendations for choosing and tuning them. Readers will gain a clearer mental model for optimizer selection and tuning in modern training scenarios.
1. Optimizers: The “Navigator” of Deep Learning
Imagine you are blindfolded and placed in a range of rolling mountains, tasked with reaching the lowest point of the valley. You have only two tools:
- Loss (the loss function): tells you your current altitude.
- Gradient (computed via backpropagation): you probe the slope of the ground with your foot, sensing which side points downhill.
The optimizer is the “walking strategy” in your head. It decides which direction to step next and how large that step should be. Given the same terrain and the same gradient signals, different strategies reach the goal with vastly different efficiency. This is why optimizers are considered the “navigator” of deep learning training, and a core component in determining whether a model can converge efficiently and stably.
2. The Three Questions an Optimizer Must Answer
A good optimizer essentially makes the right decisions on these three questions.
First, how large a step to take — learning rate control. A step that is too large may leap straight over the valley floor and up the opposite slope, causing the loss to oscillate and fail to converge. A step that is too small converges extremely slowly, driving up training costs. The ideal strategy adapts the step size: move faster in flat regions, and slower in steep regions near the valley floor.
Second, how to escape “dead ends” — breaking out of local optima and saddle points. On the mountainside there are many small pits (local minima) and plateaus (saddle points / flat regions). Relying on intuition alone to head straight downhill easily traps you in a small pit, making you mistake it for the true valley floor. In large-scale non-convex training, saddle points and flat regions are often more troublesome than isolated global minima.
Third, which direction to update — the geometry of the update. Vanilla gradients provide only a first-order descent direction, but the curvature of the loss surface is highly anisotropic — some directions are extremely steep while others are extremely flat. Accounting for this geometric difference in the update is the deepest challenge in optimizer design.
3. Evolution Stage One: From SGD to Momentum
The earliest standard Stochastic Gradient Descent (SGD) traces back to Cauchy’s solution of least squares in 1847. Its defining characteristic is treating parameter updates as a “massless particle” sliding on the loss surface — a particle that can instantly reach top speed and instantly brake to a stop, which completely violates physical intuition.
In a long, narrow “canyon” terrain, this massless model oscillates back and forth along the steep walls: the gradient repeatedly changes sign in the direction perpendicular to the valley floor, and the updates keep bouncing between the two walls without making progress along the length of the canyon, resulting in very low efficiency.
The real breakthrough came from the Soviet mathematician Boris Polyak’s Heavy Ball Method in 1964. His physical intuition is beautifully simple: give the particle “mass” and “inertia.” Once the particle gains velocity in a certain direction, it retains that extra momentum, so oscillations cancel each other out while consistent directions keep accelerating. Expressed as a formula:
# Momentum (Heavy Ball) pseudocodev = 0for t in range(steps):g = compute_gradient(theta)v = beta * v + g # accumulate "velocity/momentum"; beta is the inertia coefficienttheta = theta - lr * v # update parameters using the momentum-carrying velocity
Here beta controls how much of the historical gradient is retained: a larger value makes the trajectory smoother with weaker oscillations, but also makes direction adjustments more sluggish. Nesterov Accelerated Gradient (NAG), which came later, introduces a look-ahead strategy of “stepping forward along the momentum first, then computing the gradient at the predicted position.” This lets the particle “brake” in advance as it approaches the valley floor, offering better theoretical convergence properties. Methods of this kind are collectively called momentum-based optimizers — a natural continuation of the main line of “giving a massless particle inertia.”
4. Evolution Stage Two: Adaptive Learning Rate and Adam
Momentum solves the problem of consistent direction, but it does not answer “how large a step each parameter should take.” In practice, the gradient scales of different parameters vary enormously: sparse features, embedding layers, and pre-/post-normalization layers can differ in gradient magnitude by several orders of magnitude. A single global learning rate either makes large-gradient parameters oscillate violently or leaves small-gradient parameters barely updating.
Adaptive methods offer this answer: maintain a separate “step scaling factor” for each parameter. The core idea is that if a parameter’s gradient has consistently been large, its direction is steep, so its step size is automatically reduced; conversely, it is increased. These methods estimate scale via a moving average of squared gradients, with Adam as the typical representative. Adam combines momentum (the first-moment estimate) with adaptive scaling (the second-moment estimate) into one:
# Core Adam update logic (schematic)m = b1 * m + (1 - b1) * g # first moment: moving average of the gradientv = b2 * v + (1 - b2) * (g * g) # second moment: moving average of the squared gradientm_hat = m / (1 - b1**t); v_hat = v / (1 - b2**t) # bias correctiontheta = theta - lr * m_hat / (sqrt(v_hat) + eps) # element-wise adaptive update
Adam combines the smoothness of momentum with the robustness of adaptive scaling, and is relatively insensitive to hyperparameters, which is why it has become the de facto standard for large model training. But it is worth being cautious: Adam’s scaling is “element-wise” — it treats each number in a weight matrix as an independent entity, completely ignoring that the weights are actually a structured matrix.
5. The Bottleneck of First-Order Methods: Why We Need to “See the Matrix”
First-order methods, including Adam, use only gradient information and are essentially “probe only where you walk.” They know almost nothing about the curvature of the loss surface. Classical Newton’s method explicitly models curvature through the Hessian matrix and converges faster in theory, but it must compute and store a huge second-order matrix at every step — an entirely unaffordable cost in both computation and memory for models with billions of parameters. Quasi-Newton methods (such as BFGS/L-BFGS) mitigate this with approximations, but remain expensive in the stochastic, massive-parameter settings of deep networks.
So the question becomes: can we let the update “partially perceive” the geometry of the loss surface without introducing a full second-order matrix? This is exactly the question the new generation of optimizers tries to answer.
6. Matrix-Based Optimizers: Making Updates Respect Weight Structure
The core insight of matrix-based optimizers is that a neural network’s weights (such as the W of a linear layer) are not a loose collection of numbers, but a matrix with singular value structure. Previous methods scaled the matrix element by element, effectively discarding this structural information. Matrix-based optimizers, by contrast, transform the momentum matrix as a whole.
One fairly representative technical route is to first “orthogonalize” the matricized momentum — approximating the orthogonal factor in its polar decomposition through a few iterations (such as the Newton-Schulz iteration), thereby suppressing extreme directions with overly large singular values in the gradient and balancing the effective update across directions. Intuitively, this amounts to a gentle “preconditioning” of the update, making parameter updates more faithful to the true geometry of the loss surface while keeping the extra overhead within acceptable limits:
# Matrix orthogonalization (illustrative pseudocode; coefficients are for# illustration only and do not correspond to any specific implementation)def orthogonalize(X, iters=5):X = X / (X.norm() + 1e-7) # normalize to prevent numerical blow-upI = eye_like(X)for _ in range(iters):A = X @ X.TP = c1 * I + c2 * A + c3 * (A @ A) # polynomial approximation of the polar factorX = P @ Xreturn X# Usage sketch: first accumulate the momentum matrix, then orthogonalize it before updating the weights
It should be emphasized that such methods usually do not aim to completely replace Adam, but are “used in combination”: matrix-based updates are applied to the weight matrices of hidden layers in the network, while Adam is retained for biases, normalization parameters, embeddings, and so on. This preserves the robustness of adaptive methods while introducing higher-quality update directions on the weights most sensitive to structure. Its theoretical foundation is still an extension of the momentum and preconditioning ideas — an engineering realization of the broader direction of “approximating second-order information under limited compute budgets.”
7. New Requirements from Long Context and Large-Scale Training
As models move toward longer contexts (for example, processing on the order of a million tokens) and larger parameter scales, training places ever more demanding requirements on the optimizer:
- Stability first: ultra-long sequences make activation and gradient distributions more extreme, so the optimizer must suppress gradient spikes and avoid training divergence.
- Limited memory and bandwidth: first moments, second moments, and even the intermediate quantities of matrix-based methods all consume memory, so the optimizer’s extra state must be carefully budgeted.
- Coordination with learning rate schedules: scheduling strategies such as warmup and cosine decay are essentially dynamically controlling “step size,” working together with the optimizer’s internal adaptive scaling.
- Numerical precision: under mixed-precision training, optimizer states are often kept at higher precision, and iterative operations like orthogonalization require particular attention to numerical stability.
Together, these factors explain why “smarter updates” improve training efficiency more than “simply piling on compute.”
8. Practical Engineering Recommendations
- Establish a baseline first: build a reproducible training baseline with a mature, stable adaptive optimizer, then evaluate the marginal benefit of a new optimizer.
- Use a hybrid strategy: try matrix-based updates for weight matrices while keeping adaptive methods for normalization and biases, reducing overall risk.
- Tune the step size carefully: combine warmup and decay schedules, monitor gradient norms, and apply moderate clipping to prevent a few extreme directions from disrupting training.
- Watch memory usage: account for the extra state overhead of the optimizer, and when necessary combine distributed parallelism and state sharding to control the pressure on each device.
- Stay observable throughout: leverage the built-in optimizer and monitoring capabilities of mainstream deep learning frameworks and cloud AI training platforms (such as Baidu AI Cloud’s model training capabilities) to continuously track loss curves, gradient distributions, and learning rate trajectories, and drive tuning decisions with data.
9. Summary
The evolution of optimizers is a main line of “a massless particle gradually gaining inertia and geometric awareness”: SGD provided the descent direction, momentum methods gave it inertia and smoothness, adaptive methods (Adam) brought per-parameter adaptive step sizes, and new matrix-structure-centric optimizers attempt to let updates “understand” the geometry of the weights under controllable overhead. By grasping the three-fold problem of learning rate, escaping local optima, and update direction, you can make more informed choices and tuning decisions in the modern training scenarios of long context and massive scale, truly converting precious compute into the model’s convergence efficiency.
