Gradient Descent Optimizers: What Large Model API Upgrades Reveal About Seven Common Methods
This article explains how large model API upgrades (longer context, structured output, streaming, lower latency) are ultimately backed by training-time choices, and focuses on gradient descent optimizers. It covers how optimizers solve the "blindfolded descent" problem, the progression from SGD to momentum and Nesterov acceleration, adaptive-step methods (Adagrad, RMSProp, Adam) with decoupled weight decay, learning rate scheduling and saddle-point handling, and matrix-level methods like Shampoo and Muon. It closes with a comparison table, a minimal tuning checklist, and pseudocode, so readers can build a practical framework for choosing and tuning optimizers for LLMs.
1. Introduction: An API Upgrade Is Never a Single Feature
New versions of large model APIs tend to emphasize longer context, structured output, streaming responses, lower latency, and higher throughput. To a developer, these are features you can use directly. From a systems perspective, however, they are only the presentation layer of capability. What actually determines the ceiling is the coordination of data, training objectives, optimizers, parallel strategies, inference acceleration, and engineering governance.
During training, the optimizer is the core component that decides how parameters are updated toward a better state. A common framing calls gradient-descent-related methods the “seven weapons.” To avoid overstating the claim, it is more accurate to treat them as seven categories of common methods. This article addresses practical questions such as how to choose a large-model optimizer, how to tune Adam, learning rate warmup, and gradient clipping, and helps readers build a tuning framework they can actually apply.
2. Blindfolded Descent: Three Jobs of an Optimizer
You can picture model training as descending a mountain while blindfolded. You cannot see the global terrain, so you rely on three kinds of information:
- The loss function tells you how far you are from the goal.
- The gradient tells you the steepest local direction of ascent; its opposite is the direction of descent.
- The optimizer decides which way to step next and how far to go.
A parameter update can be abstracted as:
θ_{t+1} = θ_t + Δθ_t
The optimizer has to answer three questions:
- How big a step: a learning rate that is too large causes oscillation or divergence, while one that is too small converges slowly.
- Whether the direction is stable: raw gradients are noisy, so the direction needs to be smoothed using historical information.
- Whether it can escape local minima and saddle points: in high-dimensional non-convex terrain, noise, scheduling, or extra mechanisms help the optimizer escape.
3. From SGD to Momentum: Adding Inertia to Parameter Updates
The idea of gradient descent can be traced back to the mid-19th century. It is commonly attributed to Cauchy (1847) and his work on solving systems of equations. Standard SGD is simple in form, but it behaves like a particle with no mass: its velocity is determined entirely by the current gradient. In a long, narrow valley, the gradient oscillates back and forth across the steep direction while progress along the valley floor is slow; at a saddle point, it may stall for a long time.
The core idea of momentum is to give the particle mass. Its update rule can be written as:
vt = μ·v{t−1} − η·∇f(θ{t−1})
θ_t = θ{t−1} + v_t
Here μ is the momentum coefficient, often set to 0.9, 0.95, or even higher. Momentum works because it smooths historical gradients: on oscillating directions, positive and negative gradients cancel each other out, while on a consistent forward direction, velocity accumulates gradually.
Two implementations deserve special attention. Plain momentum computes the gradient at the current parameter point and combines it with historical velocity to form the update. Nesterov acceleration instead computes the gradient at a velocity-lookahead point. Intuitively, plain momentum is like walking while looking at your feet, whereas Nesterov looks ahead before stepping, so it senses an upcoming slope earlier.
4. Adaptive Step Size: Adagrad, RMSProp, and Adam
Momentum solves direction smoothing, but it does not solve the problem that different parameters need different step sizes. In real models, some parameters have gradients that are persistently large, while others are updated sparsely, so a single global learning rate is not appropriate. Newton’s method can adapt the step size using curvature information, but computing the full Hessian is too costly. In engineering practice, a low-cost per-parameter second-order statistic is used instead.
Adagrad accumulates the squared historical gradient. It suits sparse features, but because the accumulated quantity only grows and never shrinks, the learning rate can decay too early in later stages. RMSProp switches to an exponential moving average, which mitigates this problem. The widely adopted Adam then combines a first-moment momentum with a second-moment moving average:
mt = β1·m{t−1} + (1−β1)·gt
v_t = β2·v{t−1} + (1−β2)·gt²
θ_t = θ{t−1} − η·m̂_t / (√v̂_t + ε)
m̂_t = m_t / (1−β1^t)
v̂_t = v_t / (1−β2^t)
Adam has three key points: the first moment m is responsible for smoothing the direction; the second moment v scales the update per parameter; and the bias correction avoids steps that are too small in the early stages of training. Because Adam normalizes gradients element by element, weight decay should not simply be mixed into the gradient. The common practice is decoupled weight decay.
5. Saddle Points, Learning Rate Scheduling, and Common Misconceptions
In high-dimensional non-convex problems, local minima are not always the main obstacle; saddle points are more common. The stochastic noise from mini-batch training acts like temperature: a high temperature makes it easier for parameters to cross narrow barriers, while a low temperature lets them settle. Learning rate scheduling is therefore essentially a temperature-control process: an early phase uses a higher learning rate and momentum to move quickly through flat regions and saddle points, while a later phase lowers the learning rate through means such as cosine annealing so the model settles into a flatter solution.
Supporting techniques include: warmup, which prevents divergence caused by unreliable early estimates; gradient clipping, which limits the drastic updates caused by anomalous batches; and excluding normalization-layer weights from weight decay, which avoids scale imbalance. Research on flat solutions also shows that methods such as weight averaging can improve stability at low cost.
Common misconceptions include: momentum is not specifically designed to escape local optima; adaptive optimizers do not guarantee convergence by nature; smaller models are better suited for early hyperparameter screening, while larger models are better suited for final validation.
6. Matrix-Level Updates: From Element-Wise to Structure-Aware
Traditional momentum and Adam process gradients at the level of a single parameter. But large-model parameters are often organized as matrices, such as attention projection matrices. Element-wise scaling ignores the correlation between rows and columns. Matrix-level methods try to approximate curvature information at a lower cost.
One line of thinking is matrix preconditioning, such as Shampoo, which estimates second-order statistics separately for the row space and column space. Another line is matrix-orthogonalized momentum, such as the publicly described Muon work, which performs approximate orthogonalization on the momentum matrix, retaining only directional information so that update magnitude is governed more by the learning rate. These methods have drawn attention in large-scale training, but they also introduce additional computation, communication, and parallel-sharding complexity. If matrices are sharded into very small pieces or degenerate into vectors, the advantage diminishes.
7. Comparing the Four Categories
| Method | Direction handling | Step-size mechanism | State overhead | Suitable scenario |
|---|---|---|---|---|
| SGD | Current gradient | Global learning rate | Low | Simple tasks, baseline comparisons |
| Momentum/Nesterov | Historical velocity smoothing | Global learning rate | Medium | Valley terrain, needs stable direction |
| Adam | First + second moments | Per-parameter adaptive | High | General starting point for large models |
| Matrix-level methods | Matrix-structure preconditioning/orthogonalization | Learning-rate-led or structure scaling | Medium-high | Large matrices, complex parallel scenarios |
8. From Optimizer to API Capability
An API upgrade is not the achievement of a single optimizer, but the result of a capability chain:
Data and training objectives → optimizer and parallel training → model capability → inference acceleration → API capability
On the training side, a more stable optimizer helps support larger scale, longer context, and more task data. On the inference side, quantization, KV caching, and continuous batching determine the cost of the same capability. On the engineering side, rate limiting, canary releases, observability, and version management determine whether capability can be delivered reliably.
To judge whether a piece of API documentation is “substantial,” you can check: whether the context and output length limits are clearly stated; whether structured output, streaming responses, and batch processing are supported; whether error codes, retries, and timeouts are clearly defined; and whether the version deprecation policy is transparent.
9. A Minimal Tuning Checklist
Before you begin, it is worth confirming at least the following items:
- Model scale: parameter count, number of layers, width, number of attention heads.
- Batch size: number of samples or tokens, which affects the learning rate and noise.
- Learning rate grid: first sweep 3 to 5 levels on a small model, then transfer to the large model.
- Second-moment coefficient: β2 can be tried at 0.999; large models often also try 0.95.
- Warmup steps: 1% to 10% of total steps, or a fixed few thousand steps.
- Gradient clipping threshold: commonly 1.0, adjusted based on gradient-norm monitoring.
- Update norm / parameter norm: a sustained abnormal increase usually signals a learning rate that is too high or a mismatch between layers.
In addition, resuming from a checkpoint must save the optimizer’s first- and second-moment states; otherwise the step size may shift abruptly after recovery. A reasonable default baseline is: adaptive optimizer + decoupled weight decay + warmup + cosine annealing.
10. Pseudocode Example
The following code only illustrates the logic and is not meant for production training.
from math import sqrtdef momentum_step(params, grads, velocity, lr, mu=0.9):# Momentum: compute the gradient at the current parameter point and combine it with historical velocityvelocity = [mu*v - lr*g for v, g in zip(velocity, grads)]new_params = [p + v for p, v in zip(params, velocity)]return new_params, velocitydef nesterov_step(params, grads_fn, velocity, lr, mu=0.9):# Nesterov: compute the gradient at the velocity-lookahead point, then updatelook_params = [p + mu*v for p, v in zip(params, velocity)]look_grads = grads_fn(look_params)velocity = [mu*v - lr*g for v, g in zip(velocity, look_grads)]new_params = [p + v for p, v in zip(params, velocity)]return new_params, velocitydef adam_step(params, grads, m, v, t, lr, beta1=0.9, beta2=0.999, eps=1e-8):t += 1m = [beta1*mi + (1-beta1)*g for mi, g in zip(m, grads)]v = [beta2*vi + (1-beta2)*g*g for vi, g in zip(v, grads)]m_hat = [mi/(1-beta1**t) for mi in m]v_hat = [vi/(1-beta2**t) for vi in v]new_params = [p - lr*mh/(sqrt(vh)+eps)for p, mh, vh in zip(params, m_hat, v_hat)]return new_params, m, v, t
No matter which category you choose, the core goal is the same: with limited compute, get the parameters into a low-loss, relatively flat region faster and more stably. Understanding the trade-off between step size and direction matters more than memorizing acronyms. Next time you read API documentation, beyond the feature list, it is worth thinking about the training and inference systems behind it.
