Matrix-Based Optimizers: Breaking the GPU Memory Bottleneck in LLM Training
As model sizes climb toward the trillion-parameter range, training memory—not raw compute—has become the hard ceiling. This article breaks down why adaptive optimizers are so expensive in GPU memory (roughly 8 bytes per parameter just for the two optimizer states, or about 12.8 TB for a 1.6-trillion-parameter dense model), and explains matrix-based optimizers as an alternative: instead of tuning every element individually, they update the entire weight matrix in one shot by orthogonalizing the accumulated momentum. You will learn how the memory savings actually come about, where this approach breaks down (embedding tables, 1D parameters, norm layers, hyperparameter retuning, extra compute), and how to adopt it layer by layer alongside optimizer state sharding and careful tuning.
The optimizer memory problem in large-model training
With parameter counts heading toward the trillion scale, training cost is no longer about compute alone. Memory is the real bottleneck. Many teams find that they cannot push GPU utilization up, that batch size is forced down, that context length stays short — and the culprit is often not the model itself, but the optimizer quietly consuming most of the memory budget.
To see why, it helps to look at the memory bill of the mainstream adaptive optimizers.
Where the memory goes: why adaptive optimizers are expensive
Take the most widely used adaptive optimizer in the industry. To apply a form of “individual treatment” to each parameter, it maintains two extra states per weight:
- a first moment: a running average of the gradient, which captures the direction of the update
- a second moment: a running average of the squared gradient, which captures how active the parameter is and its overall scale
Both are typically stored in single precision to keep the numerics stable.
The bill follows directly. Those two states alone cost about 8 bytes per parameter (two 4-byte values). For a 1.6-trillion-parameter dense model, the optimizer state alone consumes roughly 8 × 1.6×10¹² ≈ 12.8 TB of memory — and that is before gradients, master weight copies, and activations.
Since total memory is fixed, whatever the optimizer takes is space unavailable for larger batch sizes or longer contexts. Teams end up shrinking batches, lengthening gradient accumulation, or increasing communication frequency — and throughput and cost both suffer.
A different perspective: from element-wise to whole matrix
The optimizer described above works element-wise. It decomposes a weight matrix into individual scalars, computes an adaptive step size for each element independently, and pays no attention to the matrix as a geometric object.
Another family of optimizers treats the entire weight matrix as a single indivisible unit. Its view: an m×n weight matrix is a high-dimensional geometric object, and updating it should not mean nudging one point at a time — instead, one whole-block update is computed and added to the weights.
That shift in perspective is the key to understanding matrix-based optimizers.
Orthogonalized momentum: mechanics, intuition, and the marching formation
The typical approach works like this:
- Accumulate the gradients into a momentum matrix.
- Orthogonalize that matrix — through a few approximate iterations (such as Newton–Schulz iterations, which use only matrix multiplications and therefore map naturally onto GPUs), its columns become approximately orthogonal and its singular values move toward being uniform.
- Multiply the orthogonalized matrix by the learning rate and add the result directly to the weights.
Why is orthogonalization useful? It flattens the magnitude of the update across all directions (singular vectors) and prevents the update from being dominated by a few high-curvature directions. That makes updates across layers and directions more balanced and more stable.
An intuitive analogy: imagine a crowd of iron balls rolling down a slope. An element-wise optimizer lets each ball go its own way, prescribing a separate rolling rhythm for every single ball — precise, but it has to store a separate state for each one. A matrix-based optimizer first arranges the balls into a square formation and lets them advance as one block in step. It only needs to remember the formation (the momentum), not an independent state per ball. One is a crowd of soldiers scattering in different directions; the other is a phalanx marching in unison. The difference in coordination overhead is obvious.
Where the memory savings actually come from
The savings come from one specific change: the optimizer only needs to maintain a single momentum buffer, with no second moment per parameter. Against the two-state element-wise approach, that roughly halves the optimizer state. Orthogonalization itself is computed on the fly through iterations, so it adds almost no resident memory.
To be precise about what is and is not saved: what shrinks is the optimizer state. Gradients and the weights themselves still occupy memory as usual. But the optimizer state is often the dominant term, and cutting it in half frees up a meaningful amount — enough for larger batch sizes or longer contexts. That is precisely what makes this approach attractive for training very large models.
“Phalanxes don’t fight battles” — legitimate objections
A tidy marching formation sounds appealing, but it invites an obvious objection: nobody keeps ranks on a battlefield; you would want skirmish lines and flexible maneuver. For matrix-based optimizers, that objection is fair, and the following points have to be faced in practice:
- Limited applicability. Orthogonalization mainly applies to the 2D weight matrices in hidden layers. It is not appropriate to force it onto embedding tables (often very large and sparse), biases (1D vectors), or normalization parameters. Those parts usually stay with a conventional optimizer.
- Hyperparameters need retuning. Learning rates and momentum coefficients have different scales and different behavior than in element-wise schemes. After switching, you have to redo the learning rate search and warmup; old configurations cannot be carried over directly.
- Extra computation. Each update requires several additional matrix iterations. The share is modest, but it still has to be weighed when pushing for peak throughput.
- Convergence theory is still developing. Compared with the mature convergence analysis of adaptive optimizers, matrix-based methods are currently more empirical, with evidence accumulating quickly. Selection should rest on measured results for the task at hand.
Engineering it: when and how to use it
A few practical points:
- Select per layer. Enable the matrix-based optimizer for 2D hidden-layer weight matrices and keep a mature adaptive optimizer for everything else (embeddings, biases, normalization, output head). This often gets you both memory savings and stability.
- Separate state precision from weight precision. Keep optimizer state in single precision for stable updates while weights and activations can run in lower precision, balancing numerical stability against memory efficiency.
- Coordinate with the sharding strategy. Sharding optimizer state across the ranks of data-parallel groups (a common industry approach) spreads the per-device memory pressure further.
- Switch gradually and monitor. In the early phase, watch the loss curve, gradient norms, and update magnitudes closely; scale up only after confirming stability.
Pseudocode for the two update units side by side:
# Element-wise adaptive update (keeps two states: first moment m, second moment v)m = b1 * m + (1 - b1) * gv = b2 * v + (1 - b2) * (g * g)m_hat = m / (1 - b1 ** t)v_hat = v / (1 - b2 ** t)W = W - lr * m_hat / (sqrt(v_hat) + eps)# Matrix-based orthogonalized update (keeps one momentum state M)M = beta * M + (1 - beta) * gO = newton_schulz(M, steps=5) # approximate orthogonalization; singular values trend toward uniformW = W - lr * O # update the whole matrix in one shot
The contrast is direct: the first maintains two states per element, the second maintains a single momentum buffer and updates the whole matrix at once. That is the source of the memory savings.
Summary
The value of matrix-based optimizers is not novelty. It is a change of perspective: treat the weight matrix as a whole, use orthogonalization to flatten update magnitudes, and compress the optimizer state from two buffers down to one. For large-model training where memory is tight and larger batches or longer contexts are the goal, this is a practical memory optimization path.
But it works as layered cooperation, not wholesale replacement: the matrix-based method for hidden layers, a mature adaptive optimizer for the remaining parameters, plus state sharding and careful tuning — that is what gets you both lower memory use and stable convergence. Understanding its applicability limits and costs matters more than following the trend.
