Optimizer Memory Bottlenecks in Large Model Training: Element-wise Adaptive vs. Matrix-Level Updates
This article explains why optimizer states often become the first memory bottleneck in very large language model training, then compares element-wise adaptive updates with matrix-level momentum updates that use orthogonalization. It covers state size, update geometry, mixed-precision implications, parameter grouping, distributed consistency, checkpoint size, numerical monitoring, and a practical way to decide whether switching optimizers is worthwhile.
The training cost of very large language models is not determined by parameter count alone. Weights, activations, gradients, and optimizer states all occupy device memory at the same time, and optimizer states are often the first part to hit the limit. For element-wise adaptive methods, each trainable parameter usually requires a first moment and a second moment. If these states are stored in single precision, the two moment estimates alone amount to roughly eight bytes per parameter. If master weights are also kept in a high-precision copy, storage rises further. At this order of magnitude, models with hundreds of billions to trillions of parameters can require several terabytes to more than ten terabytes of optimizer state. What this pressure consumes is not an abstract “percentage of memory” but the feasible space for batch size, sequence length, checkpoint frequency, and parallel partitioning.
Therefore, before discussing whether a new optimizer is valuable, three verifiable questions should be asked: Are the state tensors significantly smaller than two full-size moments? Can the update still descend stably under ill-conditioned curvature? When model parallelism and data parallelism are both present, are the extra communication and numerical error acceptable? The comparison below uses these criteria to contrast element-wise adaptive updates with matrix-level momentum updates.
Why Element-wise Adaptive Updates Consume Memory
Common adaptive moment estimation treats each parameter as an independent coordinate. For gradient g, momentum m smooths first-order information, and the second moment v estimates the per-coordinate gradient scale. The update is roughly the bias-corrected m divided by the square root of v, multiplied by the learning rate. Its advantage is automatic scaling across parameters of different magnitudes, which is often stable for embedding tables, biases, and normalization parameters.
The cost also comes from “per-coordinate”: m and v have the same shape as the parameters and cannot simply be discarded without changing the algorithm. Mixed-precision training usually reduces the precision of activations and some weights in forward and backward passes, while optimizer states tend to remain in higher precision; otherwise, small gradients can be drowned out by rounding noise. This creates a mismatch: the compute bottleneck is in matrix multiplication, but the memory bottleneck is in two states that barely participate in the computation graph.
Memory pressure amplifies cost along three paths. First, batch size is forced down, gradient noise increases, and the same target loss may require more steps. Second, sequence length is reduced, so long-context capability can only be added after the fact, and the training distribution drifts from the target distribution. Third, to fit the states, parameters must be sharded more finely, collective communication increases, and overlapping compute with communication becomes harder. Comparing only “floating-point operations per step” underestimates this chain of effects.
Treating the Matrix as the Object of a Single Update
Another approach does not maintain a second-order scale for every element in hidden layers. Instead, it keeps momentum, treats the matrix formed by fusing momentum with the current gradient as a whole direction, orthogonalizes or approximately orthogonalizes it, and then adds it to the weights in one update. The intuitive role of orthogonalization is to suppress redundant directions in the update matrix so that different singular directions take steps of more similar magnitude, rather than letting a few large singular values dominate the whole matrix.
This geometry differs from element-wise scaling. Element-wise methods assign a separate step size to each coordinate and can adapt to extremely sparse coordinates or coordinates with very different scales. Matrix methods assume that the rows and columns of the same linear layer lie in a related subspace and that the update should preserve an overall rotational or semi-orthogonal structure. Hidden-layer weights are usually exactly this kind of dense matrix, and gradients often have low rank or fast spectral decay. As a result, “organize the direction first, then take a unified step” can sometimes use fewer states and fewer steps than “let every element move on its own.”
A common implementation approximates this by applying several polynomial iterations to the matrix, such as Newton-type iterations that approximate the orthogonal polar factor, rather than performing a full singular value decomposition every time. The number of iterations is an explicit hyperparameter: fewer steps give a coarser approximation and lower cost; more steps get closer to orthogonal but increase computation and temporary buffers. It should not be claimed that a fixed number of steps is optimal for every model. Instead, loss curves, gradient norms, and update norms should be observed on the target model and a proxy model.
The state-side benefit mainly comes from “no longer storing a full-size second moment.” If only momentum is kept, optimizer state can drop to roughly half the size of adaptive moment estimation, depending on momentum precision, whether master weights are retained, and which parameters still use the old optimizer. This benefit is structural; it does not mean end-to-end training will necessarily be faster. Orthogonalization itself must read the full block matrix and may add computation and temporary memory.
Applicability Boundaries: Not Every Parameter Should Enter a Square Matrix
Matrix-level updates have clear cases where they do not apply. Embedding tables are highly sparse: only a few rows receive gradients in a step, so treating the entire table as a dense geometric object both wastes computation and conflicts with update sparsity. One-dimensional biases and normalization scale and shift parameters have no matrix structure to orthogonalize. Output heads sometimes have far more rows than columns or the reverse, and their spectral distribution also differs from that of intermediate layers. In practice, a safer approach is a hybrid strategy: two-dimensional hidden-layer weights use matrix momentum updates, while embeddings, normalization parameters, and scalar parameters continue to use adaptive moment estimation with weight decay.
Rectangular matrices also cannot mechanically be treated with the language of “square-matrix orthogonalization.” Orthogonalization should act on the smaller dimension, or on a suitable factor through polar decomposition; otherwise, the iteration may amplify the condition number. In sharded training, a single device holds only one block of a matrix. If orthogonalization requires global rows or global columns, they must be aggregated before the update, or a partition-consistent approximation must be used. Ignoring this point can make the algorithm look effective on a single GPU but drift numerically once tensor parallelism is introduced.
Another common challenge is whether real gradients are always “neatly low-rank directions,” and whether forcing orthogonality loses adaptive capacity. This concern is valid in cases where gradient coordinate scales differ greatly, noise is heavy-tailed, or updates must remain extremely sparse. In those cases, the per-coordinate denominator provided by the second moment is still useful. Matrix methods are better suited to dense linear layers, and learning rate, momentum coefficient, and weight decay should remain independent knobs rather than assuming that orthogonalization can replace all preconditioning.
What to Check Before Engineering Deployment
Before deployment, validate the change in the following order rather than replacing the entire optimizer directly.
- Parameter grouping. List which tensors are two-dimensional and dense, and which must retain element-wise adaptivity. Incorrect grouping is a common cause of loss spikes.
- Precision and buffers. Specify the data types for momentum, master weights, and orthogonalization intermediates. Intermediates are often the hidden memory peak and must be counted, not just “state bytes.”
- Distributed consistency. Confirm whether the matrix blocks participating in orthogonalization are complete at each step. Communication should overlap with backward computation where possible. After changing partitioning, retest; do not reuse single-machine hyperparameters.
- Numerical monitoring. Record the spectral norm or Frobenius norm of the update matrix, the orthogonalization residual, and the loss after grouping. If the residual does not decrease over time, it usually indicates too few iterations or too large a learning rate.
- Checkpoint size. Fewer states reduce checkpoint size, which is a real benefit. At the same time, ensure that grouping rules and shapes match exactly when resuming training, to avoid silently using the wrong optimizer.
- Proxy tasks. First align the target loss on a model with the same structure but reduced width and depth, then scale up. Looking only at the first few hundred steps can easily overestimate later stability.
An illustrative grouping logic can be written as follows. It expresses only the control flow and does not correspond to any framework-specific interface:
def param_groups(module):matrix, rest = [], []for name, p in module.named_parameters():if not p.requires_grad:continueif p.ndim == 2 and "embed" not in name and "head" not in name:matrix.append(p)else:rest.append(p)return matrix, rest
Weight decay should also be understood separately. If a matrix update already fixes the scale of the direction through orthogonalization, the decay term still controls the weight norm; it should not be assumed that “orthogonal updates provide regularization by themselves.” The interaction between learning rate and momentum is also sensitive: higher momentum makes the historical direction used by orthogonalization smoother, but it responds more slowly to distribution drift.
How to Decide Whether Switching Is Worthwhile
Use a simple accounting exercise rather than a slogan. Let the number of parameters be N. Two single-precision states for adaptive moment estimation are about 8N bytes, while a matrix scheme that keeps only one momentum is about 4N bytes, a difference of 4N. Then subtract the peak orthogonalization buffers and the device occupancy equivalent of extra communication time. If the difference is enough to raise batch size or sequence length into a range that clearly improves sample efficiency, there is an engineering reason to switch. If the model is far smaller than single-device memory, or the bottleneck is data loading and activation recomputation, the benefit will be drowned out by noise.
On the convergence side, look at “steps to reach the same validation loss × time per step,” not steps alone. Orthogonalization may reduce the number of steps, but each step is heavier. The higher the proportion of hidden layers and the larger the matrices, the easier it is to amortize the cost; with many small matrix layers, call overhead becomes less worthwhile.
Conclusion
The optimizer memory problem is fundamentally a conflict between “saving adaptive statistics for every coordinate” and “model width having grown so large that those statistics no longer fit.” Treating dense weight matrices as a single geometric update, using momentum plus orthogonalization instead of a full-size element-wise second moment, is a viable path to reducing state and improving redundancy in update directions. However, it should be limited to parameters with matching structure and used alongside traditional adaptive methods for embeddings and normalization parameters. Evaluate state bytes, peak buffers, communication scope, and wall-clock time to target loss together; only then can you tell whether the holistic update is genuinely improving training efficiency or merely moving overhead from memory to computation and communication.
