Optimizer Memory and Matrix Updates in Training Ultra-Large-Scale Models

    This article explains how memory is consumed during the training of ultra-large-scale models, with a focus on optimizer states. It compares two general approaches: element-wise adaptive updates based on first and second moments, and orthogonalized matrix updates that leverage weight matrix structure. The goal is not to declare one algorithm superior, but to build a practical framework for evaluating optimizers in terms of memory footprint, training speed, convergence quality, and engineering complexity.

    1. Before Evaluating a Model, Understand Its Training System

    When assessing a new generation of large models, common points of attention are parameter count, long-context capability, and task performance. Yet none of these metrics alone explains why a model can be trained efficiently, nor do they directly prove that a particular underlying technique has universal advantages.

    From the perspective of developers and enterprise users, more meaningful questions include: Given the same compute budget, can the model reach target quality faster? Is training stable? How much memory and communication overhead does a longer context add? Can saved training cost be converted into more thorough data learning and more reliable delivery?

    These questions involve the joint effects of model architecture, training data, numerical precision, the optimizer, and the distributed system. Among these, the optimizer is an often-underestimated component: it not only determines how parameters are updated, but also occupies a large amount of memory persistently and shapes the computation path of every training step.

    This article focuses on two general classes of methods: element-wise adaptive updates based on first and second moments, and orthogonalized updates that exploit the structure of weight matrices. The purpose is not to conclude that “one algorithm must win,” but to establish an analytical framework that can guide practical selection.

    2. What Actually Consumes Memory During Training

    When training ultra-large-scale models, counting only the weights is not enough. Memory is typically composed of:

    • Model weights;
    • Gradients produced by backpropagation;
    • Persistent optimizer states;
    • High-precision weight copies retained by some mixed-precision implementations;
    • Activations saved during the forward pass;
    • Temporary workspace, communication buffers, and memory fragmentation.

    Therefore, training memory can be summarized as:

    Training memory ≈ weights + gradients + optimizer states + high-precision copies + activations + temporary overhead.

    The most common misconception here is: low-precision weights do not mean all training states are stored in low precision. For numerical stability, some states may still be stored at higher precision.

    2.1 Why First and Second Moments Are Expensive

    Suppose the model has (P) parameters participating in optimization. If the optimizer stores one first moment and one second moment per parameter, each using 4 bytes per element, then these two states alone require:

    [
    M_{\text{states}} = P \times (4 + 4) = 8P \text{ bytes}
    ]

    For one trillion parameters, the theoretical storage for these two states is about 8 trillion bytes, roughly 8 TB, or about 7.3 TiB in binary units.

    This is only the states themselves, excluding weights, gradients, and activations. Real systems may also use state sharding, low-precision states, or offloading, so it is important to distinguish between “total global state size” and “the actual storage burden per device.”

    2.2 How Memory Pressure Propagates to Training Efficiency

    Optimizer states usually do not grow directly with sequence length, but they squeeze out the space available for activations. Long-sequence training may therefore require:

    • Reducing the per-device micro-batch size;
    • Increasing gradient accumulation steps;
    • Applying more activation recomputation;
    • Moving some states to other storage tiers;
    • Adding devices or adjusting the parallelism strategy.

    These measures may introduce additional computation, communication, or data movement costs. However, one cannot broadly claim that training cost will grow “exponentially.” The actual growth pattern depends on the attention implementation, sequence length, hardware utilization, and the parallelism scheme.

    3. Element-Wise Adaptive Updates: Benefits and Costs of Fine-Grained Control

    Adaptive methods based on first and second moments can be described by the following simplified formulas:

    [
    mt = \beta_1 m{t-1} + (1 - \beta_1) g_t
    ]

    [
    vt = \beta_2 v{t-1} + (1 - \beta_2) g_t^2
    ]

    [
    W_{t+1} = W_t - \eta \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
    ]

    Here, (g_t) is the current gradient, (m_t) smooths the update direction, (v_t) estimates gradient magnitude, and the hatted symbols denote bias-corrected quantities.

    Its key feature is element-wise adjustment of update scale: positions that consistently receive large gradients have their updates suppressed, while positions with small gradients may receive a relatively larger effective step size.

    This approach is highly general and backed by mature training experience. But it requires maintaining many states, and its handling of two-dimensional weight matrices happens mostly at the element level, without directly targeting the matrix’s singular directions as the object of the update.

    This leaves room for structured updates to be explored. However, “using matrix structure” does not automatically mean better; it still must be proven to offer a superior trade-off among quality, time, and resource consumption.

    4. Matrix Orthogonalized Updates: Reshaping the Scale Structure of Update Directions

    A large portion of parameters in neural networks naturally take matrix form, such as the linear projections in attention modules and the weights in feed-forward modules. For such parameters, one can consider treating the matrix formed by the gradient and momentum as a whole.

    Let the momentum update corresponding to a weight matrix be (G), with singular value decomposition:

    [
    G = U \Sigma V^\top
    ]

    An idealized structured transformation is:

    [
    Q = U V^\top
    ]

    The parameters are then updated using an appropriately scaled (Q):

    [
    W_{t+1} = W_t - \eta s Q
    ]

    where (s) is an implementation-dependent scaling coefficient.

    This transformation preserves the directional structure described by the singular vectors while renormalizing the scale of the nonzero singular values. Intuitively, it reduces the degree to which updates are dominated by a few strong directions, giving other directions relatively more opportunity to be updated.

    4.1 “Orthogonalization” Does Not Mean Every Matrix Becomes Square

    Neural network weights are often rectangular matrices. Strictly speaking, the result is usually a matrix with row-orthonormal or column-orthonormal properties, and rank-deficient cases require even more careful description. One should not simply assume that all cases satisfy full orthogonal matrix conditions.

    Practical implementations also often approximate this transformation iteratively rather than performing a full singular value decomposition at every step. Approximation error, iteration count, and computational precision all affect the final outcome.

    4.2 Memory Savings Come from Fewer States, Not from Orthogonalization Itself

    This is the most important distinction to understand about this method.

    If one matrix update scheme stores only a single momentum buffer while another must store both first and second moments, then—assuming identical state precision—the former eliminates one persistent state of the same scale as the parameters.

    But the orthogonalization computation itself also requires temporary space. It does not automatically compress weights, nor does it directly reduce activation size.

    Therefore, a more accurate conclusion is: certain matrix update schemes can reduce memory requirements by simplifying persistent states, but whether peak memory decreases—and by how much—still depends on temporary buffers and the distributed implementation.

    5. Fewer Training Steps Does Not Necessarily Mean Faster Training

    Let (N) be the number of training steps required to reach target quality, and (T) the average time per step. Total training time is approximately:

    [
    T_{\text{total}} = N \times T
    ]

    Matrix updates may reduce the number of steps needed while increasing the per-step cost of matrix operations and communication. Whether training ends up faster requires measuring both factors.

    5.1 Orthogonalization Introduces Extra Computation

    Approximate orthogonalization typically relies on multiple matrix multiplications. Whether these execute efficiently depends closely on matrix shapes, computational precision, and device utilization.

    If many parameter matrices are small or irregularly shaped, the costs of scheduling and temporary data handling may offset the theoretical gains. Conversely, well-shaped, compute-dense matrices have a better chance of exploiting hardware capabilities—but this still requires empirical measurement.

    5.2 Distributed Sharding May Change the Algorithm’s Semantics

    In multi-device training, a complete weight matrix may be sharded for storage. In that case, orthogonalizing each local shard independently is generally not equivalent to orthogonalizing the full matrix.

    If the algorithm requires the full matrix, extra aggregation communication may be incurred; if a local approximation is chosen instead, its impact on convergence must be validated.

    Therefore, an optimizer cannot be designed in isolation from the parameter sharding strategy. Where states are stored, where updates are computed, and when synchronization happens are all part of making the algorithm work in practice.

    6. Applicability Boundaries: Not All Parameters Should Use the Same Update

    A model contains more than two-dimensional hidden-layer weights; it may also include vector parameters, biases, normalization parameters, and word embedding matrices. Their shapes, gradient distributions, and functions differ.

    A reasonable engineering approach is to group parameters by type:

    • For two-dimensional weights suited to matrix operations, evaluate structured updates;
    • For vector and scalar parameters, use update methods appropriate to their shapes;
    • For input- and output-related matrices, separately validate update scale and training stability;
    • For parameters with special structures, design strategies based on how they actually participate in computation.

    This hybrid scheme does not imply that any algorithm is “incomplete”; it acknowledges that different parameters have different optimization needs.

    Note also that after changing the update geometry, existing configurations for learning rate, warmup length, weight decay, and gradient clipping may no longer be appropriate. Fair comparisons should grant both sides a reasonable tuning budget rather than forcing them to share the same configuration.

    7. Long Context and Sparse Architectures Require Separate Analysis

    Reducing optimizer states can indeed free memory, but this does not mean the model automatically gains long-context capability.

    Long-sequence training is also affected by positional representation, attention computation, activation storage, sequence parallelism, and training data coverage. For standard dense attention, computation grows quadratically with sequence length; with different implementations or structures, the specific costs change again.

    Therefore, three questions should be validated separately:

    1. Does the optimizer free enough memory?
    2. Can long-sequence training maintain acceptable throughput?
    3. Does the model truly benefit on tasks such as long-range retrieval and cross-segment reasoning?

    Similarly, sparsely activated models cannot be judged only by how many parameters are activated in each forward pass. Even if each step uses only a subset of parameters, the full model may still need to store a large number of weights and training states. Routing distributions, parameter update frequency, and load balancing further affect optimizer behavior.

    8. How to Design Credible Controlled Experiments

    When evaluating optimizers, it is recommended to place quality, resources, and stability in the same experimental table.

    8.1 Fix Key Experimental Conditions

    Keep model architecture, data mixture, training token count, effective batch size, numerical precision, hardware scale, and parallelism strategy as consistent as possible. If configuration changes are necessary, document the reasons clearly to avoid attributing all gains from system adjustments to the optimizer.

    8.2 Measure Quality and Time Together

    Observe at least:

    • Validation loss at the same amount of training data;
    • Model quality at the same wall-clock time;
    • Total time required to reach the preset target quality;
    • Task metrics relevant to the actual business.

    Comparing only training loss descent speed cannot rule out changes in generalization; comparing only throughput cannot demonstrate final training benefit.

    8.3 Break Down Memory and Performance

    Record the usage of weights, gradients, persistent states, activations, and temporary buffers separately, and distinguish steady-state phases from peak phases.

    Also break down the time spent on the forward pass, backward pass, optimizer update, and communication. Only then can you tell whether gains come from the algorithm itself, batch size changes, or adjustments to distributed execution.

    8.4 Validate Long-Running Behavior and Fault Recovery

    Short experiments are unlikely to expose every problem. Before formal adoption, also check:

    • Consistency of results across different random seeds;
    • Gradient anomalies and numerical overflow during long training runs;
    • Whether checkpoints completely save optimizer states;
    • Whether the loss curve remains continuous after resuming training;
    • Whether states can be correctly redistributed after changing the parallelism scale.

    If saving memory significantly increases the cost of troubleshooting or recovery, the overall benefit needs to be recalculated.

    9. Conclusion: Advantages Should Be Measured by Total Cost Under Quality Constraints

    Matrix orthogonalized updates offer a research direction worth studying: leveraging the matrix structure of weights to reshape update directions and, in some implementations, reduce persistent states. They may bring memory and convergence-efficiency benefits to ultra-large-scale training, but there is no universal conclusion independent of the model, data, and hardware conditions.

    The truly valuable criterion is not “more parameters,” “fewer states,” or “fewer steps,” but rather: given the target quality and stability requirements, does training use less total time, lower resource cost, and more manageable engineering complexity?

    For development teams, the most reliable path is to first build a memory ledger, then run representative controlled experiments, and finally scale up training. Only by accounting for both algorithmic gains and system costs can a local innovation be turned into a reproducible, maintainable training capability.