Matrix Orthogonalization for Large-Scale Model Training: Cutting Optimizer Memory Costs

    In trillion-parameter training, optimizer states are a silent but dominant memory cost—often exceeding 10 TB on their own. This article explains how matrix-level update optimizers replace per-element second-moment estimation with orthogonalization, cutting optimizer-state memory roughly in half while often reducing the number of training steps needed. It walks through the core ideas (the optimal-transport and SVD views of orthogonalization), practical approximate orthogonalization via matrix multiplication, the ongoing debate over whether small-scale results generalize to industrial scale, a pragmatic hybrid strategy that groups parameters by role, and the engineering practices—memory budgeting, state sharding, low-precision storage, smooth optimizer switching, and observability—needed to make these optimizers work reliably at scale.

    1. The Trillion-Parameter Memory Bill: Optimizer States Are “Eating” Over 10 TB

    In large-scale model training, engineers tend to focus on parameter count, context length, and cluster size, while underestimating a “silent consumer”: optimizer-state memory.

    GPU memory usage during training can be broken down into roughly five parts:

    • The parameters themselves;
    • Gradients;
    • Optimizer states;
    • Activations produced during the forward pass;
    • Various temporary and communication buffers.

    Among these, optimizer states take part in no matrix computation, yet they must remain resident in memory the entire time and grow linearly with parameter count.

    Take the element-wise adaptive optimizer used most widely in industry. This class of optimizers maintains two extra states per parameter: the first moment of the gradient (momentum) and the second moment (a moving average of the squared gradient). This is what lets them compute an adaptive step size for each parameter individually—which is exactly why they converge stably and are relatively easy to tune. But the cost is that every parameter requires two additional memory allocations.

    Suppose we train a very large model with about 1.6 trillion parameters. Under mixed-precision training, we can estimate roughly:

    Memory item Precision Bytes per parameter Total for 1.6T parameters
    Master weights (FP32) 4 bytes 4 ~6.4 TB
    Gradients 2 bytes 2 ~3.2 TB
    Optimizer states (first + second moment) 4 bytes 8 ~12.8 TB
    Forward-pass weight copy 2 bytes 2 ~3.2 TB
    Total (excluding activations and temporary buffers) — 16 ~25.6 TB

    The optimizer states alone amount to storing an entire set of about 1.6 trillion “intermediate results”—on the order of the combined memory of more than a hundred 80 GB accelerators.

    It’s worth noting that optimizer states are hard to “discount” via low precision:

    1. The second moment is essentially a moving average of squared gradients; its numerical scale keeps changing throughout training, and insufficient precision undermines the stability of step-size estimation;
    2. Even dropping to half precision only compresses 12.8 TB down to 6.4 TB—optimizer states would still be the single largest memory consumer.

    When memory is squeezed, the direct consequence is that less room is left for critical settings like batch size, context length, and gradient checkpointing, which in turn pushes up the overall cost curve of training.

    Of course, total memory isn’t the only bottleneck. That total is usually spread across hundreds or thousands of devices via sharding-based parallelism; the tighter constraint more often appears at the single-card level: each accelerator must hold its own rank’s share of parameters, gradients, and optimizer states, while also accommodating local activations and communication buffers.

    So reducing optimizer-state memory has direct value—both for the overall memory budget and for single-card feasibility. The question then becomes: can we design an optimizer that preserves or even improves convergence efficiency while cutting one of these two states?

    2. An “Extreme Compute-Saving” Challenge: A New Direction for Optimizers

    At the end of 2024, a senior machine-learning researcher launched a community “extreme compute-saving” racing challenge. The rules were simple: given a model architecture, whoever trains a small ~124-million-parameter language model to a target loss value with the least compute and time wins. To compress the gap between entries, the competition later added constraints on training steps and training time.

    These challenges demand near-brutal training efficiency; competing solutions typically have to push hard on every front at once—rewriting the computation graph, loss-function design, initialization, data layout, and low-precision training.

    Faced with the problem, most participants’ instinct was to “squeeze the element-wise adaptive optimizer dry”: carefully tuning the decay coefficients of the first and second moments, redesigning the learning-rate schedule, and pushing each step’s update magnitude to the limit. That approach does work, but after years of refinement, its room for improvement is already quite limited.

    One independent developer took a different path. In code submitted to the community, they implemented and disclosed a new optimizer for the first time, whose core idea is a single sentence: orthogonalize the gradient of the hidden-layer weight matrices (more precisely, the matrix formed by fusing momentum and gradient), then apply it to the weights as a whole.

    They observed that, at equal compute, models trained with this kind of update saw a substantial drop in the number of training steps, and on that basis set a new racing record at the time. The result quickly drew attention in the open-source community—and naturally raised a question: could it hold up under larger-scale training?

    3. From “Element-Wise” to “Matrix as a Whole”: The Core Idea of Matrix-Level Orthogonalized Updates

    To understand matrix-level update optimizers, it helps to first see clearly what element-wise adaptive optimizers are actually doing.

    3.1 The Element-Wise View: Computing a Step Size for Each “Iron Ball”

    The update of an element-wise adaptive optimizer can be summarized in three steps:

    1. First moment: an exponential moving average of the gradient, yielding a smoother descent direction;
    2. Second moment: an exponential moving average of the squared gradient, estimating how steep the loss is around each parameter;
    3. Element-wise scaling: dividing the first moment by the square root of the second moment to set the step size for each parameter individually.

    Picture a pile of iron balls rolling down a slope, each with a different initial speed and encountering a different slope along the way:

    • The optimizer must decide, separately for every ball, “how far should it roll on the next step?”;
    • To do so, it must maintain two memories for each ball, recording its prior speed and slope;
    • When the number of balls reaches the trillions, these memories become a heavy burden in GPU memory.

    3.2 The Matrix-as-a-Whole View: One Update Is a Single Step in Matrix Space

    Matrix-level update optimizers take a different perspective: a 1000×1000 weight matrix is essentially an indivisible high-dimensional geometric space.

    • One update to it is moving the matrix a certain distance along some direction in that space;
    • Under this view, the direction should be determined by the matrix’s momentum, and the update should be applied to the whole matrix as a unit, with no need to compute a step size for each element individually.

    Back to the iron-balls-rolling-downhill analogy:

    • First, arrange all the balls into a neat square formation;
    • Then move the whole formation at a uniform rhythm in the same direction.

    The way of commanding them looks simple, but the balls no longer act on their own—and that is precisely the source of the large drop in memory overhead.

    But the formation isn’t something you can just assemble casually. The magnitudes of the momentum matrix’s elements vary widely: some directions have large values, others near zero, and using it directly as the update could further amplify this imbalance. The matrix-level optimizer’s approach is to orthogonalize the matrix before applying the update.

    1) Viewing orthogonalization through optimal transport

    From an optimal-transport perspective, the “least-effort” transformation between two matrices is essentially a rotation (or reflection) plus a single overall scaling. An orthogonal matrix is exactly this kind of pure rotation. Orthogonalizing the momentum matrix is like “tidying” a complex, imbalanced update into a single uniform rotation. This way, no direction gets over-stretched while others stay almost in place.

    2) Viewing orthogonalization through singular value decomposition (SVD)

    It’s more intuitive to look at it through SVD. For any matrix M, the decomposition is:

    M = U Σ Vᵀ

    Here U and V are orthogonal matrices, and Σ is a diagonal matrix whose diagonal entries—the singular values σ₁, σ₂, …—measure how much M “stretches” along different directions.

    • Using M directly as the update, each direction is stretched to a different degree depending on the singular-value magnitude, and the large-singular-value directions dominate the update;
    • Orthogonalization can be seen as uniformly setting all non-zero singular values to 1, yielding U Vᵀ (retaining the directions corresponding to zero singular values when the matrix is rank-deficient).

    As a result, the update’s effect becomes balanced across directions—essentially replacing per-element second-moment estimation with a matrix-level constraint.

    3) The root of the memory overhead

    For a 2D weight matrix, this “treat-as-a-whole” design means:

    • Only one matrix-shaped momentum state needs to be maintained;
    • Compared with the two states (first and second moment), the optimizer-state memory is cut in half directly.

    3.3 Approximate Orthogonalization: Matrix Multiplication Instead of Decomposition

    Exact polar decomposition or SVD is expensive and numerically sensitive, making it hard to embed in the parameter update at every step. An implementation approach widely validated in the racing community is to use low-order polynomial iteration to approximate the orthogonalization factor, with the five-step third-order iteration being the most common. Just a few matrix multiplications are enough to push a matrix toward orthogonality—a very good fit for hardware whose core compute is matrix multiplication:

    1. # Illustrative pseudocode, only to demonstrate the orthogonalization idea
    2. def orthogonalize(G, steps=5, eps=1e-7):
    3. # G is the momentum matrix (transpose first when rows << columns, to shrink intermediate matrices)
    4. X = G / (G.norm() + eps) # Scale by the overall norm so the initial value falls in the convergence range
    5. for _ in range(steps):
    6. A = X @ X.T # Compute the Gram matrix
    7. X = 3.4445 * X + A @ (-4.7750 * X + 2.0315 * A @ X)
    8. return X

    This approximate orthogonalization also brings a “by-product”: the maximum singular value of the output matrix is kept near 1, meaning the update magnitude is naturally bounded—effectively acting as implicit gradient clipping, which helps training stability.

    Orthogonalizing the gradient (or momentum) and then applying it to the weights scaled by the learning rate constitutes one complete matrix-level update. Because this replaces the traditional per-element state maintenance, the formula details differ across implementations (the form of the momentum update, the choice of learning rate, etc.). And since it dispenses with per-element scaling, these optimizers usually don’t pair directly with traditional weight decay—regularization practices likewise vary by implementation.

    4. Debate and Validation: Effective at Small Scale—but Does It Extrapolate to Large Scale?

    When the matrix-level update optimizer first appeared, skepticism followed: “Who charges into battle in neat formation? You advance in skirmish lines.” This method was about to face the same question: it was initially validated only on a small language model at the 124-million-parameter scale—looking orderly and uniform—but does it really hold up at industrial scale?

    This isn’t nitpicking. On small models, training steps are relatively limited and data repetition can be high, so an optimizer’s “clever tricks” are easy to amplify. In trillion-parameter training, model scale, data scale, and parallelism strategy all change at once, and any trick that works at small scale must be re-validated.

    Pulling together public community discussion, the key points of debate reduce to three:

    1. Generalization: whether the advantage holds when parameters exceed the billion level and beyond;
    2. Hyperparameter transferability: whether configurations tuned on small models, such as the learning rate, can carry over to large models;
    3. Engineering stability: whether numerical problems or efficiency regressions arise under extreme scale, mixed precision, and complex parallel topologies.

    The community subsequently reported a series of attempts to apply such optimizers to training larger language models. According to public technical write-ups, matrix-level update optimizers have been extended to larger parameter scales and applied in the training of some models, with generally robust behavior reported. But this is still community-level public feedback, and it needs more independent, reproducible benchmarks to substantiate it.

    4.1 A Pragmatic Engineering Compromise: Hybrid Parameter Grouping

    It’s worth noting that Muon-style optimization packages often provide hybrid optimizers that combine the new optimizer with an existing adaptive optimizer. For example, some optimizer libraries offer a hybrid version that combines the matrix-level update with a mature element-wise adaptive optimizer, managing parameter groups through a unified interface:

    • Hidden-layer 2D weight matrices are handed to the matrix-level update;
    • 1D parameters (such as normalization-layer scales and biases), along with embedding and output layers, are handled by the element-wise adaptive optimizer.

    The idea behind this compromise is to decide which update rule to use based on a parameter’s “role” rather than its “count.” It lets the matrix bulk enjoy the efficiency of the matrix-level update while preserving the proven behavior of adaptive updates on special parameters. In practice, you can borrow the following illustration:

    1. # Illustration: group by parameter role and match different update strategies
    2. groups = [
    3. {"params": hidden_matrix_params, "optimizer": "matrix"}, # hidden-layer weight matrices
    4. {"params": other_params, "optimizer": "elementwise"} # normalization, embeddings, output, etc.
    5. ]

    5. Engineering Practice: Memory Optimization Can’t Rely on “Swapping the Optimizer” Alone

    The value of matrix-level update optimizers goes beyond saving memory: reducing the number of training steps needed at equal compute means shorter wall-clock time and lower compute cost. But no single technique is enough to solve the memory problem in large-scale training—it has to be combined with other engineering measures.

    1. Do a memory budget first. List parameters, gradients, optimizer states, activations, and temporary buffers as separate line items. Only once you see clearly which item is the bottleneck can you decide whether to prioritize switching optimizers or adjusting the parallelism/splitting strategy.

    2. Shard the optimizer states. The half of the state memory saved by the matrix-level update optimizer doesn’t mean sharding is no longer needed. Rather, at the same cluster size, each card can leave more headroom for activations and communication buffers, creating conditions for larger batches and longer contexts.

    3. Low-precision storage. Storing optimizer states in 16-bit and computing in higher precision is a common way to further compress memory usage. Watch out for the precision trade-off of the momentum state, and use monitoring to catch numerical anomalies promptly.

    4. Smooth switching. When replacing the optimizer mid-training, the original optimizer’s state can’t be directly reused by the new one. The safe approach is to switch at phase boundaries, reset the scheduler, and set a brief learning-rate warmup to avoid obvious jitter in the loss curve.

    5. Observability. Add monitoring and alerting for key training metrics, at minimum including:

      • Peak memory per card;
      • Time per step;
      • Gradient norm;
      • Norm of the update matrix.

      Once memory approaches its ceiling or the loss curve behaves abnormally, you can quickly localize whether it’s a data, parallelism-configuration, or numerical-precision issue.

    6. Validate before scaling up. A new optimizer strategy should first be benchmarked against a baseline on small- to medium-scale models to confirm the direction of change in the loss curve at equal compute, and only then scaled up gradually to the target size—rather than being applied directly to ultra-large-scale training.

    6. Conclusion: Formation or Skirmish Line—Scale Is the Final Judge

    The reason the matrix-level update optimizer stood out from a small racing competition is that it dared to break out of the “element-wise adaptive” mindset:

    • Treating the weight matrix as a whole;
    • Replacing second-moment estimation with orthogonalization;
    • Keeping only one momentum state, halving the optimizer-state memory;
    • Trading “queuing up as a whole” for a substantial drop in training steps.

    Still, back to the “should the battlefield use formation or skirmish lines” debate, the answer may lie not in the form itself but in the scenario. In large-scale training, the constraints of memory, compute, and time exist simultaneously, and any optimization technique must be validated at the target scale.

    For engineering teams, the rational approach isn’t to choose between the two optimizers, but to divide labor by role, use them in combination, and hold the stability line with thorough observability. It’s foreseeable that coordinated optimization of optimizers and parallelism strategies will remain an important direction for cutting costs and improving efficiency in large-scale training.