How the Muon Orthogonalized Optimizer Cuts Memory Pressure in Trillion-Parameter Training

    A technical look at why optimizer states, rather than model weights, become the binding memory constraint at trillion-parameter scale — how to estimate AdamW's footprint with explicit accounting conventions, how Muon's single momentum state roughly halves optimizer-state memory while leaving total training memory well above the savings implied, and what teams must account for in practice: matrix-only applicability, Newton-Schulz iteration cost, mixed-optimizer strategies, hyperparameter re-tuning, numerical stability, monitoring, and rollback thresholds.

    1. Why Optimizer States Squeeze the Memory Budget in Trillion-Parameter Training

    Memory usage in very large model training is usually split across several components: model parameters, gradients, activations, temporary buffers, and the often underestimated optimizer states. Once parameters reach the hundreds of billions or the trillions, optimizer states stop being a “training detail” and start determining batch size, context length, parallelism strategy, and overall training cost.

    The core advantage of a conventional optimizer like AdamW is element-wise adaptive updating. It maintains a first moment and a second moment for every parameter, and scales the step size of each coordinate based on a running average of squared gradients. This is very stable at small scale, but at extreme parameter counts it also carries a heavy memory cost.

    Consider a trillion-scale model. If two fp32 states are kept per parameter, the optimizer alone can occupy anywhere from several terabytes to tens of terabytes. At that point the memory question is no longer “does the model fit,” but “do the optimizer states, the gradients, and the parameter copies fit together.”

    2. AdamW’s Memory Ledger: You Have to Fix the Accounting Convention First

    When discussing optimizer memory, saying “it uses a lot” is not enough. You need an explicit convention: parameter count N, number of states, precision of each state, whether parameter copies are included, whether gradients are included, and whether you are measuring a single device or the whole cluster.

    A common formula is:

    1. Optimizer state memory = parameter count N × number of states × bytes per state

    For AdamW, two states are typically maintained per parameter: the first moment m and the second moment v. With fp32 states:

    1. Optimizer state per parameter = 2 × 4 bytes = 8 bytes

    Assume a model with 1.6 trillion parameters, so N = 1.6 × 10^12:

    1. Optimizer states only = 8 × 1.6 × 10^12 bytes
    2. = 1.28 × 10^13 bytes
    3. ≈ 12.8 TB

    This figure counts only AdamW’s first and second moments. With mixed-precision training and additional fp32 master weights, bf16/fp16 parameter copies, and gradients, the total grows further. In a common configuration:

    1. Two fp32 AdamW states: 8 bytes/param
    2. fp32 master weights: 4 bytes/param
    3. bf16/fp16 gradients: 2 bytes/param
    4. Total: ≈ 14 bytes/param

    For 1.6 trillion parameters:

    1. 14 × 1.6 × 10^12 bytes ≈ 22.4 TB

    If gradients are also kept in fp32, or the implementation carries additional buffers, the number rises again. So the claim that AdamW “can swallow a dozen-plus terabytes” is not an exaggeration, but it must be stated as optimizer-state-only; if master weights, gradients, and temporary buffers are included, the accounting has to be redone.

    This also explains why optimizer states can become the bottleneck in trillion-parameter training. A single 80GB or 192GB GPU cannot hold a complete set of optimizer states on its own, and a single node with multiple GPUs may still fail to hold the full parameters, gradients, and optimizer states together. That is why techniques such as ZeRO, FSDP, tensor parallelism, pipeline parallelism, CPU offload, NVMe offload, and activation checkpointing are introduced to split or offload the memory pressure.

    3. How Muon Reduces Optimizer Memory

    Muon is a family of orthogonalized optimizer algorithms and implementations, most commonly applied to matrix-shaped parameter updates. Its core difference from AdamW is this: AdamW maintains first and second moments from an element-wise perspective, while Muon emphasizes treating a 2D weight matrix as a single whole.

    For 2D weight matrices in hidden layers, Muon typically keeps only a momentum state, then orthogonalizes — or approximately orthogonalizes — the matrix formed from momentum and gradient, producing an update direction that is applied to the weights in one step. Because it does not need to store a second moment per parameter the way AdamW does, Muon has clear potential to compress optimizer-state memory.

    Comparing optimizer states alone, assuming the same state precision:

    Optimizer States per parameter Bytes per parameter (fp32) Example with 1.6T parameters
    AdamW 2 states: m, v 8 bytes ≈ 12.8 TB
    Muon 1 state: momentum 4 bytes ≈ 6.4 TB

    Under this convention, Muon’s optimizer-state memory is roughly halved. The boundary matters though: this “halving” applies to optimizer states only, and excludes model parameters, gradients, fp32 master weights, temporary orthogonalization buffers, communication buffers, activations, and other memory.

    In real training runs, the benefit also depends on the mixing strategy. Many teams do not replace every parameter with Muon; they apply Muon only to 2D hidden-layer weight matrices and keep AdamW for embeddings, output layers, normalization parameters, and biases. As a result, the overall memory saving across the model is usually below 50%, depending on what fraction of parameters are matrices and whether the orthogonalization step introduces additional temporary buffers.

    Dimension AdamW Muon Notes
    Update perspective Element-wise adaptive Matrix-level update Muon treats the 2D weight matrix as a whole
    Primary state First moment, second moment Momentum One fewer state
    Optimizer state memory Higher Lower Roughly half at equal precision
    Temporary overhead Lower Orthogonalization iterations may create temporary matrices Must be included in engineering estimates
    Scope of applicability General Best suited to 2D matrices 1D parameters, embeddings need caution
    Hyperparameter transfer Mature Requires re-tuning AdamW settings cannot be carried over directly

    4. Muon Pseudocode and Implementation Notes

    The pseudocode below illustrates the idea and is not directly runnable. A real implementation has to be adapted to the specific Muon variant, parallel framework, precision strategy, and numerical stability approach.

    1. # Pseudocode, not directly runnable
    2. def muon_update(
    3. W, # 2D weight matrix
    4. grad, # Gradient matrix with the same shape as W
    5. momentum_buffer, # Momentum matrix with the same shape as W
    6. lr, # Learning rate
    7. beta=0.9, # Momentum coefficient
    8. ns_iters=5, # Number of Newton-Schulz iterations
    9. eps=1e-6 # Numerical stability term
    10. ):
    11. # Update the momentum; the exact form may vary between Muon implementations
    12. momentum_buffer = beta * momentum_buffer + (1 - beta) * grad
    13. # Approximate orthogonalization of the momentum, or of a
    14. # matrix formed by fusing momentum with the gradient.
    15. # eps can be added here to avoid division by zero or numerical instability
    16. O = newton_schulz(momentum_buffer + eps, iters=ns_iters)
    17. # Apply the orthogonalized update matrix to the weights
    18. W = W - lr * O
    19. return W, momentum_buffer

    A few parameters in this pseudocode deserve particular attention.

    First, the momentum coefficient controls how smooth the momentum is. It is similar to beta in Adam, but both the appropriate value and the update formula can differ by implementation. After switching optimizers, the AdamW momentum configuration cannot simply be reused.

    Second, the number of Newton-Schulz iterations is a trade-off. Too few iterations mean insufficient orthogonalization, and the update matrix may drift away from the intended geometric structure. Too many iterations add matrix multiplications and compute time, eating into part of the gains from the memory savings.

    Third, epsilon or eps is used for numerical stability. Orthogonalization involves matrix multiplications, normalization, sign handling, or inverse approximations, and extreme gradients, anomalous parameters, or low-precision computation can all cause instability. Reasonable precision control is therefore required.

    Fourth, Muon is generally better suited to 2D matrices. If a parameter is a 1D vector — bias, LayerNorm or RMSNorm parameters — or if it is a sparse embedding table, applying matrix orthogonalization updates directly may not be appropriate.

    5. Why Orthogonalized Updates Can Work

    Geometrically, a raw gradient matrix is often anisotropic: some directions carry larger values, others smaller ones. Updating directly along the raw gradient can move the model quickly in some directions and too slowly in others, sometimes producing oscillation.

    The goal of orthogonalization is to make the distribution of singular values of the update matrix more balanced, so that no single direction absorbs a disproportionate share of the update energy. Put plainly, it is not simple per-element rescaling; it is an attempt to make the update of the matrix as a whole smoother and more even.

    This can be understood as a matrix-level constraint on the update: when updating a weight matrix, what matters is not only how much each individual element changes, but also whether the scale of the transformation of the whole matrix stays stable. It complements AdamW’s per-coordinate normalization: AdamW tracks the historical gradient scale of each parameter individually, while Muon focuses more on the update structure of the 2D matrix as a whole.

    That said, orthogonalization does not mean “always better.” It rests on several premises: the parameters are suitable for matrix representation, the matrix shapes are reasonable, the numerical process is stable, and the iterative approximation is accurate enough. If the parameter distribution is unusual, the matrix is badly ill-conditioned, or the iteration count is too low, Muon’s benefits can be unstable and may even increase training variance.

    6. Longer-Tail Questions: What Muon Is, How to Choose, and Common Mistakes

    Question 1: What is Muon?

    Muon can be viewed as a family of specific orthogonalized optimizer algorithms and implementations, commonly used for matrix-level updates of matrix parameters. It is not a general term for all optimization methods; it refers specifically to a class of optimizers that generate update directions through orthogonalization or approximate orthogonalization.

    Question 2: How do you choose between Muon and AdamW?

    If the model contains many 2D hidden-layer weight matrices and the team wants to relieve memory pressure by reducing optimizer states, it makes sense to pilot Muon on the hidden-layer matrices first. If the model is small, parameters are dominated by embeddings or 1D vectors, or the team already has a mature AdamW training pipeline, blind replacement is not advisable.

    A common strategy is a hybrid optimizer: Muon for 2D hidden-layer matrices, AdamW for embeddings, output layers, normalization parameters, and biases. This captures memory advantages on the bulk of the parameters while preserving the stability of a mature optimizer elsewhere.

    Question 3: What mistakes are common with orthogonalized optimizers?

    Common mistakes include feeding 1D parameters into a matrix orthogonalization pipeline; ignoring the memory impact of temporary orthogonalization buffers; assuming Muon must halve total memory; reusing the AdamW learning rate without re-searching it; setting the Newton-Schulz iteration count poorly; lacking monitoring of gradient norms, loss spikes, and the singular value distribution of update matrices; and having no rollback mechanism in place.

    7. Engineering Rollout Checklist

    Check Recommended practice Acceptance criterion
    Memory formula Account separately for parameters, gradients, optimizer states, master weights, and temporary buffers Each category’s byte count and accounting convention can be stated explicitly
    Optimizer state comparison Compare AdamW’s two states against Muon’s one state Optimizer states only, excluding other memory
    Hybrid strategy Try Muon on 2D hidden-layer matrices, keep AdamW elsewhere No one-shot full replacement
    Throughput testing Fix model, data, and parallelism, then compare tokens/s, step time, and MFU Muon’s extra compute overhead is acceptable
    Hyperparameter transfer Re-search learning rate, momentum coefficient, weight decay, and update scaling AdamW hyperparameters are not reused
    Numerical stability Monitor NaN, Inf, gradient norm, parameter norm, and update matrix singular values No persistent anomalies
    Rollback thresholds Define trigger conditions for loss spikes, gradient explosions, and metric anomalies Automatically fall back to AdamW or lower the learning rate on anomaly

    Concretely, a team can start with small-scale alignment experiments and scale up gradually. First, compare Muon against AdamW on memory, throughput, and convergence curves with a mid-sized model. Second, confirm whether Muon is enabled only on 2D matrices. Third, re-run the memory accounting including temporary buffers and communication overhead. Fourth, use monitoring to continuously observe loss, gradient norm, the singular value distribution of update matrices, and parameter norm. Fifth, set explicit rollback thresholds so an experimental optimizer cannot trigger uncontrolled instability late in training.

    8. Practical Advice and Outlook

    For teams planning trillion-parameter or otherwise very large training runs, orthogonalized optimizers such as Muon offer a different line of thinking: instead of focusing only on element-wise adaptive scaling, look for more efficient updates in the matrix structure itself. Its value is not merely “storing one fewer state,” but pushing optimizer design from element-wise optimization toward matrix-level optimization.

    It still needs to be stressed that Muon is not a universal replacement. It suits parameters with an obvious matrix structure, 2D weights in particular, especially hidden-layer parameters; for embeddings, output layers, and normalization layers, a hybrid optimizer strategy is usually the safer choice. Meanwhile, the “memory halving” claim requires a strict accounting convention — savings in optimizer state cannot be equated directly with halving total training memory.

    Overall, Muon’s benefits include lower optimizer-state memory, a clearer matrix-level view of updates, and potentially better training stability on some tasks. The costs include extra orthogonalization compute, numerical tuning effort, hyperparameter migration effort, and engineering complexity. Whether to adopt it should be decided through the memory formula, throughput testing, convergence validation, and monitoring with rollback mechanisms taken together.

    As parameter counts keep pushing toward the trillions, whoever can sustain stable training with less memory buys room for larger batch sizes, longer contexts, and lower training cost. The orthogonalized optimizer direction that Muon represents is a technical path worth serious evaluation in this context.