The Evolution of LLM Training Optimizers: Muon Principles and Usage

    Optimizers play a critical role in neural network training by dictating parameter updates, directly impacting model convergence and computational efficiency. Recently, Muon has emerged as a promising new optimizer. This article explores the core responsibilities of optimizers, traces their evolution from SGD to AdamW, and provides a detailed breakdown of Muon's working principles, implementation details, and best practices for applying it to large language model (LLM) training.

    An optimizer determines the direction and magnitude of parameter updates during model training. Its effectiveness directly impacts whether a model can converge stably and how much computational power the training process will consume. New optimizers like Muon have recently attracted significant attention. This article breaks down Muon from both theoretical and practical perspectives.

    1. The Two Core Responsibilities of Optimizers

    Training a neural network can be likened to searching for the lowest point in a highly uneven terrain. The loss function indicates the current elevation, while the gradient points in the steepest direction. The optimizer must decide the next step based on the gradient, primarily solving two problems.

    First, it must determine the step size. A step that is too large will cause parameters to oscillate around the minimum, preventing convergence. A step that is too small makes training extremely slow. The learning rate is the parameter that controls step size. In practice, it usually needs to be adjusted over the training process, such as through a warmup phase followed by gradual decay.

    Second, it must avoid getting stuck in unfavorable positions. High-dimensional terrains contain numerous local optima and saddle points. Simply following the gradient descent might halt progress in these areas. Optimizers require additional mechanisms to encourage parameters to keep moving toward better regions.

    2. From SGD to Momentum: The Early Evolution of Optimizers

    Stochastic Gradient Descent (SGD) is the most fundamental optimization method. It calculates gradients using mini-batches of data and updates parameters along the negative gradient direction. The concept of steepest descent dates back to Cauchy’s work in 1847.

    Standard gradient descent has a clear flaw: in narrow canyon-like terrains, the gradients along the canyon walls are very large. Parameters bounce back and forth between the walls, while progress along the canyon’s direction is extremely slow.

    In 1964, Soviet mathematician Boris Polyak introduced the Heavy Ball Method. This approach incorporates the movement direction of the previous step into the update, essentially adding inertia to the parameters. The oscillations along the canyon walls cancel each other out, while the components along the canyon’s direction accumulate, resulting in smoother progress. It is worth noting that momentum does not guarantee escaping all local optima; its primary function is to suppress oscillation and accelerate convergence.

    Subsequently, SGD with momentum and adaptive methods emerged. Adam maintains first-order and second-order moment statistics of gradients for each parameter, automatically adjusting the step size based on historical gradient magnitudes. Building upon Adam, AdamW separates weight decay from gradient updates, becoming one of the most widely used optimizers in large model training.

    3. How Muon Works

    Muon was publicly proposed by Keller Jordan and others in 2024. It improves the direction of parameter updates specifically for two-dimensional weight matrices within a neural network’s hidden layers.

    The basic steps of Muon are as follows:

    1. Calculate the gradient G for the current mini-batch data.
    2. Accumulate momentum: M = μ·M_prev + G, where μ is typically close to 0.95.
    3. Orthogonalize the momentum matrix M to obtain an approximate orthogonal matrix. If the Singular Value Decomposition (SVD) of M is M = USVᵀ, the orthogonalized result can be written as UVᵀ.
    4. Multiply this orthogonal matrix by the learning rate to update the parameters.

    There are two key takeaways here. First, the object of orthogonalization is the momentum matrix, not the raw gradient. Momentum first smooths the historical gradients, and then orthogonalization is applied. Second, orthogonalization makes the singular values of the update matrix close to 1. In raw gradients, a few directions often dominate the singular values, causing updates to concentrate in those specific directions. After orthogonalization, the update magnitudes across different directions become uniform, allowing the parameter matrix to be adjusted simultaneously across more dimensions.

    Because full SVD is computationally expensive, practical implementations typically use Newton-Schulz iteration to approximate the orthogonalization. This is a set of iterations consisting only of matrix multiplications, making it highly suitable for GPUs. Common implementations require only a few iterations (around 5 steps) to achieve a sufficiently good approximation. More iterations yield more precise orthogonalization but at the cost of greater computational overhead.

    It is important to emphasize that the base version of Muon does not maintain second-order moment statistics. Its difference from Adam lies in the geometric structure of the update, not in individually scaling each parameter.

    4. Applicability Boundaries: When to Use Muon vs. AdamW

    Muon’s design premise is that parameters are two-dimensional matrices suitable for matrix-level orthogonalization. In a typical large language model, parameters can be roughly categorized as follows:

    Parameter Type Typical Examples Common Choice Description
    2D Hidden Layer Weights Attention projections, MLP weights Muon Primary target for Muon
    Embedding and Output Layers Word embedding matrices, LM head AdamW Related to the discrete structure of the vocabulary; practitioners typically retain AdamW