MLA Attention and the Muon Optimizer: A Practical Deep Dive into Training Large Language Models

    This article breaks down two techniques that address the core bottlenecks of large language model training: MLA (Multi-latent Attention), which compresses the KV cache to cut memory use, and the Muon optimizer, which applies matrix orthogonalization for more balanced parameter updates. It explains how each technique works, where it delivers gains, and the pitfalls to watch for when combining them in practice.

    Introduction: The Dual Bottlenecks of Large Model Training

    When training and serving large language models, engineering teams routinely face two core constraints: memory footprint and convergence efficiency. The former determines how long a context a model can support and how large a batch you can run; the latter determines whether you can reach a high-quality solution quickly within a limited compute budget. Recent technical reports from a new generation of open-source large models highlight two components in particular — an attention mechanism that compresses the key-value cache (MLA, Multi-latent Attention), and an optimizer based on matrix orthogonalization (Muon). Setting aside any specific product, this article starts from general technical principles and explains why these two techniques are useful, how they work, and where the pitfalls lie in practice.

    Part 1: MLA — Leveraging Low-Rank Compression to Break the Memory Bottleneck

    1.1 Why Standard Attention Is Memory-Hungry

    Standard multi-head attention must cache the Key and Value tensors for every historical token during autoregressive generation. This is known as the KV cache. Its size scales with “number of layers × number of heads × head dimension × sequence length.” When the context grows from a few thousand to hundreds of thousands of tokens, the KV cache expands linearly and becomes the dominant memory cost, directly slowing inference throughput.

    1.2 The Core Idea: Joint Low-Rank Compression

    The key insight behind MLA is this: instead of storing the full K and V for each head separately, first project each token’s representation into a shared low-dimensional latent vector, and cache only this compressed vector. During inference, up-projection matrices reconstruct the K and V that each head needs. Because the compression dimension is far smaller than “number of heads × head dimension,” the memory footprint of the KV cache drops substantially.

    The following code sketch captures the core data flow:

    1. # Simplified illustration of MLA forward pass (pseudocode, not a real implementation)
    2. # Jointly compress input x into a low-dimensional latent vector c; only cache c
    3. c = x @ W_dkv # low-rank down-projection: d_model -> d_latent (d_latent is far smaller than the total head dimension)
    4. k_content = c @ W_uk # up-projection reconstructs the content part of Key
    5. v = c @ W_uv # up-projection reconstructs Value

    During training, the up-projection matrices can also be combined with the down-projection through matrix absorption, folding the equivalent transformation into adjacent weights. This lets the forward and backward passes complete without explicitly materializing large tensors, further improving compute and memory efficiency.

    1.3 The Coupling Problem with Positional Encoding and the Decoupling Solution

    Low-rank compression is not free. Rotary Position Embedding (RoPE) is position-dependent: it applies a position-related rotation to the Key, an operation that interferes with the compressible structure of the content part and is hard to “absorb” into a fixed low-rank projection.

    The industry commonly resolves this with decoupled RoPE: attention is split into two paths — a “content channel” that carries semantics and can be low-rank compressed, and a “position channel” with only a small number of dimensions dedicated to carrying positional information. The two are concatenated before attention scoring. This preserves both the memory savings from low-rank compression and the ability to model long-range positions.

    1. # Illustration of decoupled positional encoding
    2. k_rope = x @ W_rope # only a few dimensions take part in rotary positional encoding
    3. q = concat(q_content, q_rope) # query concatenates the content channel and the position channel
    4. k = concat(k_content, k_rope) # key concatenates likewise
    5. scores = softmax(q @ k.T / sqrt(d))

    1.4 Inference Gains in Long-Context Scenarios

    Precisely because the KV cache is significantly compressed, MLA offers two advantages for long-text workloads. First, the per-token memory cost drops, making larger batches or longer contexts possible. Second, the volume of data read from and written to the cache shrinks, easing the memory-bandwidth bottleneck in inference and yielding better efficiency in high-throughput scenarios. It’s worth emphasizing that the concrete gains correlate strongly with model size, sequence length, and hardware bandwidth, so real measurements should be the reference rather than absolute numbers quoted out of context.

    Part 2: The Muon Optimizer — Orthogonalized Updates in Matrix Space

    2.1 The Evolution from SGD to Adam

    Stochastic Gradient Descent (SGD) and its momentum variants are simple and efficient, but they are sensitive to the learning rate and gradient scale. Adam applies per-element adaptive scaling using first- and second-order momentum; it is robust, easy to tune, and has long been the mainstream choice for training large models. However, Adam’s processing is per-element, ignoring the structural information of a weight matrix as a linear transformation taken as a whole.

    2.2 The Core: Orthogonalizing the Momentum

    Muon (roughly, “momentum after Newton–Schulz orthogonalization”) takes a different view: it treats a two-dimensional weight matrix as a linear operator and argues that its ideal update direction should be close to an orthogonal matrix — meaning the singular directions are updated in a balanced way rather than concentrated on a few large singular values. To this end, Muon orthogonalizes the momentum matrix (mathematically equivalent to taking the orthogonal factor of the polar decomposition) and updates the parameters accordingly.

    2.3 How Newton–Schulz Iteration Approximates Orthogonalization

    Performing a full matrix decomposition directly is expensive and unfriendly to GPUs. Muon instead uses the Newton–Schulz iteration, a sequence of polynomial transformations that gradually approximate the orthogonalization result. It involves only matrix multiplications, is naturally suited to parallel acceleration, and yields a usable orthogonal approximation in just a few iterations.

    1. # Muon update core illustration (pseudocode)
    2. def muon_update(param, momentum_buf, grad, lr, beta, ns_steps=5):
    3. momentum_buf = beta * momentum_buf + (1 - beta) * grad
    4. g = momentum_buf
    5. # use polynomial iteration to approximate the orthogonal factor of g
    6. for _ in range(ns_steps):
    7. g = newton_schulz_step(g)
    8. param -= lr * g
    9. return param, momentum_buf

    2.4 A Hybrid Division of Labor with Adaptive Optimizers

    Orthogonalization is mainly applicable to two-dimensional weight matrices. For embedding layers, output layers, and one-dimensional parameters (biases, normalization parameters, and so on), the parameter count is either very large or the shape is unsuitable for orthogonalization. In practice, a hybrid division of labor is therefore common: Muon is used for the two-dimensional weight matrices in hidden layers, while the remaining parameters are still handled by an Adam-style optimizer. The “Muon Split” mentioned in the original technical reports is essentially this idea of splitting optimizers by parameter type.

    Part 3: Key Challenges of Combining the Two

    3.1 Interaction Between the Compressed Representation and the Optimizer

    The down-projection, up-projection, and positional projection in MLA are all matrix parameters and are natural targets for Muon. But the compressed low-rank structure means the singular value distribution of these matrices may be unusual, and the orthogonalized update changes the distribution of effective step sizes. A common phenomenon teams observe during training is that certain projection matrices update too aggressively or converge too slowly, requiring targeted adjustments to the learning rate or warmup strategy.

    3.2 Numerical Stability

    The Newton–Schulz iteration involves repeated matrix multiplications and normalizations, which can easily produce numerical drift under mixed-precision (bf16/fp16) training. A common practice is to accumulate in higher precision (such as fp32) at critical steps, and to apply reasonable bias correction and clipping to the momentum, preventing gradient spikes from amplifying orthogonalization errors.

    3.3 Joint Hyperparameter Tuning

    MLA’s compression dimension and the dimension of the decoupled position channel are coupled with Muon’s momentum coefficient, number of iterations, and learning rate, making it difficult to tune each one independently. The recommendation is to first use a small model to determine the order of magnitude via grid or Bayesian search, then scale up to the target size to validate stability.

    Part 4: Engineering Practice Recommendations

    • Verify correctness before performance: Compressed attention and custom optimizers can both introduce subtle bugs. Always use differential testing (comparing against standard attention / Adam under equivalent conditions) to ensure numerical consistency.
    • Enable in phases: Prioritize stability in the early stages of training — fix the Muon iteration count and unify the learning rate first, then gradually tune once the curves stabilize.
    • Measure the memory-throughput tradeoff empirically: The benefits of KV compression correlate strongly with sequence length, batch size, and hardware bandwidth. Test in the target scenario rather than copying others’ conclusions.
    • Rely on general-purpose infrastructure: Large model training usually requires supporting capabilities such as container platforms, distributed storage, logging services, and monitoring/alerting. These make it easier to observe memory usage, throughput, and loss curves, and to locate anomalies quickly.

    Part 5: Conclusion

    MLA and Muon each respond to a core pain point of large model training from a different dimension — memory and convergence efficiency. The former uses joint low-rank compression and positional decoupling to substantially lower KV cache overhead and strengthen long-context capability; the latter uses the idea of matrix orthogonalization to make the update directions of two-dimensional weights more balanced, working in coordination with adaptive optimizers. Combining the two can deliver considerable gains, but it places higher demands on numerical stability and coordinated hyperparameter tuning. Understanding the underlying principles, following the engineering rhythm of “correctness before performance,” and leveraging general-purpose cloud infrastructure for observability are the keys to truly putting these efficient training techniques into production.