MLA KV Cache Compression and Muon Split: Architecture, Optimization, and Deployment

    This article explains how Multi-head Latent Attention (MLA) compresses KV caches by storing a low-dimensional joint latent representation instead of full keys and values. It compares MLA with MHA, GQA, and MQA, discusses decoupled RoPE and matrix absorption, and examines the trade-offs between memory savings, quality, and end-to-end performance. It also introduces Muon Split, a per-head orthogonalized update strategy for MLA projection matrices, followed by training-monitoring practices, implementation guidance, and a staged evaluation framework.

    1. Why Long-Context Inference Runs Out of Memory First

    In standard multi-head attention (MHA), every input position produces a query vector q, a key vector k, and a value vector v. When generating a new token, the model must read the keys and values from all previous positions. Inference systems therefore store them in a KV cache so that historical states do not need to be recomputed at every step.

    The total KV cache size can be estimated as follows:

    1. Cache size in bytes =
    2. number of layers
    3. × cached elements per token per layer
    4. × context length
    5. × batch size
    6. × bytes per element

    For MHA, the number of cached elements per token per layer is:

    1. 2 × n_h × d_h

    Here, n_h is the number of attention heads, d_h is the dimension of each head, and the factor of 2 accounts for both keys and values.

    Consider the following configuration:

    • 60 layers
    • n_h = 128
    • d_h = 128
    • Context length: 128K tokens, or 131,072 tokens
    • Batch size: 1
    • Data type: bfloat16, with 2 bytes per element

    Under MHA, the cache size is approximately:

    1. 60 × (2 × 128 × 128) × 131072 × 1 × 2 bytes
    2. ≈ 480 GiB

    A single request would therefore require approximately 480 GiB of memory, far beyond the capacity of many common accelerators. This shows that, for long-context inference, the KV cache can become the primary bottleneck rather than the model weights themselves.

    Memory pressure also creates several related problems:

    1. Lower serving concurrency: Each request consumes more memory, reducing the number of requests that can run on the same device.
    2. Lower long-context throughput: Every generated token must read a large amount of historical cache data, making memory bandwidth a bottleneck.
    3. Higher deployment cost: Supporting larger caches requires more devices or higher-capacity hardware.
    4. Harder continuous batching: Large differences in request lengths make cache allocation and memory fragmentation more difficult.

    For this reason, compressing the KV cache while preserving as much model capability as possible is an important direction for long-context architecture design.

    2. MLA: Caching a Joint Latent Representation

    The central idea of Multi-head Latent Attention (MLA) is to avoid separately caching complete keys and values. Instead, MLA compresses them into a shared low-dimensional latent representation and stores only that representation.

    Let h_t be the hidden state of token t. The key-value compression and reconstruction process can be expressed as:

    1. c_t^KV = W^DKV h_t (down-projection and cached representation)
    2. k_t^C = W^UK c_t^KV (content key reconstructed from the latent)
    3. v_t^C = W^UV c_t^KV (value reconstructed from the latent)

    The dimension of c_t^KV is d_c, which is much smaller than n_h × d_h.

    The query path is also compressed and reconstructed:

    1. c_t^Q = W^DQ h_t
    2. q_t^C = W^UQ c_t^Q

    Handling Positional Encoding

    MLA must also handle rotary positional embeddings (RoPE). RoPE applies a position-dependent rotation matrix. If RoPE were applied directly to reconstructed keys, the position-dependent rotation would appear between projection matrices and could not be fully absorbed into the query-side projection. During inference, the system would then need to explicitly reconstruct every historical key.

    MLA addresses this with decoupled RoPE. The position-aware component is separated into a small independent vector, while the remaining content component does not carry positional information:

    1. k_t^R = RoPE(W^KR h_t) (positional key, dimension d_r, shared by all heads)
    2. q_t^R = RoPE(W^QR c_t^Q) (positional query, generated per head)
    3. k_t = [k_t^C ; k_t^R]
    4. q_t = [q_t^C ; q_t^R]

    The content key k_t^C is position-independent, allowing W^UK to be absorbed into the query projection. The positional key k_t^R is small enough to cache directly.

    Cache Contents and Matrix Absorption

    For each token at each layer, MLA only needs to cache:

    • c_t^KV, containing d_c elements
    • k_t^R, containing d_r elements

    The total is therefore d_c + d_r elements. The complete k_t^C and v_t^C tensors do not need to be stored during inference.

    Because matrix multiplication is associative, the attention score can be rewritten as:

    1. q_t^C · k_j^C = (q_t^C (W^UK)^T) · c_j^KV

    This allows W^UK to be combined with the query-side projection. Attention scores can then be computed directly against the latent representation c_j^KV, without explicitly reconstructing the content key for every historical token.

    Similarly, W^UV can be combined with the output projection W^O on the value path. This technique is commonly referred to as matrix absorption. It avoids explicitly rebuilding complete keys and values for all historical tokens, reducing both computation and memory traffic.

    The Cost of Compression

    Compression is not entirely lossless. The latent dimension d_c, the projection matrices, and the training objective all affect the final model quality.

    If d_c is too small, the model may have difficulty preserving:

    • Long-range dependencies
    • Entity relationships
    • Fine-grained semantic information

    If d_c is too large, the memory savings become less significant. The design must therefore balance latent capacity against cache reduction.

    3. KV Cache Comparison: MHA, GQA, MQA, and MLA

    The following table compares the number of cached elements per token per layer. For GQA, g denotes the number of key-value groups. MQA is a special case of GQA where g = 1.

    Method Cached elements per token per layer Relative size in the example configuration Positional encoding Main quality risk
    MHA 2 × n_h × d_h 1 RoPE applied directly to keys Baseline
    GQA 2 × g × d_h g / n_h Same as MHA Quality may decline when the number of groups is too small
    MQA 2 × d_h 1 / n_h Same as MHA Sharing one key-value set across all heads can reduce head diversity
    MLA d_c + d_r (d_c + d_r) / (2 × n_h × d_h) Decoupled RoPE key Depends on d_c and the training objective

    Using:

    • n_h = 128
    • d_h = 128
    • g = 8
    • d_c = 512
    • d_r = 64

    the cache sizes are:

    • MHA: 32,768 elements, used as the baseline.
    • GQA: 2,048 elements, approximately 1/16 of MHA.
    • MQA: 256 elements, approximately 1/128 of MHA.
    • MLA: 576 elements, approximately 1/57 of MHA.

    MQA provides the smallest cache, but all heads share the same keys and values, which can cause a noticeable loss of expressive capacity. MLA has a cache size close to GQA while retaining an independent content reconstruction path for each head.

    These figures describe cache size only. The quality impact of compression must be validated with actual evaluation data.

    Higher Compression Is Not Always Better

    The compression dimension determines the trade-off between efficiency and quality. Stronger compression can substantially reduce memory use, but it also creates a tighter information bottleneck. Training must learn which information is essential to retain.

    Design decisions should consider:

    • The target context length
    • The number of layers and hidden dimension
    • The number of attention heads
    • Whether the task depends heavily on exact copying, retrieval, or reasoning
    • The available accelerator memory and memory bandwidth

    Code, structured text, and long documents are particularly sensitive to precise position tracking. Excessive compression may first affect cross-section associations and long-distance retrieval on these tasks. Evaluation should therefore measure more than average generation quality. It should also test long-range retrieval and consistency across multiple turns.

    Lower Cache Size Does Not Guarantee End-to-End Speedup

    MLA reduces cache capacity and cache traffic, but the model still needs to perform projections and other matrix operations associated with reconstruction or matrix absorption.

    During decoding, workloads are often memory-bandwidth bound, so reducing the cache can provide a direct benefit. During prefill, workloads are more often compute-bound, and the additional projections may offset part of the gain.

    End-to-end performance should therefore be measured separately using:

    • Per-step decoding latency
    • Time to first token
    • Long-context throughput
    • Per-request memory usage
    • Serving concurrency

    The theoretical cache size alone does not determine the final speed. The location of compute and bandwidth bottlenecks depends on the hardware and batching strategy.

    4. Muon Split: Per-Head Orthogonalized Updates

    The Basic Muon Update

    Muon is an optimizer designed for two-dimensional weight matrices. Let G_t be the gradient at step t, and let M_t be the momentum buffer. The update can be written as:

    1. M_t = μ × M_{t-1} + G_t
    2. O_t ≈ Orth(M_t)
    3. W_t = W_{t-1} - η × (O_t + λ × W_{t-1})

    Here:

    • μ is the momentum coefficient.
    • η is the learning rate.
    • λ is the weight decay coefficient.
    • Orth(M_t) denotes orthogonalization of M_t.

    Given the singular value decomposition:

    1. M_t = UΣV^T

    the orthogonal factor is UV^T. Practical implementations commonly approximate this operation with Newton–Schulz iterations instead of performing a full singular value decomposition.

    Muon is generally applied to two-dimensional hidden-layer weights, such as attention projection matrices and feed-forward network weights. Embeddings, output layers, normalization parameters, and biases continue to use AdamW.

    The main differences between Muon and AdamW are:

    • AdamW maintains first- and second-moment estimates for every element and applies element-wise adaptive scaling.
    • Muon maintains only a momentum buffer, reducing optimizer-state memory.
    • Muon orthogonalizes the update for the entire matrix, making update magnitudes more balanced across singular directions.
    • Muon requires separate tuning of the learning rate, momentum coefficient, number of Newton–Schulz iterations, and weight decay.

    How Muon Split Works

    The MLA up-projection matrices, such as W^UK and W^UV, concatenate all attention heads. Their approximate shape is:

    1. d_c × (n_h × d_h)

    If the entire matrix is orthogonalized in one operation, the updates for different heads become coupled. The gradient direction of one head can affect the global orthogonal basis, and the update scales of all heads are normalized together.

    Muon Split divides the matrix along the head dimension, orthogonalizes each block independently, and then concatenates the results:

    1. W^UK = [W_1, W_2, ..., W_{n_h}]
    2. O_i = Orth(M_i), i = 1, ..., n_h
    3. O = [O_1, O_2, ..., O_{n_h}]

    Each head’s update is therefore determined only by the gradient associated with that head.

    In this article, Muon Split refers to this implementation in which the matrix is split by attention head and each block is orthogonalized independently. The exact name and implementation details should be verified against the original source. If a project does not use the term Muon Split, the same method can be described as per-head orthogonalized updates.

    Applicability and Limitations

    Muon Split has several limitations:

    • It applies only to two-dimensional matrices that can be divided by attention head, and the head dimension must be divisible as required by the split.
    • When there are many heads and each block is small, the information available for each individual orthogonalization may be limited. Its benefits must be verified experimentally.
    • Splitting by head increases the number of orthogonalization calls and may reduce training throughput. The impact should be measured in practice.

    General parameter grouping should not automatically be called Muon Split. The term should be reserved for updates that are actually split by head and orthogonalized independently.

    5. Training Stability Risks and Monitoring

    With compressed attention, the model no longer stores complete key-value representations directly. Instead, it stores and reconstructs them through latent variables. This changes the gradient propagation path and the scale relationships between modules.

    A training log that records only total loss cannot reveal these issues early enough. The following four risks should be monitored with dedicated metrics and debugging actions.

    1. Latent-Variable Scale Drift

    If the numerical range of c_t^KV continually expands, the reconstruction and attention calculations may be amplified abnormally. If the range becomes too small, useful signals may be buried.

    Monitoring metrics:

    • Mean and variance of c_t^KV
    • Maximum absolute value
    • Fraction of elements exceeding a defined threshold

    Debugging actions:

    • Check the learning rate of the parameter group containing W^DKV.
    • Consider adding normalization to the latent representation.
    • If the metrics continue to deteriorate, roll back to the most recent stable checkpoint.

    2. Excessively Concentrated Attention

    When the scale of the query-key dot product becomes too large, the Softmax input can contain extreme values. Attention may then concentrate on only a few positions, reducing the use of broader context and causing an imbalanced gradient distribution.

    Monitoring metrics:

    • Attention entropy for each head
    • Maximum attention weight per head
    • Maximum attention logit

    Debugging actions:

    • Apply clipping or normalization to attention logits.
    • Reduce the learning rate.
    • Check whether the scaling of the decoupled RoPE component is appropriate.

    3. Inconsistent Convergence Across Modules

    The compression layer, reconstruction layers, and main network may have different optimization difficulty. If one component converges too quickly while others are still exploring, the loss may decrease slowly or fluctuate periodically.

    Monitoring metrics:

    • Gradient norm for each parameter group
    • Update ratio:
    1. ||ΔW|| / ||W||

    Debugging actions:

    • Use different learning rates for different modules.
    • Check whether the update ratios of the compression and reconstruction layers differ substantially from those of the rest of the model.

    4. Numerical Errors Amplified by Mixed Precision

    In low-precision training, matrix multiplication, normalization, and Softmax are sensitive to numerical range. Additional compression and reconstruction transformations can increase the risk of overflow and precision loss.

    Monitoring metrics:

    • Number of inf and NaN values in gradients
    • Number of update steps skipped because of overflow
    • Maximum absolute activation value in each layer

    Debugging actions:

    • Preserve FP32 precision in critical operations such as accumulation and Softmax.
    • Reduce the initial loss scale.
    • Check whether the maximum activation value suddenly increases in a particular layer.

    If a loss spike does not recover after several update steps, roll back to a checkpoint instead of continuing training.

    6. Engineering Implementation: From Cache Layout to Staged Validation

    1. Coordinate Cache Layout with Compute Kernels

    If the cache is compressed but the runtime repeatedly expands it into complete keys and values, the expected benefit will not be achieved. The implementation should allow c_t^KV and k_t^R to participate directly in cache reads and writes, while avoiding allocation of complete historical k and v tensors whenever possible.

    The cache can be allocated in fixed-size pages. The appropriate page size depends on the specific kernels and hardware and should be determined through measurement rather than copied as a universal constant.

    2. Separate Prefill from Incremental Decoding

    The prefill stage processes the entire input sequence and performs a large amount of computation. Incremental decoding reads the historical cache at every step and is often limited by memory bandwidth.

    Because the two stages benefit differently from compression, benchmarks should report them separately:

    • Time to first token, which includes prefill
    • Time per generated token

    A single average speed number is not sufficient.

    3. Manage Variable-Length Requests

    Production workloads usually contain requests with widely different sequence lengths. Cache management must handle:

    • Paging
    • Reuse
    • Release
    • Fragmentation management

    Very long requests can be assigned to a separate queue or subject to a length limit. This prevents a small number of requests from consuming all available memory and reducing overall concurrency.

    4. Validate in Stages

    A staged validation process helps distinguish model-quality problems from system implementation problems. The thresholds in the following table must be calibrated against the team’s baseline model; the table describes the validation structure rather than universal numerical requirements.

    Stage Validation target Pass criterion Action on failure Required log fields
    1 Output consistency with the baseline Maximum logit absolute error is below the defined threshold, and top-1 token agreement reaches the target Check matrix absorption and decoupled RoPE implementation Layer ID, maximum error, sequence length
    2 Quality at different sequence lengths Score degradation in each length bucket stays within the predefined limit Adjust d_c or retrain with modified settings Length bucket, task name, score
    3 Memory, latency, and throughput Peak memory decreases significantly and per-token generation time remains below the target Check whether kernels are repeatedly expanding the cache Batch size, peak memory, time to first token, time per token
    4 High concurrency and long-duration operation Memory does not continuously increase, and P99 latency stays within the target range Apply rate limiting or roll back to the previous version Memory-over-time curve, request error code

    If the first stage fails, the issue is more likely to be an implementation error. If the second stage fails after output consistency has been confirmed, the problem is more likely related to compression capacity or training.

    7. Evaluation Framework: Metrics and Measurement Conventions

    A complete evaluation should cover four categories: quality, resources, performance, and stability. Every metric should have a precise definition, measurement method, and comparison baseline.

    1. Quality Metrics

    • Long-context retrieval: Insert key information into a very long document and measure whether the model can accurately reproduce it.
    • Cross-section reasoning: Create question-answering tasks that require connecting information across multiple sections and report accuracy.
    • Code understanding: Use long source files and test questions about function calls, variable origins, and related dependencies.
    • Multi-turn consistency: Check whether the model preserves facts and constraints across multiple dialogue turns and report the contradiction rate.

    Quality comparisons should use MHA or GQA baselines with similar parameter counts and training data.

    2. Resource Metrics

    • Peak device memory: The maximum memory allocated during inference. In PyTorch, torch.cuda.max_memory_allocated can be used, while reserved memory should also be recorded.
    • Per-request cache usage: Calculate the theoretical value using:
    1. number of layers
    2. × cached elements per token per layer
    3. × context length
    4. × bytes per element

    Then compare it with the measured value.

    • Device utilization: Measure the utilization of compute units and memory bandwidth using the relevant hardware profiling tools.

    3. Performance Metrics

    • Time to first token (TTFT): Time from request arrival until the first output token, including prefill.
    • Time per output token (TPOT):
    1. (end-to-end latency − TTFT) / (number of output tokens − 1)
    • Throughput: Total number of tokens generated per second at steady state.
    • Tail latency: P95 and P99 end-to-end latency under a fixed concurrency level.

    All performance metrics should be reported by input-length bucket. Suggested buckets are:

    • 1K
    • 4K
    • 16K
    • 64K
    • 128K

    4. Stability Metrics

    • Memory growth during long-running tests: Run the system continuously for several hours and record memory over time to identify possible leaks.
    • Failure recovery: Inject an overlong or failed request and measure how long the system takes to return to normal latency.
    • Low-precision numerical stability: Count overflows, NaN occurrences, and skipped update steps during training or inference.

    5. Test Design Considerations

    Compression methods should not be evaluated only on datasets with average-length inputs. Build segmented test sets ranging from short text to extremely long contexts, and check whether quality or performance changes suddenly within a particular length range.

    For every comparison, keep the following conditions fixed:

    • Hardware
    • Batch size
    • Numerical precision
    • Sampling parameters

    Only the variable under evaluation should change.

    8. Conclusion

    MLA compresses the KV cache by storing a low-dimensional joint latent representation instead of complete keys and values. Decoupled RoPE and matrix absorption allow this compressed representation to work with positional encoding and attention computation without explicitly reconstructing all historical keys and values.

    In the example configuration, MLA reduces the per-token, per-layer cache from 32,768 elements to 576, producing a substantial reduction in memory requirements. Muon Split takes a different role during training: it divides MLA up-projection matrices by attention head and orthogonalizes each block independently, preventing updates from different heads from being coupled through a single global orthogonalization step.

    In production, validation should begin with output consistency and then proceed to quality, resource usage, performance, and stability. Cache reduction is only the starting point. Final results depend on the latent dimension, training behavior, kernel implementation, hardware characteristics, and workload distribution.