MQA in Open-Source LLMs: Why Models Are Returning and What the Architecture Trade-offs Are

    A new open-source LLM preview has drawn attention for moving from MLA back to MQA. This article summarizes developer-reported capability gains, compares MHA, GQA, MQA, and MLA on KV-cache size, inference bandwidth, expressiveness, and engineering complexity, and explains how teams can evaluate such models on their own tasks before adoption.

    Introduction

    A preview version of a new-generation open-source large model has recently attracted attention. Developer feedback indicates that it has made progress over the previous generation in mathematical reasoning, code generation, and Agent instruction execution. There is also a relatively uncommon architectural adjustment: MLA, which was commonly used in the previous two generations, has been abandoned, and the model has returned to multi-query attention (MQA).

    This article focuses on that architecture choice and compares the trade-offs among several attention mechanisms. The performance data and parameter descriptions cited here come from the original report or official claims. Before making a selection, rely on official documentation and public evaluations.

    Capability Performance: Clear Progress, More Evidence Needed

    From developer feedback, this generation improves on the previous one in mathematical reasoning, code generation, and Agent tasks, and hallucination issues have also improved to some degree. However, descriptions such as “significant” or “leapfrog” are meaningful only when accompanied by data. When reading such claims, focus on three points: which benchmarks were used, whether the test setup was consistent with the comparison models, and what the comparison objects were.

    In complex code generation and long-context logical reasoning, the model still has a gap compared with current top closed-source models. Its overall level is roughly comparable to other strong open-source models from the same period. Capability improvements may come from better pretraining data quality, adjustments to the alignment strategy, and architecture changes, but there is no official ablation data yet on the contribution of each part. Judging this point requires waiting for the official technical report.

    For teams making a selection, the safer approach is to run comparison tests on their own tasks rather than directly replacing an existing solution. A minimal test table can include the following fields: task type, benchmark or sample set, comparison model, test settings (temperature, maximum output length, few-shot count), evaluation metrics, and conclusion and confidence level.

    Architecture Choice: From MLA Back to MQA

    To understand this reversion, it helps to first look at the evolution of attention mechanisms. Standard multi-head attention (MHA) stores independent Key and Value tensors for each head. During inference, the size of the KV cache is proportional to the number of heads. In long-context and high-concurrency scenarios, this creates significant memory pressure.

    MQA lets all query heads share the same set of Key and Value. With 64 heads as an example, the KV data that needs to be cached for each token at each layer drops to 1/64 of that required by MHA, and the memory bandwidth that must be read during inference also decreases. The cost is that model expressiveness may be constrained, and training stability requires more tuning.

    GQA sits between the two, letting a group of query heads share one set of KV. MLA compresses KV into low-dimensional latent variables. It retains more expressiveness under a similar cache budget, but its implementation is more complex.

    The table below provides a qualitative comparison of the four mechanisms. The conclusions describe trends under common configurations; actual values depend on the number of heads, head_dim, and compression dimension.

    Mechanism KV cache per token per layer Inference bandwidth pressure Expressiveness Engineering complexity
    MHA Largest; proportional to the number of heads High Usually the strongest Low
    GQA Medium; reduced by group count Medium Relatively strong Low
    MQA Smallest; only one KV group (one head_dim each for K and V) Low Relatively limited Low
    MLA Depends on latent dimension; comparable to or larger than MQA depending on configuration Medium-low Relatively strong High

    Note that MLA’s cache is not necessarily smaller than MQA’s. MQA caches K and V per layer for a total of 2 × head_dim values. MLA caches compressed latent variables plus accompanying positional-encoding dimensions; the size relationship between the two must be calculated for each specific configuration. MLA’s advantage is that it retains more head expressiveness under the same cache budget, not that it is absolutely more memory-efficient.