LLM Architecture Evolution and Engineering Selection Guide: GQA, MLA, Linear Attention, MoE and RoPE

    Model architecture has become a core variable in LLM efficiency, training stability, and inference cost—not just parameter count. This article traces how language model architectures evolved from early statistical studies through n-grams, RNNs, and LSTMs to Transformers, then breaks down the key building blocks of modern LLMs: reducing KV cache and memory overhead with GQA, MLA, and sparse attention; choosing between linear attention and hybrid architectures; Mixture of Experts (MoE) routing and load balancing; SwiGLU and gated feed-forward networks; normalization and residual connections; RoPE versus ALiBi for positional encoding; and tokenizers. It also covers directions driven by newer requirements, such as continual learning and test-time training, and ends with a practical, scenario-based checklist for architecture selection.

    1. Why Scaling Alone Is Not Enough

    Over the past few years, gains in large language model capability were often attributed to growth in parameter count, data volume, and compute. But engineering practice has made one thing increasingly clear: architecture itself is now a core variable that determines model efficiency, training stability, and inference cost.

    New designs such as Grouped Query Attention (GQA), Multi-head Latent Attention (MLA), sparse attention, linear attention, fine-grained Mixture of Experts (MoE), multi-token prediction, and looped architectures are almost all optimizing the same set of problems: how to reduce memory footprint, how to improve long-context throughput, how to keep training stable, and how to make inference cheaper.

    At the same time, new requirements such as continual learning and test-time training are pushing models toward designs that can update state or weights dynamically during inference. In other words, today’s mainstream architectures are not the endpoint. For developers, the key to understanding architectural evolution is not memorizing terminology, but judging what problem each component solves, what it costs, and which scenarios it fits.

    2. The Longer View: How Language Model Architectures Evolved

    Stretch the timeline out and you find that language model architectures did not evolve through simple linear replacement—they crossed and re-crossed several times.

    • Around 1913, researchers analyzed the arrangement of vowels and consonants in classical Russian verse novels, using data to show that conditional dependencies exist within language.
    • In 1948, information theory treated language as a random source, laying the foundation for probabilistic sequence modeling.
    • In the speech recognition era, n-gram models estimated the probability of local word sequences statistically and became an important early approach to language modeling.
    • Around 1990, Recurrent Neural Networks (RNNs) began to be applied to sequences.
    • In 1997, Long Short-Term Memory (LSTM) was introduced, using gating mechanisms to mitigate gradient problems in long sequences.
    • After 2017, the Transformer became the mainstream architectural foundation for large language models.

    It is worth noting that many designs that look “new” today have prototypes in much earlier ideas. For example, the Fast Weight Programmers proposed around 1991 included certain update rules based on additive outer products that can be seen as analogous to “a linear Transformer without the normalization form.” Kernelized linear attention, which appeared around 2020, can in turn be understood as adding normalization or stabilization mechanisms to that idea from a mathematical standpoint.

    Similarly, the multiplicative gating idea in LSTM is conceptually close to gated structures such as SwiGLU in modern feed-forward networks, and the additive pathway in LSTM that preserves the error flow is similar in design intent to the residual connection. It is important to stress that these are historical analogies and conceptual correspondences, not strict mathematical equivalences. Understanding this lineage helps us quickly judge, when facing a new architecture, whether it is a genuinely new breakthrough or an engineering reassembly of existing components.

    3. Breaking Down the Key Components of a Modern Transformer

    A good architectural understanding usually does not start from “bigger and stronger.” It starts by decomposing components. Below we go module by module: attention, linear models, MoE, feed-forward networks, normalization, positional encoding, and tokenizers.

    3.1 Reducing KV Cache and Memory Overhead

    The compute and memory cost of standard self-attention grows quadratically with sequence length. During inference, the Key and Value tensors of historical tokens must also be cached—the KV cache. In long-context scenarios, the KV cache often affects memory footprint and throughput more than the model parameters themselves.

    Three main routes have emerged around this bottleneck:

    Route Representative methods Problem it solves Main cost or caveat
    Query sharing GQA, MQA Let multiple query heads share a small number of key/value heads, reducing the KV cache The query-to-key/value head ratio must be tuned; quality may be slightly affected
    Low-rank compression MLA Project Key/Value into a lower-dimensional latent space to substantially shrink the cache More complex to implement; requires additional projection and reconstruction design
    Sparsification Local windows, global tokens, dynamic routing Compute only a subset of attention connections to reduce compute Precise long-range retrieval may suffer; requires training-time adaptation

    The core of GQA is “multiple query heads sharing one group of key/value heads.” In most scenarios it significantly reduces inference memory with only a small quality cost. MLA compresses key/value information through a low-rank latent representation and is better suited to scenarios with high demands on long context and memory efficiency. Sparse attention is better treated as a system-level optimization, combined with local windows, global summary tokens, or dynamic routing.

    In engineering terms, if the goal is long-context inference, three things should be examined first: KV cache footprint, time-to-first-token latency, and long-text throughput. Do not look only at parameter count.

    3.2 Choosing Between Linear Language Models and Hybrid Architectures

    Linear attention uses kernel functions or state-update mechanisms to reduce attention complexity from quadratic to linear, while compressing long history into a recurrent state of fixed size. State Space Models (SSMs) and gated linear recurrent networks also belong to this broader family of ideas.

    The advantage of linear models is that memory usage is more controllable during long-sequence inference, with higher throughput potential. The difficulty is that their precise retrieval ability is usually weaker than full attention. In real systems, more and more models adopt hybrid architectures: some layers retain global attention while others use linear or recurrent modules. This preserves long-sequence efficiency while keeping precise access to key tokens.

    If your workload is mainly very long documents, codebases, or log streams, linear attention and attention-hybrid architectures are worth close attention. If your workload depends heavily on precise multi-hop retrieval, it is not advisable to simply replace all attention layers.

    3.3 How MoE Balances Parameter Count and Compute

    MoE uses a routing mechanism so that each token only activates part of the expert networks, which keeps per-token compute under control even as total parameter count grows. Current focus areas include finer-grained expert partitioning, more stable routing strategies, and better load balancing methods.

    In engineering terms, MoE is not “bigger is better.” Common concerns include the number of experts, routing temperature, capacity factor, auxiliary load-balancing loss, and the expert activation strategy at inference time. If routing is unbalanced, some experts may be overloaded while others sit idle, ultimately hurting training efficiency and inference stability.

    MoE fits scenarios with a clear requirement: expand parameter capacity without making every token pay the full compute cost. If a team lacks the ability to tune routing and load balancing, adopting MoE can simply add system complexity.

    3.4 Why Activation Functions and Feed-Forward Networks Favor SwiGLU

    The Feed-Forward Network (FFN) usually accounts for a substantial share of a model’s parameters. Gated Linear Units (GLUs) and their variant SwiGLU improve information selection and representational capability through multiplicative gating, and have become a common choice in modern architectures.

    The following is an illustrative implementation showing the basic shape of a SwiGLU/GLU feed-forward layer:

    1. import torch.nn.functional as F
    2. # x: [batch, seq_len, d_model]
    3. # w_gate: [d_ff, d_model]
    4. # w_up: [d_ff, d_model]
    5. # w_down: [d_model, d_ff]
    6. def gated_ffn(x, w_gate, w_up, w_down):
    7. # SiLU is the activation function commonly used in SwiGLU
    8. gate = F.silu(F.linear(x, w_gate)) # [batch, seq_len, d_ff]
    9. value = F.linear(x, w_up) # [batch, seq_len, d_ff]
    10. return F.linear(gate * value, w_down) # [batch, seq_len, d_model]

    Here gate * value is the essence of the gating structure: one branch learns gate weights, the other provides candidate values, and the two are multiplied before being projected back down to the model dimension. Compared with a plain FFN, this kind of structure usually has stronger representational capability, but it also consumes more parameter budget, so real designs often need to adjust the intermediate dimension accordingly.

    3.5 How Normalization and Residual Connections Work Together

    Normalization layers directly affect training stability. LayerNorm is the classic choice, but Root Mean Square Normalization (RMSNorm) removes the mean-centering step, costs less to compute, and has proven an effective replacement in the training of many large models.

    Residual connections give deep networks a stable information pathway. They make gradients easier to propagate and allow models to stack more layers. In modern Transformers, a Pre-LN or similar pre-normalization layout usually helps deep training stay stable, and the combination of residual connections with gated feed-forward and attention modules is a common engineering skeleton.

    A reliable default combination is: RMSNorm + Pre-LN + residual connections + gated feed-forward. This combination is not guaranteed to be optimal on every task, but it is usually a dependable starting point.

    3.6 RoPE or ALiBi?

    Self-attention is inherently insensitive to order, so positional information must be injected explicitly. Rotary Position Embedding (RoPE) applies a rotational transformation to queries and keys, encoding relative position information into the attention scores. Its advantages are natural expressiveness and good compatibility with a variety of attention implementations, and it has become an important default choice in modern models.

    However, one point deserves special emphasis: native RoPE has limited long-text extrapolation ability. If you want to extend a trained context length to something longer, the common practice is to combine it with NTK-aware scaling, YaRN, interpolation, or position-scaling strategies. Do not assume that “RoPE naturally extrapolates to arbitrary lengths.”

    Method Mechanism Advantages Caveats
    RoPE Encodes relative position by rotating queries and keys Natural expressiveness, good engineering compatibility Limited native extrapolation; usually needs scaling, interpolation, or fine-tuning
    ALiBi Adds a distance-dependent bias to attention scores Simple extrapolation, friendly to long context Somewhat simplified expressiveness; complex positional relationships may be less flexible than with RoPE

    When choosing: if you already have a mature RoPE base, evaluate RoPE plus a position-scaling scheme first. If you are designing from scratch and value simple extrapolation more, an ALiBi-style approach is worth evaluating.

    3.7 Why the Tokenizer Is the Foundation on the Input Side

    The tokenizer determines the mapping from text to tokens, making it the foundation of the model’s input side. Byte Pair Encoding (BPE) and its family trade off vocabulary size, compression rate, and multilingual capability.

    Tokenizers may look unremarkable, but they directly affect three things: sequence length, inference cost, and how well the model adapts to different languages. A vocabulary that is too small can split text into too many tokens and stretch the context; a vocabulary that is too large can increase the cost of the embedding and output layers. For multilingual scenarios, the tokenization quality of low-resource languages deserves particular attention.

    4. Directions Driven by New Requirements: Continual Learning and Test-Time Training

    Beyond optimizing conventional components, two new requirements are shaping architecture design.

    The first is continual learning. After deployment, models still need to absorb new knowledge without excessively forgetting old capabilities. This requires architectures to be more plastic in terms of state updates, parameter freezing, and modular extension.

    The second is test-time training. It focuses on making lightweight updates to model parameters or hidden states during inference, using the context of the current task. Some frontier work explores models whose weights or state can be updated at inference time, making the model better adapted to dynamic inputs. That said, these directions are still evolving, and engineering adoption requires careful evaluation of stability, latency, and controllability.

    5. A Practical Checklist: How Should You Choose an Architecture?

    No architecture is absolutely best; there are only architectures that match your scenario. Below is an engineering-oriented selection checklist.

    1. High context-length requirements: Evaluate KV cache compression schemes first. If the target context is long, focus on GQA, MLA, or hybrid linear attention. Measure peak KV cache with real business text rather than looking only at theoretical complexity.

    2. Sensitive to throughput and cost: Consider linear attention / state space hybrid structures, or MoE sparse activation. But also evaluate memory capacity, routing stability, and inference batching efficiency. MoE is not a free lunch—unstable expert routing can cancel out the sparse gains.

    3. Training stability is the priority: Favor the combination of RMSNorm, Pre-LN, residual connections, and gated feed-forward. If loss spikes appear during training, check normalization placement, learning rate, residual scaling, and gradient clipping first.

    4. Long-text extrapolation: If you use RoPE, evaluate NTK, YaRN, or interpolation strategies alongside it. Be sure to run long-text positional extrapolation tests before launch, rather than validating only within the training length.

    5. Multilingual or multi-domain input: Prioritize a tokenizer evaluation. Check token compression rates for low-resource languages, code, mathematical notation, and structured text, since this directly inflates or shrinks context cost.

    6. Common pitfalls: Don’t look only at parameter count, don’t ignore the KV cache, don’t adopt MoE blindly without validating routing stability, and don’t assume RoPE extrapolates for free.

    Overall, the direction of LLM architecture evolution is shifting from simply piling on scale toward finer component coordination and engineering trade-offs. For developers, what is truly useful is not chasing every new term, but building a capability for judgment: understanding what problem each module solves, knowing the scenarios where it is worth using, and being clear about where it will introduce new costs.