# LLM Architecture Evolution and Selection: How to Choose Across 7 Key Directions

    LLM architecture is no longer decided by scaling alone. This article reviews how language model architectures evolved and breaks down seven design directions — GQA, MLA, sparse attention, state-space models, fine-grained sparse experts, and multi-token prediction — with their trade-offs, real costs, and the metrics to validate them. You'll find guidance on long-context KV cache compression, linear attention recall, MoE load balancing (with a PyTorch forward-pass example), SwiGLU activation choices, normalization and residual connections, position encoding extrapolation, and tokenizer selection, plus a quick-reference selection table and notes on how architecture drives parallelism and deployment decisions.

    1. Start With the Conclusion: Why Architecture Deserves a Re-Evaluation

    Let’s state the conclusion up front: scaling up is not the only variable. The underlying architecture is equally decisive for the “efficiency–quality–cost” triangle.

    Every flagship release keeps refining the same set of designs: grouped-query attention (GQA), multi-head latent attention (MLA), sparse attention, state-space models, fine-grained sparse experts, multi-token prediction, and more. New requirements — continual learning, test-time training — are pushing architectures to evolve further. The prevailing view in the field is that today’s architectures are far from the end of the road.

    The goal of this article is to answer three practical questions: how to choose, what the cost is, and how to verify.

    2. How Architectures Got Here: A Nearly Century-Long Main Line

    The evolution of language models spans almost a century. The key milestones:

    • 1913: The mathematician Markov analyzed vowel and consonant patterns in Pushkin’s verse novel Eugene Onegin, becoming the first to reveal conditional dependence in language with data.
    • 1948: Shannon, in information theory, modeled language as a stochastic source, establishing the probabilistic view of language modeling.
    • Around 1990: n-gram models from speech recognition, followed by recurrent neural networks (RNNs); in 1997, Hochreiter and Schmidhuber introduced the long short-term memory network (LSTM), using gating mechanisms to mitigate vanishing gradients.
    • 2017: The Transformer architecture appeared and became the foundation of today’s LLM ecosystem.

    A longer observation window offers useful cross-validation, but keep in mind these are analogies in mathematical form, not strict equivalences:

    • The additive outer-product update rule of fast weights (Schmidhuber, 1992) can, under certain conditions, be viewed as “linear attention without normalization”; kernelized linear attention (Katharopoulos et al., 2020) added a normalization mechanism to it.
    • The multiplicative gating of LSTM was carried into feed-forward networks, giving rise to the gated linear unit GLU (Dauphin et al., 2017) and variants such as SwiGLU; the additive information pathway of LSTM is in the same lineage as residual connections (He et al., 2016, an idea originating from Highway Networks, Srivastava et al., 2015).

    3. Breaking Down the Seven Directions: How to Choose and What It Costs

    3.1 Long-context memory blew up: choosing among GQA, MLA, and sparse attention

    Self-attention’s compute grows quadratically with sequence length (so does the memory footprint of the attention matrix during training), while the KV cache grows linearly with context length. GQA and MLA are precisely what compress the latter. Three paths:

    • GQA: multiple query heads share one set of key/value heads, compressing the KV cache with almost no quality loss.
    • MLA: compresses keys and values into a low-dimensional latent space, then projects them back.
    • Sparse / local attention: computes only a subset of attention pairs, trading some globality for efficiency.

    How do GQA and MLA differ? GQA reduces the number of KV heads; MLA relies on low-rank compression. The former is simpler to implement, the latter achieves a higher compression ratio but requires tuning the latent dimension. A common mistake is looking only at the theoretical length without checking actual recall.

    To verify: estimate memory with 2 × num_layers × num_kv_heads × head_dim × bytes_per_precision × context_length; measure the effective length with Needle-in-a-Haystack or RULER-style tasks, and compare theoretical context length against actual recall accuracy.

    3.2 What if linear attention recalls poorly: can linear language models replace attention?

    Linear models try to break the quadratic complexity barrier by replacing softmax attention with linear operators that can be computed efficiently as recurrences — typically state-space models and various linear / gated linear attention variants.

    Linear attention and SSMs with a fixed-size (stateful) state can compress history into a state of fixed size, yielding roughly constant decoding cost per step. But the state size still grows with the model dimension, and for very long sequences memory still grows linearly with length — just far less than a KV cache. The advantages are clear for very long contexts and on-device deployment. However, most hybrid linear models still trail full attention or hybrid attention on exact retrieval recall in long contexts.

    What to do if linear attention recalls poorly? If the task depends on precisely referencing one segment of the history, prefer a hybrid design or full attention, and measure whether recall accuracy meets the requirement.

    3.3 How to fix MoE load imbalance: mixture of experts

    MoE expands parameter capacity without a proportional increase in compute through sparse activation: a router selects a small number of experts to compute each token, so total parameter count is very large while the parameters activated per token are only a fraction. Finer-grained experts, a higher number of activated experts, and shared experts further balance capacity, efficiency, and specialization. The key failure mode is load imbalance (some experts overloaded, others idle), which is why a load-balancing loss and a capacity factor are mandatory.

    The forward logic looks like this (PyTorch-based, needs nn.ModuleList; this only illustrates the forward pass — training additionally requires expert parallelism and a load-balancing loss):

    1. import torch
    2. import torch.nn.functional as F
    3. def moe_forward(x, experts, router, top_k=2):
    4. # x: [tokens, hidden]; experts: nn.ModuleList; router: nn.Linear(hidden, num_experts)
    5. logits = router(x)
    6. topk_w, topk_idx = logits.topk(top_k, dim=-1)
    7. weights = F.softmax(topk_w, dim=-1) # normalize only across the k selected experts
    8. out = torch.zeros_like(x)
    9. for e, expert in enumerate(experts):
    10. mask = (topk_idx == e).any(dim=-1) # [tokens] whether the token is routed to expert e
    11. idx = mask.nonzero(as_tuple=True)[0] # [n_e]
    12. if idx.numel() == 0:
    13. continue
    14. w_e = (weights * (topk_idx == e)).sum(dim=-1)[idx] # [n_e] gating weight of each token on expert e
    15. out[idx] += w_e.unsqueeze(-1) * expert(x[idx]) # evaluate expert e only for the routed tokens
    16. return out

    3.4 Why everyone uses SwiGLU now: how to choose an activation

    GLU and its variants (such as SwiGLU) introduce element-wise multiplicative gating between two linear transformations, strengthening the expressiveness of the feed-forward network, and they have become the modern mainstream. Compared with ReLU/GELU, these structures often perform better at the same parameter count while retaining good optimizability. Be explicit about the accounting:

    • With hidden dimension d_ff held constant, parameters and compute are roughly 1.5× that of an equivalent FFN.
    • If designed for equal parameters (d_ff reduced to 2/3, i.e. 8/3 · d), parameter count is on par and compute is lower.

    Choose the configuration according to your compute budget.

    3.5 What if training keeps diverging: normalization and residual connections

    Normalization layers (LayerNorm, RMSNorm — the latter drops mean centering to save compute) determine training stability and convergence speed. Pre-Norm is the default choice for large-model training. Key points: pure Pre-Norm tends to suffer representation degradation in very deep stacks, so in practice it is combined with warmup, residual scaling (e.g. DeepNorm), or hybrid normalization; the position of normalization, along with learnable scale/shift, is a trade-off between stability and efficiency. Residual connections provide an identity-mapping fast path, mitigating gradient vanishing in deep stacks and making “trainable deep networks” possible.

    3.6 Everything collapses as the context grows: position encoding and extrapolation

    Self-attention is insensitive to order, so position information must be injected via position encoding — from absolute encodings, sinusoidal/learned encodings, to today’s mainstream rotary position embedding (RoPE). When you go beyond the training length, options include position interpolation (PI), NTK-aware scaling, and YaRN, each with its own applicable conditions and train-free vs. fine-tuning trade-offs. Extrapolated length is not effective recall length, and a flat perplexity does not mean long-context ability is adequate — measure retrieval accuracy.

    3.7 How to choose a tokenizer: BPE and granularity

    The tokenizer determines the granularity of the text-to-token mapping, affecting sequence length, vocabulary size, and multilingual coverage. Subword methods such as byte pair encoding (BPE) are the current mainstream. Key points: the larger the vocabulary, the more parameters in the embedding and output layers; the same text consumes more tokens in some languages, so compression rates are uneven; digit-by-digit splitting affects arithmetic and code performance. When choosing, look at compression rate, multilingual token cost, and digit/code splitting strategy.

    3.8 Quick-reference architecture selection table

    Architecture direction Advantages Main cost Suitable scenarios Validation metrics
    GQA / MLA Sharply compresses KV cache, near-lossless quality Implementation complexity, latent-space design tuning Long context, high-concurrency inference serving Throughput, memory usage, recall accuracy
    Linear models / SSM Near-constant-time inference, friendly to very long sequences Exact retrieval recall may drop On-device, very long text, low latency Retrieval accuracy, extrapolation length, decode latency
    MoE Large capacity, controllable per-token compute Load imbalance, memory and communication overhead Chasing the quality ceiling with ample compute Expert load distribution, activated parameter ratio, throughput
    SwiGLU / GLU Stronger FFN expressiveness at equal parameters ~1.5× params and compute at the same d_ff General pre-training Perplexity, convergence speed
    Pre-Norm / RMSNorm Stable deep training, RMSNorm is cheaper Normalization placement and deep stability need tuning General pre-training Loss spikes, gradient norm
    RoPE and length extrapolation Good relative position awareness, mature extrapolation options Beyond training length requires interpolation/YaRN, etc. Long context Perplexity and retrieval accuracy at different lengths
    BPE tokenizer Good subword coverage, few OOV tokens Larger vocab means more embedding params, uneven multilingual compression General and multilingual Compression rate, multilingual token cost, digit and code splitting

    4. Architecture Evolution Driven by New Requirements

    Continual learning and test-time training are pushing architectures in new directions. Under the traditional strict separation of “training then inference”, weights are frozen at inference time; a new generation of architectures explores dynamically updating weights or state during use, so the model keeps absorbing new information and adapting to distribution shifts. This imposes entirely new design requirements on memory mechanisms, state management, and training stability — and it is an important frontier for architectural innovation.

    5. How Architecture Determines the Engineering Configuration

    An architecture choice ultimately has to land on an engineering system, and it directly determines the parallelism and deployment strategy:

    • MoE requires expert parallelism and relies on all-to-all communication, a major source of memory and bandwidth bottlenecks.
    • Linear models have a fixed inference state size, which favors low latency.
    • GQA/MLA reduce the KV cache and thereby raise the per-device concurrency ceiling.

    Selection considerations include: parallelism strategy (how data/tensor/pipeline parallelism match the architecture), memory and bandwidth bottlenecks, serving metrics (time-to-first-token P50/P99, throughput, cost per request), and cost monitoring (compute utilization and cost per request). In addition, training and log data should undergo compliance review and redaction with least-privilege access, and training cost and deployment risk should be assessed in advance.

    6. Conclusion: An Actionable Decision Checklist

    From the continuous optimization of attention efficiency, to linear models breaking the quadratic bottleneck, to sparsification and stateful designs reshaping capacity and memory mechanisms, architectural innovation is advancing in parallel with scaling. When putting it into practice, use the table in Section 3:

    • Very long context (see the GQA/MLA row): prefer GQA/MLA and validate recall.
    • Low latency / on-device (see the linear model row): consider linear models or SSMs.
    • Chasing the quality ceiling with ample compute (see the MoE row): MoE is an option, but you must configure load balancing.
    • General scenarios: start from the Pre-Norm + SwiGLU + RoPE + GQA combination.

    Grasping the internal logic and trade-offs of these dimensions is the key to understanding where next-generation language models are heading — and to making the right engineering decisions.