How LLM Architectures Evolved in 2025: A Deep Dive into the Structural Choices Behind Mainstream Models

    Despite a hundredfold-plus increase in parameters, the overall skeleton of today's leading open-source LLMs is still a stack of Transformer decoder layers. This article examines the architectural decisions behind that continuity: how absolute positional encoding gave way to rotary positional encoding (RoPE), multi-head attention (MHA) to grouped-query attention (GQA), GELU to SwiGLU, and LayerNorm to RMSNorm, alongside the role of mixture-of-experts (MoE) and the co-design of architecture and engineering. It also explains why cross-model architecture comparisons are so difficult, and what developers should weigh when choosing a stack: inference cost, long-context positional encoding strategies, KV Cache handling, and operator-level efficiency.

    Introduction: Seven Years of Evolution: Incremental Change or a Step Change?

    Since the first generation of autoregressive decoder-only architectures emerged, large language models (LLMs) have come a long way. Put the architecture diagrams of those early representatives side by side with those of today’s flagship open-source models and a striking pattern emerges: parameter counts have grown by orders of magnitude, yet the overall skeleton is remarkably similar, still a stack of Transformer decoder layers.

    Beneath that surface similarity, however, nearly every module has been refined:

    • Absolute positional encoding has gradually given way to rotary positional encoding (RoPE).
    • Standard multi-head attention has largely yielded to grouped-query attention (GQA).
    • Activation functions have moved from the earlier GELU to the gated SwiGLU.

    Scattered as these changes may seem, they share the same goal: to substantially reduce the compute cost of training and inference while preserving, or even improving, modeling capability.

    This article sets aside benchmark scores and training algorithms and focuses on architecture itself. It reviews the key structural decisions behind today’s leading text models and explains why each change was made.

    Why Comparing LLM Architectures Is So Difficult

    Before diving into the details, it helps to acknowledge one hard truth: comparing architectures side by side is notoriously difficult. The variables that shape final performance go well beyond network structure:

    • Training data: models differ widely in the scale, mixture, and cleaning strategy of their training data, which is often undisclosed.
    • Training techniques: learning rate schedules, data curriculum design, and regularization strategies vary from team to team.
    • Hyperparameters: these are rarely documented consistently and are hard to reproduce.

    As a result, even models with highly similar architectures can perform very differently. The value of this article therefore lies not in declaring which model is stronger, but in identifying the structural choices that leading teams have converged on and extracting engineering lessons that can be applied elsewhere.

    1. Positional Encoding: From Absolute Coordinates to Relative Rotation

    Transformer’s self-attention is inherently order-agnostic, so positional information must be injected explicitly.

    Absolute positional encoding assigns each position a fixed vector, either learned or computed, and adds it directly to the token embeddings. Its main weakness is poor extrapolation: once the inference sequence exceeds the training window, performance often drops sharply.

    Today’s mainstream approach is Rotary Position Embedding (RoPE). RoPE injects positional information into the query (Q) and key (K) vectors in relative form, using rotation matrices. Attention scores therefore naturally reflect the relative distance between tokens rather than their absolute coordinates, giving the model better length extrapolation.

    A simplified illustration:

    1. import torch
    2. def apply_rope(x, positions, dim):
    3. # x: [batch, seq_len, num_heads, head_dim]
    4. # Pair the first half and second half of each head dimension for rotation
    5. freq = 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))
    6. angles = positions.unsqueeze(-1) * freq # [seq_len, dim/2]
    7. cos, sin = angles.cos(), angles.sin()
    8. x1, x2 = x[..., ::2], x[..., 1::2]
    9. rotated = torch.stack([x1 * cos - x2 * sin,
    10. x1 * sin + x2 * cos], dim=-1)
    11. return rotated.flatten(-2)

    In practice, RoPE has also spawned a range of length-extrapolation enhancements, such as position interpolation and frequency scaling. These let models handle longer effective contexts even when trained on limited sequence lengths. For developers, choosing an architecture that supports RoPE and extrapolation enhancements is a key consideration for long-context workloads.

    2. Attention: Trading Expressiveness for Inference Efficiency

    Standard multi-head attention (MHA) maintains a separate key (K) and value (V) projection for each attention head. The problem: during autoregressive decoding, KV Cache size grows with the number of heads. The more heads, the more memory the cache consumes, which is especially costly for long sequences.

    To ease this bottleneck, the field has developed two main approaches:

    • Multi-Query Attention (MQA): all query heads share a single set of K/V. Memory usage drops substantially, at some cost to expressiveness.
    • Grouped-Query Attention (GQA): a middle ground that divides query heads into groups, with each group sharing one set of K/V. It strikes a better balance between inference memory and model quality.

    The table below summarizes the trade-offs among the three approaches:

    Approach KV Cache Size Expressiveness Typical Use Case
    MHA Large Strongest Training and offline inference with ample memory
    GQA Medium Strong Mainstream online inference (recommended default)
    MQA Smallest Moderate Deployments with extremely tight memory constraints

    GQA has become the default choice for current flagship models, a direct response to the reality of inference cost sensitivity.

    3. Activation Functions: The Rise of Gated Structures

    Early feed-forward networks (FFNs) typically used GELU or ReLU as the activation function. In recent years, gated structures have become mainstream, with SwiGLU as the canonical example.

    Unlike a single activation function, a gated FFN usually has three projection branches: the gate and up branches are each linearly transformed and then multiplied element-wise, before activation. The third, down branch completes the layer by providing the information pathway. The simplified form:

    1. def swiglu_ffn(x, w_gate, w_up, w_down):
    2. # Multiply the gate branch and up projection element-wise, then apply the down projection
    3. gate = torch.nn.functional.silu(x @ w_gate)
    4. return (gate * (x @ w_up)) @ w_down

    At equal parameter counts, this structure often delivers better modeling performance, at the cost of slightly more parameters and compute in the FFN. Mainstream architectures have also generally removed the bias terms from both the attention and feed-forward layers. Doing so simplifies the architecture and is friendlier to training stability and memory utilization.

    4. Normalization and Stable Training

    The choice of normalization layer directly affects whether large models converge stably. LayerNorm has largely been replaced in recent years by the lighter-weight Root Mean Square Normalization (RMSNorm). RMSNorm drops the mean-centering step and only rescales activations by their root mean square. This makes it cheaper to compute, and it has proven stable in large-scale training.

    Mainstream models also tend to use a Pre-Norm structure: each sub-layer, whether attention or feed-forward, is preceded by normalization, with a residual connection added afterward. Compared with Post-Norm, Pre-Norm yields smoother gradient flow in deep networks and is less prone to divergence during training.

    5. Sparsity and Co-Designed Engineering

    Beyond dense architectures, the Mixture of Experts (MoE) has emerged as another major technical route. MoE uses a gating network to route each token to only a few expert sub-networks, achieving “large total parameters, small active parameters” per token. This expands model capacity while keeping inference cost under control.

    These architectural innovations do not exist in isolation; they are deeply co-designed with engineering implementations:

    • KV Cache chunking and quantization determine whether long-context inference is feasible.
    • Fused attention operator implementations significantly affect training and inference throughput.
    • Sparse activation imposes new requirements on hardware and parallelism strategies.

    In other words, an architecture cannot be judged by its parameter counts on paper alone. It must be assessed together with its ability to be implemented efficiently.

    A Sober View Beyond Architecture

    Back to the question from the opening: do these improvements add up to a breakthrough? A more measured conclusion is that they look more like continuous refinement of existing foundations than a paradigm-level reconstruction. What ultimately determines model performance is still the joint optimization of data quality and training methods. Every architectural adjustment, at its core, seeks a better balance among capability, memory, and throughput.

    For engineering practitioners, this points to several takeaways:

    1. Prioritize inference cost when choosing a stack: GQA and RMSNorm, for example, are mature defaults.
    2. Pay attention to positional encoding for long-context workloads: prefer architectures that support RoPE and extrapolation enhancements.
    3. Account for KV Cache strategy when planning quantized deployments: it is often the true source of memory bottlenecks.
    4. Watch how architecture and operators work together: a good design must be paired with an efficient implementation.

    On cloud platforms, these capabilities are typically supported by general-purpose infrastructure such as containerized training and inference platforms, object storage, and logging, monitoring, and alerting. This infrastructure helps bridge architecture selection and engineering deployment.

    Conclusion

    From absolute positional encoding to rotary positional encoding, from multi-head attention to grouped-query attention, and from GELU to SwiGLU, LLM architecture has evolved along a clear shared direction: continuously unlocking efficiency gains in training and inference without sacrificing modeling capability. Understanding the trade-off logic behind these structural decisions is more valuable in the long run than memorizing the name of any single model. For developers, mastering this “capability, efficiency, and cost” trade-off framework is the most reliable anchor for navigating a fast-moving technology landscape.