LLM Core Architecture Evolution in 2025: RoPE, GQA, RMSNorm, SwiGLU, and MoE Explained

    Nearly seven years after modern large-scale pretraining architectures first appeared, 2025's flagship open-source models still rest on the same skeleton as the early designs: token embeddings, a stack of Transformer layers, and an output projection. What changed is what happens inside those layers. This article maps the structural evolution of LLM architectures: positional encoding from absolute embeddings to RoPE, attention variants from MHA to MQA, GQA, and MLA, normalization and residual placement from LayerNorm to RMSNorm and from Post-Norm to Pre-Norm, the shift to SwiGLU in the feed-forward network, sparse scaling with Mixture-of-Experts, long-context techniques built on KV cache compression, and how reasoning models reuse the same backbone. It also explains why controlled ablation is the only reliable way to judge whether an architectural change actually helps, and what developers need to understand for model selection and inference optimization.

    It has been roughly seven years since modern large-scale pretraining architectures were first proposed. Put an architecture from around 2019 next to a 2025 flagship open-source model and something interesting appears: at the skeleton level, the two are still highly similar. Both are built on the same backbone of token embeddings, a stack of Transformer layers, and an output projection. What actually changed is the internal components and the efficiency work around them.

    Scientifically judging how much those architectural differences contribute to model capability is a problem the field widely recognizes as hard. Dataset composition, training strategy, and hyperparameter configuration differ enormously between models, and complete, reproducible disclosure is often missing. This article therefore does not discuss benchmark scores or training algorithms. It focuses on the structural evolution of the architecture itself, giving developers a clear map of that evolution and a way to distinguish changes that solidify the foundations from changes that break the paradigm.

    1. Positional Encoding: From Absolute Positions to Rotary Embeddings

    Self-attention in a Transformer is permutation-invariant, so the job of positional encoding is to inject order information. Its evolution falls into roughly three stages:

    • Absolute positional encoding: early designs added a fixed position vector directly to token embeddings. Simple to implement, but limited in length extrapolation; performance drops sharply once the input exceeds the pretraining length.
    • Learned position embeddings: positions are mapped to trainable parameters. Slightly better within the pretraining range, but still bounded by the maximum sequence length set during training.
    • Rotary Position Embedding (RoPE): the current mainstream approach. It applies a position-dependent phase to query and key vectors through a rotation matrix, so attention scores naturally carry relative position information. This makes it more robust for length extrapolation and long-context scenarios.

    The core idea of RoPE can be understood from the sketch below:

    1. import torch
    2. def apply_rope(x, positions, head_dim):
    3. # Pair the vector dimensions into complex-like pairs and apply a rotation phase by position
    4. x = x.view(*x.shape[:-1], head_dim // 2, 2)
    5. angles = positions.unsqueeze(-1).unsqueeze(-1) # angle for each position
    6. cos, sin = torch.cos(angles), torch.sin(angles)
    7. x_even, x_odd = x[..., 0], x[..., 1]
    8. rotated_even = x_even * cos - x_odd * sin
    9. rotated_odd = x_even * sin + x_odd * cos
    10. return torch.stack([rotated_even, rotated_odd], dim=-1).flatten(-2)

    The value of relative position encoding is that the model does not need to memorize independent parameters for every absolute position. It only needs to understand one relative relationship, “how far apart” two tokens are, which fits variable-length sequences and longer context windows naturally.

    2. Attention Mechanisms: Balancing Expressiveness and Cache Efficiency

    Standard multi-head attention (MHA) maintains separate query, key, and value projections for each head. Expressiveness is strong, but during inference the historical keys and values that must be cached (the KV cache) grow linearly with the number of heads, which is expensive in GPU memory. To compress that overhead without giving up quality, the field has followed a clear technical path:

    • Multi-Query Attention (MQA): all query heads share a single set of keys and values. The KV cache shrinks substantially, at a small cost in expressiveness.
    • Grouped-Query Attention (GQA): a compromise between MHA and MQA. Query heads are divided into groups, and heads within a group share keys and values. It has become the default choice in current flagship models, striking a good balance between cache efficiency and quality.
    • Multi-head Latent Attention (MLA): compresses keys and values with a low-rank projection and caches only the resulting latent variables. This pushes KV cache usage down further while mitigating the accuracy loss introduced by MQA.

    3. Normalization and Residual Connections: Engineering Training Stability

    • From LayerNorm to RMSNorm: RMSNorm drops mean centering and scales only by the root mean square. It is cheaper to compute and numerically more stable, and is now the standard configuration for the FFN and attention sub-layers in mainstream models.
    • From Post-Norm to Pre-Norm: Pre-Norm places normalization before the sub-layer computation, keeping the residual main path cleaner and gradient flow more stable. Deep networks become easier to train and less sensitive to the learning rate.

    4. Activation Functions and Feed-Forward Networks: Why SwiGLU Won

    Inside the feed-forward network (FFN), the activation function has evolved from ReLU/GELU to SwiGLU. SwiGLU uses a gated structure — a Swish activation multiplied element-wise with a linear projection — and delivers stronger expressiveness at the same parameter count. One common engineering detail: adding the gate doubles the FFN parameter count, so the hidden dimension is typically scaled by roughly 8/3 to keep the parameter budget comparable to a classic FFN. Architectural evolution, in other words, is usually a precise trade-off between quality gains and parameter budget.

    5. Mixture of Experts (MoE): The Sparse Path to Scale

    Mixture of Experts replaces a dense FFN with multiple expert sub-networks plus a router, enabling sparse scaling in which the total parameter count is large while the parameters actually activated per token stay small. The engineering difficulty concentrates in three places: the routing strategy (how to pick experts for each token), load balancing (avoiding overload on a few experts), and expert parallelism (how to partition and communicate efficiently across multiple devices). MoE gives a model larger knowledge capacity while keeping inference cost controllable, and it remains an important direction for scaling large models.

    6. Long Context and the KV Cache: Architecture Working With Engineering

    The KV cache is the main consumer of GPU memory during inference. GQA and MLA compress the cache directly at the architectural level. On top of that, the field combines attention sparsification, sliding windows, and enhancements to positional encoding extrapolation, together pushing context windows from a few thousand tokens to hundreds of thousands and beyond. It is worth noting that long-context capability rarely comes from a single architectural component. It is usually the combined result of positional encoding, attention variants, cache optimization, and the length of the training data.

    7. Architectural Characteristics of Reasoning Models

    A class of “reasoning models” focused on strengthening chain-of-thought reasoning is typically built on the architecture of a same-generation foundation model. The differences show up mainly in how the post-training stage constructs data and optimizes the model, not in the model structure itself. This reinforces a judgment: on a mature backbone, unlocking capability through training and post-training is often more cost-effective than redesigning the structure.

    8. The Methodological Difficulty of Comparing Architectures

    To determine whether an architectural change genuinely helps, the most reliable approach is an ablation under controlled variables: fix the dataset and training budget, swap only the target component, and observe convergence speed and downstream task performance. Comparing structural differences between different models in isolation easily produces misleading conclusions, because the observed differences may come from data, compute, or tuning rather than from the architecture. For developers, understanding the mechanism behind components such as RoPE, GQA, RMSNorm, SwiGLU, and MoE is more valuable than memorizing the parameter configuration of a particular model.

    9. The Engineering Perspective

    In real projects, teams usually do not need to reimplement all of these components from scratch. With general-purpose model services on cloud platforms, containerized deployment, centralized logging, and monitoring and alerting, developers can complete model selection, batch evaluation, elastic inference deployment, and continuous iteration at low cost, and concentrate their effort on business fit and quality validation instead of rebuilding low-level structure.

    10. Conclusion

    Overall, architectural evolution in 2025 looks more like continuous polishing of a mature skeleton than a rewrite. Rotary position embeddings, grouped-query attention, RMSNorm, Pre-Norm, SwiGLU, and mixture of experts have gradually settled into default configurations. What they deliver is mostly steady improvement in efficiency and stability rather than an upheaval of the basic paradigm. Any true paradigm shift still needs time to prove itself. For developers, mastering the design trade-offs behind these general-purpose components is the foundation for sound architectural choices, inference optimization, and iterative quality improvement.