LLM Architecture in 2025: How RoPE, GQA, and SwiGLU Refined the Transformer

    The decoder-only Transformer remained the foundation of many large language models in 2025, but its components changed substantially. This article examines how rotary position embeddings encode relative position, grouped-query attention reduces inference memory, and SwiGLU reshapes the feed-forward network. It also explains how normalization supports stable training, why reasoning gains do not necessarily require new architecture, and what to consider when comparing or deploying models.

    1. A Familiar Architecture with Important Changes

    Compare a representative autoregressive language model from around 2019 with a flagship open-source model released in 2024 or 2025, and the resemblance is striking. Both are built around a stack of Transformer decoder blocks with multi-head self-attention, residual connections, and normalization.

    The familiar outline hides several consequential changes. Rotary position embeddings (RoPE) have largely replaced absolute position embeddings in newer open models. Grouped-query attention (GQA) has become a common alternative to standard multi-head attention (MHA), and gated SwiGLU feed-forward layers are increasingly used in place of conventional GELU-based layers. Each change reflects a different trade-off among computational efficiency, model capacity, and training stability.

    Architecture alone does not determine model quality. Data quality, training strategy, and hyperparameter choices vary widely and are often incompletely documented. Rather than ranking models by benchmark scores or unpacking training algorithms, this article focuses on what changed in the architecture and why.

    2. Position Encoding: From Absolute Embeddings to RoPE

    Self-attention does not inherently encode token order, so a Transformer needs a way to represent position. Early models commonly added a learned or sinusoidal position vector to each token representation. This treats position as a separate additive signal, and models can struggle to generalize when inference sequences exceed the lengths seen during training.

    Rotary position embeddings take a different approach. Rather than adding position vectors to token embeddings, RoPE applies position-dependent rotations to pairs of dimensions in the query (Q) and key (K) vectors. In the resulting query–key dot product, the positional contribution depends on the relative offset between the tokens, not on their absolute positions. The score still depends on the content of the query and key.

    1. # Illustrative pseudocode: rotate pairs of dimensions in Q or K.
    2. def apply_rope(x, position, theta=10000.0):
    3. dim = x.shape[-1] // 2
    4. # Each pair of dimensions uses a different rotation frequency.
    5. inv_freq = 1.0 / (theta ** (arange(0, dim) / dim))
    6. angles = position * inv_freq
    7. cos_angles, sin_angles = cos(angles), sin(angles)
    8. x_even, x_odd = x[..., ::2], x[..., 1::2]
    9. out_even = x_even * cos_angles - x_odd * sin_angles
    10. out_odd = x_even * sin_angles + x_odd * cos_angles
    11. return interleave(out_even, out_odd)

    RoPE encodes relative position without adding learned parameters and works with different attention variants. It can also support longer contexts when paired with interpolation or scaling strategies, although RoPE alone does not guarantee reliable extrapolation beyond the training context. These properties have made it a default choice in many contemporary open-source models.

    3. Attention: From Multi-Head to Grouped-Query Attention

    In standard multi-head attention, each query head has its own key and value heads. This gives each head independent representations, but during autoregressive decoding, the model must cache the keys and values from previous tokens. With other dimensions held constant, the size of this key–value cache, or KV cache, grows linearly with the number of key–value heads, putting pressure on GPU memory and memory bandwidth.

    Grouped-query attention keeps multiple query heads but lets each group share one key–value head. It sits between two designs:

    • Multi-head attention (MHA): Each query head has a corresponding key–value head, resulting in the largest KV cache of the three designs.
    • Multi-query attention (MQA): All query heads share a single key–value head, minimizing cache size but potentially limiting expressive capacity.
    • Grouped-query attention (GQA): Groups of query heads share key–value heads, balancing cache size against model capacity.
    1. # Illustrative mapping for evenly sized query groups.
    2. group_size = num_q_heads // num_kv_heads
    3. kv_head_index = q_head_index // group_size
    4. K, V = K_all[kv_head_index], V_all[kv_head_index]

    GQA can substantially reduce inference memory use and bandwidth pressure with little loss in model quality. This makes long-context workloads and high-concurrency serving more practical. Its adoption reflects a broader shift: a model must be economical to serve, not just feasible to train.

    Every Transformer block includes a feed-forward network (FFN) alongside attention. The FFN accounts for a large share of the model’s parameters and provides much of its nonlinear transformation capacity.

    Conventional FFNs commonly used ReLU or GELU between two linear projections. SwiGLU instead uses three projections: one produces a gate, one produces values, and the last projects their element-wise product back to the output dimension.

    1. # Standard FFN: out = W2(act(W1(x)))
    2. # SwiGLU FFN: out = W2(Swish(W1(x)) ⊙ W3(x))
    3. def swiglu_ffn(x, W1, W2, W3):
    4. gate = swish(x @ W1)
    5. value = x @ W3
    6. return (gate * value) @ W2

    The gate lets the network regulate which features pass through based on the input. SwiGLU can improve model quality at a comparable or smaller parameter budget, but the extra projection changes the cost at a given width. In practice, models using SwiGLU typically narrow the FFN’s intermediate dimension to keep parameter counts and computation in check.

    5. Normalization and Residual Paths Support Stable Training

    Several less visible design choices help keep deep Transformer stacks trainable:

    • Normalization placement: Pre-normalization, or Pre-Norm, applies normalization before the attention or FFN sublayer rather than after the residual addition. It generally makes deep networks easier to train and has become the more common choice. Some models also use lighter normalization schemes to reduce overhead.
    • Residual connections: Residual paths help gradients flow through deep stacks and remain a core part of modern LLMs.
    • Initialization and scaling: Careful weight initialization and residual-branch scaling can help mitigate exploding or vanishing gradients in deep networks.

    These choices may be less visible than a new attention mechanism, but they matter for whether a model can be trained reliably and efficiently.

    6. Reasoning Models Often Build on General-Purpose Architectures

    Many reasoning-focused models build on general-purpose flagship base models. Methods such as reinforcement learning can encourage longer chains of reasoning without requiring fundamentally new decoder modules. In these cases, the main change is in how the model is trained, not how its blocks are assembled. As core architectures converge, data, training methods, and engineering optimizations become more important sources of differentiation.

    For teams choosing a model, architectural considerations such as context support and inference cost matter alongside the maturity of the training and deployment toolchain. Container platforms can schedule training and inference workloads as demand changes, object storage can hold large datasets and model weights, and logging, monitoring, and alerting can help teams track both training metrics and production service quality.

    7. Why Architectural Comparisons Are Difficult

    It is hard to attribute performance differences between LLMs to architecture alone:

    • Training setups differ. Datasets, training methods, and hyperparameters can vary substantially, and public documentation is often incomplete.
    • Benchmarks measure different things. No single score captures how a model will perform across real workloads.
    • Implementation matters. Operator fusion, quantization, and parallelization strategies can significantly affect throughput and cost.

    The useful question is not which architecture ranks highest in isolation, but which trade-offs fit the workload. A smaller KV cache, for example, is particularly valuable when serving long contexts or handling many concurrent requests.

    8. Conclusion: A Stable Backbone, Refined Components

    By 2025, the broad Transformer decoder design had stabilized, while its individual components continued to evolve. RoPE encodes relative position and provides a basis for extending context. GQA makes inference more economical by shrinking the KV cache. SwiGLU uses gating to improve the FFN’s expressive efficiency, while Pre-Norm and residual paths help keep deep networks trainable.

    None of these refinements replaces the Transformer backbone, but together they change what is practical to train and deploy. For developers, understanding why each component was adopted—and what it trades off—is more useful than memorizing the configuration of any one model.