LLM Architecture Evolution in 2025: From Attention Mechanisms to Activation Functions
This article reviews the architectural evolution of decoder-only large language models, comparing early GPT-style designs with 2024–2025 flagship open models. It examines positional encoding, attention variants, normalization, activation functions, and mixture-of-experts layers, focusing on structural changes rather than benchmark results or multimodal design. A component comparison table and an outlook on future architectural directions are included.
1. Overall View of Architectural Evolution: What Changed and What Did Not
Several years have passed since the original GPT-style architecture based on the Transformer decoder was introduced. Comparing classic models from around 2019 with the latest open-source flagship models released in 2024–2025 reveals a striking fact: at the macro level, these models remain highly similar. The combination of autoregressive decoding, causal attention masks, residual connections, and stacked feedforward network layers has never been fundamentally overturned.
The changes have occurred at a finer-grained module level. Positional encoding has evolved from learnable absolute positional encoding to rotary positional encoding. Multi-head attention has gradually given way to grouped-query attention. The GELU activation function has been replaced by the more efficient SwiGLU. Normalization has also shifted from post-normalization to pre-normalization. Viewed individually, none of these improvements is a revolutionary breakthrough. Taken together, however, they create a generational difference in training stability, inference efficiency, and long-context capability in modern large models.
A question worth considering is this: are we still refining the same architectural foundation, or are we already standing at the threshold of a new paradigm? By analyzing architectural components one by one, this article attempts to provide an answer based on technical facts.
It should be noted that comparing the performance of different models is extremely difficult. Data composition, training recipes, and hyperparameter settings vary greatly and are often not fully disclosed. Therefore, this article deliberately avoids discussing benchmarks and training algorithms and focuses instead on structural changes in the architecture itself—an area where the community can form a relatively broad consensus. Multimodal capabilities involve separate topics such as vision encoders and cross-modal alignment. This article focuses only on architectural design related to text capabilities.
2. Positional Encoding: From Absolute Positions to RoPE
2.1 Why Change Positional Encoding?
Early models used learnable absolute positional encoding, assigning a trainable vector to each position. This approach is simple and direct but has an obvious flaw: the model cannot extrapolate to positions not seen during training, which limits long-context extension. Sinusoidal positional encoding offers some extrapolation ability, but its integration with content semantics is not flexible enough.
2.2 The Core Idea of RoPE
Rotary positional encoding (RoPE) encodes positional information as complex rotations, rotating query and key vectors according to their position angles. Its elegance lies in the fact that when two vectors are used in an attention inner product, the result depends only on their relative position difference. This naturally satisfies relative position invariance.
From an implementation perspective, the computation of RoPE can be summarized as follows:
# Core RoPE logic (illustrative)def apply_rope(x, positions):# Split vectors into 2D pairs along the feature dimensioncos, sin = precompute_rotations(positions) # Precompute rotation componentsx1, x2 = x[..., ::2], x[..., 1::2]rotated = complex_multiply(x1, x2, cos, sin)return interleave(rotated)
2.3 Extrapolation and Interpolation Techniques
Mainstream models in 2024–2025 generally introduce position interpolation or related frequency adjustment techniques on top of RoPE. By scaling rotation angles, the model can support context windows several times, or even tens of times, longer than the pretraining length. These techniques are simple to implement in engineering practice: usually only the frequency base or scaling factor needs to be adjusted, together with a small amount of continued training on long text. This is one of the key reasons long-context capability has spread rapidly in the past two years.
3. Attention Mechanisms: From MHA to GQA and Latent Attention
3.1 The Inference Bottleneck: Memory Pressure from the KV Cache
During autoregressive generation, the system must cache the key and value vectors of all historical tokens to avoid repeated computation. As model size, context length, and batch concurrency increase, KV cache memory usage often exceeds that of the model weights themselves, becoming the primary bottleneck for inference throughput.
3.2 Grouped-Query Attention
The idea behind grouped-query attention (GQA) is highly pragmatic: retain multiple query heads to preserve expressiveness, but substantially reduce the number of key-value heads. For example, multiple query heads can share one set of key-value heads. This reduces the KV cache size proportionally and significantly improves decoding throughput, while empirical evidence suggests only minimal damage to model quality. This design has now become standard in open-source flagship models.
3.3 More Aggressive Exploration
Some of the latest models further explore cross-layer KV cache sharing and approaches that decompose and compress key/value projections into a low-dimensional latent space before expanding them again, seeking a better balance between quality and cache size. These methods are more complex to implement, but they represent the frontier of attention efficiency optimization.
4. Normalization and Activation Functions: Refining Stability and Efficiency
4.1 Pre-Normalization Has Become the De Facto Standard
The original Transformer used post-normalization, where normalization is placed after the residual branch. Deep models trained this way are prone to gradient instability. Pre-normalization moves normalization to the input of each sublayer. Combined with residual connections, this creates a smoother gradient path and makes it possible to train networks more than one hundred layers deep. Modern large models also commonly freeze the scale parameters in normalization after training to further improve numerical stability during fine-tuning.
4.2 SwiGLU: The Victory of Gated Activation
Traditional feedforward networks use a single activation function. SwiGLU transforms the feedforward layer into a two-branch gated structure: one branch computes the gate signal, while the other carries content information, and the two are multiplied element by element. Empirical results show that under the same parameter budget, the gated structure delivers consistent quality improvements, even though it requires an additional projection matrix in the feedforward layer. Today, SwiGLU-like structures have nearly replaced all classic activation choices.
5. Mixture of Experts: Conditional Computation as a Scaling Path
The most notable structural change in 2024–2025 flagship models is the widespread adoption of mixture-of-experts (MoE) architectures. The core idea is to replace the feedforward network in each Transformer layer with multiple parallel “expert” subnetworks, with a router dynamically selecting a small number of experts—usually several—for each token to participate in computation.
The advantage is that it decouples total parameter count from computation. The total number of model parameters can scale to extremely large sizes, while each token activates only a small subset of parameters, keeping training and inference computational costs within a manageable range. In engineering practice, expert parallelism—distributing different experts across different devices—has become the third mainstream parallelization strategy alongside data parallelism and tensor parallelism.
MoE also introduces new challenges. Router load balancing, training stability, and expert capacity planning all require carefully designed auxiliary losses and infrastructure support. Overall, however, sparse activation has been shown to be the most cost-effective scaling path currently available.
6. Architecture Comparison Overview
| Component | Early GPT-Style Architecture | Mainstream 2025 Flagship Architecture |
|---|---|---|
| Positional encoding | Learnable absolute positional encoding | RoPE + length extrapolation/interpolation |
| Attention | Multi-head attention | Grouped-query attention / latent compression approaches |
| Normalization | Post-layer normalization | Pre-normalization (RMS form) |
| Activation function | GELU | SwiGLU gated structure |
| Feedforward network | Dense | Sparse mixture of experts |
7. Conclusion and Outlook
Looking back at seven years of architectural evolution, we can draw a cautious conclusion: the Transformer decoder skeleton has not been overturned, but every peripheral component around it has undergone systematic reconstruction. Positional encoding addressed length generalization; attention variants addressed the inference memory bottleneck; normalization and activation functions addressed training stability; and MoE addressed the cost of scaling. Together, these improvements explain why today’s models can carry longer contexts and larger parameter counts with greater efficiency.
Looking ahead, a true architectural paradigm shift may come from long-sequence modeling approaches that replace attention, linear-complexity attention, and hybrid architectures combining these methods with conventional Transformers. These approaches are still in a catch-up phase, but the architectural landscape of 2025 already shows that gradual refinement and paradigm exploration can proceed in parallel. For developers and researchers, understanding the principles and motivations behind these component-level improvements is a necessary foundation for grasping the next generation of model designs.
