From GPT to the Next Generation of Open-Source LLMs: A 2025 Overview of Architecture Evolution

    Large language models have changed dramatically in scale and capability, but their architectural foundation remains surprisingly stable. This article examines how Transformer-based LLMs have evolved from 2019 to 2025, focusing on RoPE, GQA, SwiGLU, MoE, RMSNorm, QK-Norm, and other component-level improvements. It explains how these refinements improve training stability, inference efficiency, and effective context length while preserving the Decoder-Only Transformer as the dominant design.

    1. Why the Transformer Remains the “Skeleton” Seven Years Later

    Nearly seven years have passed since early large language models based on the Transformer Decoder architecture emerged. When models from around 2019 are compared with flagship open-source models released in 2024 and 2025, one striking fact stands out: their high-level structures remain remarkably similar.

    They are all autoregressive, Decoder-Only architectures. They rely on multi-head self-attention to model long-range dependencies within a sequence, and they generally follow the same basic pattern:

    Token embeddings → stacked Transformer blocks → output projection

    This leads to an important conclusion: the Transformer’s core design has stood the test of time. Self-attention provides global sequence modeling, residual connections support stable gradient flow, and feed-forward networks deliver nonlinear representation capacity.

    Researchers have explored many alternatives over the past seven years, including pure state-space models, linear-attention variants, and sparse architectures beyond mixture-of-experts models. However, these approaches have either shown unstable performance on certain tasks or remain less mature in terms of engineering ecosystems, inference efficiency, or verified scaling behavior. As a result, none has displaced the Decoder-Only Transformer as the dominant architecture.

    That said, an unchanged macrostructure does not mean that progress has stopped. The real evolution has taken place in the architecture’s individual components. At scales ranging from tens of billions to trillions of parameters, seemingly minor changes can produce major differences in training stability, inference cost, and context length.

    2. The Evolution of Core Components

    2.1 Positional Encoding: From Absolute Positions to RoPE

    Early language models commonly used learned absolute positional embeddings, assigning a trainable vector to each position in the sequence. This approach was simple and direct, but it had several limitations:

    • Poor extrapolation: The maximum sequence length observed during training effectively became the upper limit at inference time. Performance could degrade sharply beyond that limit.
    • Parameter overhead: The positional embedding table grew linearly with the maximum sequence length.
    • Limited relative-position inductive bias: The model had to learn additional information about the distance between two tokens.

    Later approaches, including sinusoidal positional encoding and relative position bias, addressed some of these issues. Today, however, nearly all leading models use Rotary Position Embedding (RoPE).

    RoPE injects positional information into Query and Key vectors through rotational transformations. As a result, attention scores naturally depend on the relative positions of two tokens. Its main advantages include:

    1. A mathematical form equivalent to relative positional encoding, providing strong potential for length generalization;
    2. No additional trainable parameters;
    3. Good compatibility with the Key-Value Cache (KV Cache), which supports inference optimization.

    The industry has also developed several RoPE extensions. By interpolating rotation frequencies or adjusting the base frequency, these techniques allow models to extend their context windows from tens of thousands of tokens to the million-token range with relatively little additional fine-tuning data.

    This combination of architecture-level support and low-cost extension has become a major technical foundation for the rapid adoption of long-context models.

    2.2 Attention Mechanisms: From MHA to GQA and MLA

    Standard Multi-Head Attention (MHA) has a major engineering bottleneck during inference: KV Cache memory usage grows linearly with sequence length and batch size. With typical multi-head configurations, the cache can easily consume several to dozens of gigabytes, placing serious limits on throughput and concurrency.

    Grouped-Query Attention (GQA) has become the most widely used compromise. It allows multiple Query heads to share a group of Key and Value heads. For example, mapping 32 Query heads to 8 groups of KV heads reduces KV Cache usage to one quarter of the original size, while quality degradation is negligible in many evaluations.

    The evolution can be summarized as follows:

    • MHA: Each Query head has an independent Key and Value head. It generally provides the best quality but requires the most cache memory.
    • MQA: All Query heads share a single set of Key and Value heads. It minimizes cache usage but can cause a more noticeable quality drop.
    • GQA: Query heads share Key and Value heads in groups. It provides a practical balance and has become a de facto standard in flagship models since 2024.
    • Further compression: Some newer models are exploring methods that project Key and Value representations into a low-rank latent space before caching them. This approach aims to reduce cache size further while preserving model quality and represents a potential direction beyond GQA.

    2.3 Feed-Forward Networks and Activation Functions: The Rise of SwiGLU

    Early models typically used GELU as the activation function in a standard feed-forward network: the input was first expanded, passed through a nonlinear transformation, and then projected back to the model dimension.

    Modern architectures have largely shifted toward the SwiGLU gated structure:

    1. # Schematic SwiGLU feed-forward layer
    2. def swiglu_ffn(x):
    3. gate = W_gate @ x # Gating branch
    4. up = W_up @ x # Value branch
    5. return W_down @ (silu(gate) * up)

    The key innovation is the gating mechanism. One branch passes through the SiLU activation function and is then multiplied element-wise by another branch before the result is projected back to the model dimension.

    Both experimental results and engineering experience indicate that SwiGLU produces better loss curves under the same parameter budget. To keep the total parameter count comparable, models using SwiGLU usually reduce the hidden dimension from the traditional four times the model dimension to approximately 8/3 times the model dimension.

    Another major development in feed-forward networks is the Mixture-of-Experts (MoE) architecture. Instead of using one large feed-forward network, MoE divides the network into multiple experts. A router selects only a small number of experts for each token. This allows the model to increase its total parameter count while keeping the computation required for each individual token under control.

    Several flagship open-source models released in 2024 and 2025 have adopted MoE designs, making this architecture an increasingly common choice for very large models.

    2.4 Normalization Strategies: Pre-Norm, RMSNorm, and QK-Norm

    The location and form of normalization layers have a direct impact on training stability.

    • Post-Norm to Pre-Norm: Early models applied normalization after the residual connection, which could make deep networks difficult to train. Modern models generally move normalization before each sublayer, creating a Pre-Norm structure. This improves gradient flow and makes very deep stacks—sometimes exceeding one hundred layers—more practical.
    • LayerNorm to RMSNorm: RMSNorm removes mean centering and only performs scale normalization. It is simpler and faster to compute, with little or no loss in effectiveness, and has become the default choice in many modern architectures.
    • QK-Norm and related techniques: Some newer models additionally normalize Query and Key vectors inside the attention mechanism. This helps prevent attention logits from growing abnormally and can further improve stability during very long training runs.

    2.5 Other Notable Architectural Refinements

    Several additional changes have also become common:

    • Attention bias choices: Early models often added learnable biases to attention logits. Modern architectures commonly remove them and rely on RoPE for positional information.
    • Vocabulary and tokenization: Vocabulary sizes have grown from roughly 50,000 tokens in early systems to approximately 100,000–250,000 tokens. Multibyte tokenizers can improve token efficiency for multilingual text and code.
    • Parameter sharing and residual scaling: Techniques such as sharing weights between input embeddings and the output projection, as well as scaling the initialization of deep residual branches, are widely used to improve training stability.
    • Inference-aware design: Architectural decisions are increasingly made together with inference optimizations. GQA reduces KV Cache usage, RoPE supports context-length extrapolation, and MoE reduces computation per token. Each of these choices helps make large-scale deployment more economical.

    3. The Nature of This Evolution: Refinement Rather Than a New Paradigm

    Taken together, these changes suggest that large language models have not undergone a second major architectural revolution since 2019. Instead, they have experienced a process of continuous engineering refinement.

    Each change—RoPE, GQA, SwiGLU, RMSNorm, and MoE—may appear incremental when considered separately. Collectively, however, they serve three clear objectives:

    1. Greater training stability: Making it practical to complete training runs involving trillions of parameters and tens of trillions of tokens;
    2. Lower inference costs: Preventing deployment expenses from growing faster than model capabilities;
    3. Longer effective context: Supporting retrieval-augmented generation, long-document understanding, AI agents, and other higher-level applications.

    Architecture comparisons also have inherent limitations. Models differ significantly in their data mixtures, training methods, and hyperparameters, many of which are not fully disclosed. It is therefore difficult to attribute performance differences solely to architecture.

    Nevertheless, studying architectural changes remains valuable precisely because architecture is one of the most visible and reproducible aspects of a model. It provides a direct view of how the engineering community balances cost, stability, and capability.

    4. Conclusion

    For developers, understanding these architectural details is not merely an academic exercise. Whether you are selecting a model, planning a private deployment, fine-tuning a domain-specific system, or evaluating the feasibility of building a model in-house, properties such as RoPE’s length extrapolation, GQA’s cache requirements, and MoE’s activated-parameter ratio directly affect practical cost and performance decisions.

    The central theme of LLM architecture in 2025 is macro-level continuity combined with micro-level refinement. Understanding this principle provides a useful framework for reading the design documents of next-generation models.

    Future developments are likely to include further sparsification and linearization of attention, more sophisticated MoE routing strategies, deeper integration of unified multimodal architectures, and smaller architectures optimized for edge deployment. These innovations will continue to grow on top of the Transformer foundation that has been refined over the past seven years.