Mainstream LLM Architecture Evolution in 2025: A Deep Dive

    This article takes a deep look at how mainstream open-source LLM architectures have evolved by 2025, comparing the early base models of 2019 (the GPT-2 generation) with flagship open models like DeepSeek V3 and LLaMA 4. It argues that the Transformer decoder backbone has stayed intact, with real change concentrated in three areas: position encoding (moving to RoPE with extrapolation schemes), attention (moving to GQA and low-rank KV compression), and feed-forward/parameter organization (SwiGLU and MoE). The article walks through each module, explains how they work together, compares common design trade-offs, offers a model-selection checklist, and describes a load-test–observe–optimize workflow for taking an architecture from experiment to production.

    1. Introduction: Seven Years Later, Has the LLM “Skeleton” Changed?

    About seven years have passed since the first wave of large autoregressive language models were released. Comparing the base models that appeared around 2019 (the GPT-2 generation) with the flagship open-source models released between late 2024 and 2025 (such as DeepSeek V3 and LLaMA 4) reveals an intriguing pattern: their overall structure remains highly similar.

    Both adopt the Transformer decoder architecture, with causal self-attention at the core. Each token passes through multiple stacked layers, predicts only the next token at a time, and is trained autoregressively with a next-token prediction objective. However, local designs have changed noticeably:

    Design dimension Early base models Current mainstream flagship models
    Position encoding Learnable absolute position embeddings Rotary Position Embedding (RoPE), with long-context extrapolation schemes
    Normalization LayerNorm RMSNorm, mostly Pre-Norm
    Attention Multi-Head Attention (MHA) Grouped-Query Attention (GQA) or variants like low-rank KV compression
    Activation function ReLU, GELU Gated linear units such as SwiGLU
    Parameter organization Dense Dense, or sparse Mixture-of-Experts (MoE)
    Context length 512–1024 tokens 128K and beyond

    This raises a question worth answering seriously: are these changes genuine breakthroughs, or just refinements to the same architectural foundation?

    My view is that they are mostly “solving new problems within the existing framework.” Over the past few years, developers have had to address long context, expensive inference costs, large-scale training stability, and higher data efficiency. The responses have primarily been engineering modifications rather than a complete rewrite. However, sparsification (MoE) and latent-space attention compression have already reached the structural level and can be seen as the directions closest to a “structural breakthrough.”

    There is also an easily overlooked point: position encoding, attention, normalization, and activation functions are not independent of one another. For example:

    • GQA dramatically shrinks the KV cache, making long-context inference economically feasible;
    • RMSNorm reduces normalization overhead and makes it easier to train deep networks stably;
    • RoPE, once combined with GQA, sparse attention, and other schemes, is what supports context windows of hundreds of thousands of tokens.

    So the value of comparing component by component lies in understanding how the modules work together, rather than judging in isolation whether a single component is “more advanced.”

    Fairly comparing LLMs is very difficult. Training data, training methods, and hyperparameters vary widely and are often not disclosed in detail; differences in hardware, quantization schemes, and inference engines also significantly affect measured performance. However, structural information about the architecture itself is relatively transparent and can usually be read directly from the model configuration. Therefore, this article does not discuss benchmark scores or training algorithms. Instead, it focuses on “what current flagship open-source models do at the architecture level,” giving an overall overview first and then breaking it down layer by layer.

    Note: In early 2025, the release of DeepSeek R1 drew wide attention. Built on the architecture introduced by DeepSeek V3 in December 2024 and optimized for reasoning tasks, R1 is discussed here because V3 was adopted at scale only in 2025.

    2. Overall Architecture Overview: Understanding the Modern LLM at a Glance

    The main structure of a mainstream modern LLM can be summarized by the diagram below (M = hidden dimension, H = number of attention heads, d_h = per-head dimension, E = number of experts):

    1. Input Token IDs
    2. │
    3. ▼
    4. ┌──────────────────────────────┐
    5. │ Token embedding layer │ ← Optional: weight sharing with the output layer
    6. └──────────────────────────────┘
    7. │
    8. ▼
    9. ┌──────────────────────────────┐
    10. │ Transformer Block × N layers │
    11. │ ┌────────────────────────┐ │
    12. │ │ x = x + Attention( │ │ ← Causal self-attention, Q carries RoPE
    13. │ │ RMSNorm(x)) │ │ all layers use GQA / MLA
    14. │ └────────────────────────┘ │
    15. │ ┌────────────────────────┐ │
    16. │ │ x = x + FFN( │ │ ← First N layers are dense SwiGLU
    17. │ │ RMSNorm(x)) │ │ later layers are MoE (with shared experts)
    18. │ └────────────────────────┘ │
    19. └──────────────────────────────┘
    20. │
    21. ▼
    22. ┌──────────────────────────────┐
    23. │ RMSNorm → Linear → Vocabulary probabilities │ ← Optional MTP prediction head
    24. └──────────────────────────────┘

    Note that this diagram is a generalized illustration rather than a one-to-one match to any specific model’s internal configuration. However, the basic skeleton of mainstream models can be described by it: positional information is injected via RoPE; attention uniformly uses GQA or MLA; normalization is uniformly RMSNorm; the feed-forward network is uniformly SwiGLU; and MoE is commonly used as the backbone, with some models also using MTP for auxiliary training. Next, we break it down module by module.

    3. Position Encoding: From Absolute Positions to Rotary Position Embedding

    Self-attention is inherently “permutation equivariant”: if you shuffle the input order without adjusting the attention mask, the model’s prediction for the same position stays unchanged (the causal mask in a decoder does adjust with position). Therefore, positional information must be injected explicitly.

    3.1 Comparison of Three Generations

    Absolute position encoding. Early approaches fell into two categories: one used fixed sine/cosine functions; the other (such as GPT-2) used learnable position embedding vectors added to the word embeddings. The advantage is simplicity and intuitiveness; the drawback is poor extrapolation: the model only saw a fixed length during training, and beyond that there are no corresponding learnable vectors, so performance drops noticeably.

    Relative position schemes. Later research made attention focus on “how far apart” tokens are rather than “which position” they are, such as learnable relative position biases. These generalize better to length, but the implementation is complex, and the extra parameters increase memory overhead.

    Rotary Position Embedding (RoPE). RoPE incorporates positional information into the Q and K vectors through “rotation”:

    • Dimensions are grouped pairwise, treating each 2D subspace as a planar vector;
    • A token at position m rotates its own Q/K by a position-dependent angle;
    • When the two sets of vectors are inner-producted, the result depends only on the relative distance m − n.

    RoPE combines the implementation simplicity of absolute position encoding with the relative nature of relative position encoding, and has become the current mainstream choice. Long context, however, remains its weakness.

    3.2 How RoPE Works: Why the Inner Product Depends Only on Relative Distance

    Consider a query vector q and a key vector k in a 2D subspace. After rotations by positions m and n, they become R(m)q and R(n)k. Their inner product is:

    1. ⟨R(m)q, R(n)k⟩ = qᵀ R(m)ᵀ R(n) k = qᵀ R(n − m) k

    In other words, after the rotation matrices of the two positions are multiplied, only the relative angle remains, so the attention score naturally depends only on the relative distance.

    Further, a d-dimensional vector can be split into d/2 two-dimensional subspaces. The rotation angle for group i is m·θ_i, where θ_i = base^(−2i/d), and base is usually 10000. Lower dimensions (larger i) rotate faster and are responsible for capturing short-range relationships; higher dimensions rotate more slowly and are responsible for capturing long-range relationships. This is consistent with the multi-scale positional encoding idea used in fields such as speech.

    1. def build_rope(seq_len, head_dim, base=10000.0):
    2. """Build the RoPE cos/sin table: low dims rotate fast (short range), high dims rotate slowly (long range)"""
    3. half = head_dim // 2
    4. inv_freq = 1.0 / (base ** (torch.arange(0, half, dtype=torch.float32) / half))
    5. angles = torch.outer(torch.arange(seq_len, dtype=torch.float32), inv_freq) # [T, half]
    6. return angles.cos(), angles.sin()
    7. def apply_rope(x, cos, sin):
    8. """Apply rotation to Q or K of shape [B, H, T, head_dim]; V is not rotated"""
    9. x1, x2 = x.chunk(2, dim=-1) # front and back halves of the dims
    10. cos = cos[None, None, :, :].to(x.dtype)
    11. sin = sin[None, None, :, :].to(x.dtype)
    12. return torch.cat([x1 * cos - x2 * sin,
    13. x1 * sin + x2 * cos], dim=-1)

    It is worth emphasizing that RoPE usually applies only to Q and K, not to V. RoPE is an orthogonal transformation that does not change the vector length, keeping the scale of the inner product stable, which is friendlier to training. Learnable position embeddings, in contrast, introduce different norms for different positions — one of the key reasons RoPE is widely adopted.

    3.3 The Key to Going Beyond the Training Length: Extrapolation and Interpolation

    RoPE does not natively support inference beyond the training length. Intuitively, when the rotation angle of a 2D subspace reaches an integer multiple of π, tokens at different positions get identical or even opposite representations. The high-frequency components beyond the training length have barely been “seen” during training, so the model tends to fail when extrapolating directly.

    Common industry approaches fall into three categories:

    1. Position interpolation: map out-of-range integer positions back into the training-length range, equivalent to compressing the position scale;
    2. Frequency scaling: adjust base, or sample non-uniformly by wavelength, so that high-frequency dimensions vary more gently;
    3. Band/module-specific adjustment: apply temperature scaling to attention, or use local window attention in some layers, forming a hierarchical “global + local” structure.

    All of these schemes require a moderate amount of continued long-text training (long-context extension) to work stably.

    3.4 Engineering Tips

    • Check base and scaling parameters: the base, scaling factors, and other parameters used at inference must match the model’s training configuration, otherwise long-text performance will noticeably degrade;
    • Watch the cache layout: since RoPE acts directly on K in the cache, optimizations such as KV quantization and paged caching (e.g., PagedAttention-style paged caches) must confirm compatibility with the rotation order;
    • Long context is a combined effort: long-context capability depends on the combined effect of position-encoding extrapolation, attention implementation, memory management, and training strategy — it cannot be attributed to RoPE alone.

    4. Attention Mechanism: From MHA to GQA and Latent Compression

    4.1 The Bottleneck in Autoregressive Inference Is Memory Bandwidth

    During training, layer activations can be recomputed, so the process is compute-bound. But autoregressive decoding generates only one token at a time, and every step must read all model weights and the KV cache — a typical memory-bandwidth-bound scenario. The larger the model, the longer the context, and the more concurrent requests, the faster the KV cache grows, directly limiting serving throughput.

    The KV cache size can be estimated as:

    1. KV cache bytes ≈ 2 × L × H_kv × d_h × B × S × bytes
    2. # 2 for one copy each of K and V; L = number of layers; H_kv = number of KV heads; d_h = per-head dimension
    3. # B = batch; S = sequence length; bytes = bytes per element (FP16/BF16 = 2)

    Here, 2 × L × H_kv × d_h is the “cache width per token,” which is determined by the model structure.

    4.2 Three Attention Variants

    • MHA (Multi-Head Attention): each query head has its own K and V heads, offering the strongest representational power and the largest cache.
    • MQA (Multi-Query Attention): all query heads share one set of K and V, shrinking the cache to 1/H of the original (H = number of heads), though quality may drop.
    • GQA (Grouped-Query Attention): a compromise. Query heads are divided into several groups, and each group shares one set of K and V.

    GQA can be seen as an interpolation between MHA and MQA: when the number of groups equals the number of heads, it is MHA; when it equals 1, it is MQA. Most models adopt GQA for the following reasons:

    • The cache width shrinks to 1/g of MHA (g = number of query heads per group), yielding significant benefits for long context and high concurrency;
    • Compared with MQA, the quality loss is small;
    • At inference, you only need to repeat K/V g times along the head dimension to align with Q, which is simple to implement.
    1. def expand_kv(k, v, n_rep):
    2. """Repeat each group's K/V n_rep times along the head dimension to align with Q heads (illustrative)"""
    3. if n_rep == 1:
    4. return k, v
    5. b, h_kv, s, d = k.shape
    6. k = k[:, :, None].expand(b, h_kv, n_rep, s, d).reshape(b, h_kv * n_rep, s, d)
    7. v = v[:, :, None].expand(b, h_kv, n_rep, s, d).reshape(b, h_kv * n_rep, s, d)
    8. return k, v

    4.3 Going Further: Low-Rank Joint Compression

    GQA compresses the “number of heads” of K/V while keeping the dimension of each head unchanged. A more aggressive approach is to compress the “dimension”: introduce a low-rank latent vector outside attention, project K and V from it, cache only the latent variables at inference, and compute attention directly in the latent space. Its key points include:

    • At inference, the up-projection matrix is “absorbed” into the query projection and the output projection, so there is no need to explicitly reconstruct the full K/V;
    • Positional information is introduced separately through a decoupled RoPE path;
    • Cache width = latent-variable dimension + decoupled RoPE dimension, which is smaller than GQA and more expressive than MQA.

    This class of methods belongs to the same “reduce the KV cache” idea as GQA; the difference lies in the compression axis and whether it relies on matrix absorption. The essence of choosing among them is a trade-off between KV cache, compute overhead, implementation complexity, and ecosystem compatibility.

    4.4 Inference-Stage Optimizations: Paged Caching and Fused Operators

    • Paged KV cache management: similar to OS virtual-memory paging, the cache is split into fixed-size blocks allocated on demand, reducing fragmentation and reserved waste, and making prefix sharing (such as system prompts) easier.
    • Fused attention operators: compute softmax in blocks to avoid writing the huge attention matrix back to memory, reducing memory usage and memory accesses — an important prerequisite for feasible long context.
    • Quantization: storing the KV cache in INT8/FP8 further reduces bandwidth pressure, but its impact on quality must be evaluated.

    4.5 New Pressure from Reasoning Models

    Reasoning models like DeepSeek R1 largely follow the base model in architecture, with changes concentrated in post-training (large-scale reinforcement learning, and so on). But they output very long chains of thought — a single request can easily run into thousands or even tens of thousands of tokens:

    • KV cache usage multiplies, and decode accounts for a larger share;
    • The compute cost per request increases, making service cost sensitive;
    • They place higher demands on attention efficiency, KV compression, and cache management.

    This explains why mainstream architectures in 2025 widely adopt GQA or stronger KV compression schemes.

    5. Normalization and Feed-Forward Networks: RMSNorm and SwiGLU

    5.1 From LayerNorm to RMSNorm

    LayerNorm subtracts the mean and divides by the standard deviation of each token’s features, then applies scaling and shifting:

    1. y = (x − mean(x)) / sqrt(var(x) + ε) * γ + β

    RMSNorm scales only by the root mean square, without subtracting the mean and without a bias:

    1. y = x / sqrt(mean(x²) + ε) * γ

    It has three advantages:

    1. It skips the mean computation and subtraction, reducing computation;
    2. It has no bias parameters;
    3. Practice shows comparable quality, so it has become the mainstream configuration.

    RMSNorm is usually placed before sublayers (Pre-Norm), used with residual connections to make gradients in deep networks more stable. Some models adopt QK-Normalization, normalizing Q and K within attention to stabilize attention scores and avoid numerical explosion.

    5.2 From GELU to SwiGLU

    A classic feed-forward network (FFN) has two linear transformations with an activation function in between. The activation function evolved from ReLU → GELU → the Gated Linear Unit (GLU) family, with SwiGLU now mainstream:

    1. FFN_SwiGLU(x) = (Swish(xW_gate) ⊙ xW_up) W_down
    2. Swish(x) = x · sigmoid(βx), commonly β = 1 (i.e., SiLU)

    Compared with GELU, SwiGLU splits the hidden layer in two:

    • one path passes through the activation function and acts as a “gate”;
    • the other passes through directly;
    • the two are multiplied element-wise.

    This multiplicative gating brings stronger feature selection and representational power. Its cost is three weight matrices: if the intermediate dimension is kept unchanged, the parameter count is about 1.5× the original, so in practice the intermediate dimension is usually reduced to about 2/3 of the original, keeping the total parameter count roughly on par with the GELU version.

    1. def ffn_swiglu(x, w_gate, w_up, w_down):
    2. gate = F.silu(x @ w_gate) # gate branch
    3. up = x @ w_up # data branch
    4. return (gate * up) @ w_down # element-wise multiply, then project down

    6. Mixture of Experts (MoE): Sparsification Becomes the Mainstream for Flagship Models

    6.1 Dense vs. Sparse

    Every token in a dense model passes through all parameters. MoE replaces the feed-forward network with multiple parallel “expert” subnetworks, and a gating network selects one or a few experts for each token. The model’s total parameters can be very large, but each token activates only a small subset.

    Both DeepSeek V3 and LLaMA 4 adopt MoE. V3 uses a “fine-grained experts + shared experts” design: experts are first split more finely, then always-active shared experts are added to carry general knowledge. This decouples parameter count from computation:

    • Total parameters determine knowledge capacity and memory usage;
    • Activated parameters determine the compute cost per token.

    This is the core reason MoE quickly became standard in flagship models: it enlarges model capacity while keeping inference cost controllable.

    6.2 Routing, Load Balancing, and Communication

    1. ┌── Expert 1 ──┐
    2. Token ─► Gate ─┼── Expert 2 ──┼─► Weighted sum ─► Output
    3. (Top-K routing)└── Expert E ──┘

    MoE’s engineering challenges are mainly three:

    • Load balancing: if a few experts are over-selected, the rest are under-trained and overall capacity is wasted. Common approaches include: adding a load-balancing auxiliary loss to the training objective; setting a bias term for each expert to dynamically adjust routing scores (no gradients needed).
    • Capacity factor and token dropping: each expert can process a limited number of tokens per batch. Beyond that, tokens are dropped, or a capacity factor reserves headroom — a trade-off between throughput and quality.
    • Distributed communication: when experts are spread across devices, two All-to-All communications are needed to dispatch and collect tokens, and communication can easily become the bottleneck. It must be combined with strategies such as expert parallelism, pipeline parallelism, and computation-communication overlap.

    6.3 New Problems Introduced by MoE

    • Memory usage: even when only some experts are activated, all expert weights must reside in memory, and memory capacity is counted by total parameters;
    • Inference scheduling: different experts have different popularity, and dynamic routing makes batching more complex;
    • Fine-tuning cost: updating expert parameters involves more communication, making full-parameter fine-tuning expensive, so parameter-efficient fine-tuning is often used.

    MoE’s benefit lies not in a faster single token, but in “larger capacity at the same per-token cost,” so it must be evaluated together with KV compression, quantization, parallelism strategies, and so on.

    7. Other Design Details Worth Noting

    • Weight tying: the input embedding and output projection share weights, reducing parameters and memory usage;
    • Vocabulary design: a reasonably larger vocabulary improves compression, so the same text maps to fewer tokens, lowering training and inference costs; the larger the vocabulary, the greater the output-layer and softmax overhead, so a balance is needed;
    • QK-Normalization: normalizing Q and K stabilizes attention scores and makes it easier to train deep, wide-FFN networks stably;
    • MTP (Multi-Token Prediction): predicting several subsequent tokens at each position as an auxiliary training objective, forcing representations to plan ahead and possibly improving representation quality. Some schemes also combine with speculative decoding at inference.

    Individually, these details have limited impact, but combined they determine whether a model can be trained stably to a very large scale.

    8. Architecture Comparison and a Selection Evaluation Checklist

    8.1 Comparison of Mainstream Design Trade-offs

    Dimension Common approach Main benefit Main cost
    Position encoding Learnable absolute Simple and intuitive Poor extrapolation
    RoPE Relative positions, simple to implement Requires extrapolation beyond training length
    Normalization LayerNorm Classic and stable Slightly heavier compute
    RMSNorm Saves compute, easy to scale Quality depends on overall configuration
    Attention MHA Strong representational power Largest KV cache
    GQA Significantly smaller cache, small quality loss Slightly lower representational power
    Low-rank latent compression Smallest cache Complex implementation
    Feed-forward network GELU Mature and stable Average parameter efficiency
    SwiGLU Stronger representational power Requires adjusting the intermediate dimension
    Parameter organization Dense Simple to implement and deploy Limited capacity at the same compute
    MoE Decouples capacity from computation Large memory, complex communication

    8.2 Selection Evaluation Checklist

    1. Context length requirements: clarify how much context the task needs. When exceeding the training length, confirm whether the model has undergone long-context extension and which extrapolation scheme it uses.
    2. Inference cost structure: chat applications have a high decode share, so prioritize KV cache width and memory bandwidth; long-document summarization and RAG applications have a high prefill share, so prioritize compute utilization and parallelism strategies.
    3. Deployment resource constraints: an MoE model’s memory requirement is counted by total parameters. When GPU memory is insufficient, consider quantization or multi-GPU parallelism, but evaluate the communication overhead.
    4. Fine-tuning and extensibility: dense models have a mature fine-tuning path; MoE full-parameter fine-tuning is costly, so consider parameter-efficient fine-tuning.
    5. Ecosystem compatibility: confirm the support of inference frameworks, quantization schemes, and hardware backends for the target architecture (such as a specific KV compression method), to avoid “works great on paper but can’t run in production.”
    6. Stability: watch for issues like loss spikes and abnormal gradient norms, and introduce stabilization measures such as QK-Normalization when necessary.

    9. Engineering Deployment: From Architecture to Observable Performance

    Architecture analysis ultimately has to land on measurable metrics. It is recommended to build a “load test — observe — optimize” loop:

    • Load test: run benchmarks with a request distribution close to real workloads (input length, output length, concurrency), recording time to first token (TTFT), per-request throughput, total system throughput, memory usage, and KV cache utilization;
    • Observe: split latency into prefill and decode, and use memory and bandwidth metrics to locate the bottleneck;
    • Optimize: adjust item by item according to the bottleneck, for example:
      • Batching and scheduling (continuous batching);
      • KV cache quantization and paged management;
      • Tensor parallelism / pipeline parallelism;
      • Speculative decoding;
      • Expert parallelism and computation-communication overlap for MoE.

    At the deployment level, you can rely on general-purpose container platforms on the cloud to host inference services, use object storage to manage model weights and datasets, use log services to collect request and performance metrics, and use monitoring and alerting to promptly detect issues such as latency spikes and memory overflow. The general-purpose cloud products offered by cloud providers can be combined with the practices above to carry a project from experimentation to production.

    10. Summary and Outlook

    Returning to the opening question: seven years later, LLM architecture has indeed not been “reinvented from scratch.” Core elements such as the Transformer decoder backbone, residual connections, and softmax attention remain. Changes are concentrated in three directions:

    • Position encoding: from learnable absolute positions to RoPE, with accompanying extrapolation schemes;
    • Attention: from MHA to GQA / low-rank compression, to support long context and high concurrency;
    • Feed-forward and parameter organization: SwiGLU replacing GELU, and MoE decoupling capacity from computation.

    We can expect the evolution over the coming period to unfold along three lines:

    • Sparsification: fine-grained experts, shared experts, and more flexible routing;
    • Attention efficiency: more efficient sequence modeling such as linear attention and state space models, and architectures that mix them with standard attention;
    • Software-hardware co-design: architectural choices increasingly influenced by hardware characteristics and the inference-framework ecosystem.

    For developers, rather than chasing the short-term conclusion of “which model is the strongest,” it is better to master a stable architecture-analysis framework: understand each module’s design motivation and cost, and then make reasonable trade-offs among quality, cost, and latency based on your own business.