2025 LLM Architecture Deep Dive: Transformer Stabilization, Attention Variants, and KV Cache Compression
This article examines the main architectural directions of large language models in 2025, focusing on lowering inference cost, supporting longer contexts, and improving reasoning under limited activated parameters. It reviews Transformer stabilization techniques such as Pre-Norm, RMSNorm, RoPE, and gated feed-forward networks, then compares attention variants including MHA, MQA, GQA, and latent-space compression. It also provides a formula for estimating KV cache size, helping architects and model-serving engineers understand the practical trade-offs behind current LLM architecture choices.
1. Introduction: Drivers of Architecture Evolution
The capability growth of large language models has long depended on the simultaneous expansion of parameter scale, training data, and compute investment. In real-world deployment, however, inference cost, memory usage, and response latency quickly become bottlenecks. The architectural evolution of 2025 is mainly organized around three goals: reducing per-inference cost, supporting longer contexts, and enabling stronger reasoning with a limited number of activated parameters. Understanding these three threads is more valuable than memorizing any specific structure.
This article is intended for two types of readers. One is architects who need to evaluate technical directions. The other is engineers responsible for deploying and operating model services. The numerical values in this article are illustrative calculations used to explain orders of magnitude; they do not represent measured results from any specific model.
2. Basic Skeleton: Stabilizing and Refining the Transformer
The Transformer remains the foundational structure for the vast majority of large language models. Mainstream industry improvements focus on several details that directly affect training stability and inference efficiency.
2.1 Pre-Norm Normalization Placement
Pre-Norm places the normalization layer before the residual branch, allowing gradients to propagate more stably in deep networks. Compared with the earlier Post-Norm approach, it is more tolerant of hyperparameters such as learning rate, making it a common choice for deep models.
2.2 RMSNorm
LayerNorm first subtracts the mean and then divides by the standard deviation. RMSNorm removes the mean-centering step and scales only by the root mean square. Its computation is as follows:
y = x / sqrt(mean(x^2) + eps) * g
Here, g is a learnable scaling parameter, and eps is a small constant that prevents division by zero. By eliminating the mean calculation, the memory access and compute overhead of each layer are reduced. In practice, its effect is usually close to that of LayerNorm.
2.3 Rotary Position Embedding
Rotary Position Embedding (RoPE) applies position-dependent rotations to query and key vectors, so that the attention score between two tokens depends only on their relative distance. The advantage of this design is that position information is decoupled from content, and the context length can be extended by adjusting the rotary base frequency. This point will be discussed further in the long-context section later.
2.4 Gated Feed-Forward Networks
Feed-forward layers commonly adopt variants of gated linear units, such as SwiGLU. The gating structure allows the network to dynamically control the proportion of information that passes through. Under the same parameter count, such structures usually perform better than ordinary two-layer feed-forward networks, so they have become a common choice.
3. Attention Mechanisms: From Multi-Head to Compressed Cache
During autoregressive generation, the model needs to cache the keys and values of historical tokens at every layer. This memory is called the KV cache. In long-sequence and high-concurrency scenarios, the KV cache often exhausts GPU memory faster than the model weights. Therefore, most attention variants are centered on reducing the cache.
3.1 Main Attention Variants
- Multi-Head Attention (MHA): Each query head has an independent key-value head. It has the strongest expressive capability but the largest cache size.
- Multi-Query Attention (MQA): All query heads share the same set of key-value heads. It has the smallest cache but usually incurs a visible quality loss.
- Grouped-Query Attention (GQA): Query heads are divided into several groups, and key-value heads are shared within each group. It balances quality and cache size and is currently the most widely used option.
- Latent-space compression: Keys and values are first projected into low-dimensional latent vectors and cached, then restored during computation. This further reduces cache size at the cost of additional projection and restoration compute.
3.2 KV Cache Capacity Estimation
The number of bytes in the KV cache can be estimated with the following formula:
KV cache bytes = 2 × layers × KV heads × head dimension × bytes per element × sequence length × batch size
where the coefficient 2 accounts for the keys and values, each occupying one copy.
