Large Language Model Architecture Evolution: From Attention Optimization to Linearization and Sparsification

    This article traces the evolution of large language model architectures, from early statistical language models to today's Transformers and their variants. It explains why architecture innovation is becoming central as gains from scaling slow, and reviews six research areas covering Transformer components: attention and KV cache management, linear models, mixture-of-experts, feed-forward layers, normalization and residual connections, and positional encoding and tokenizers. It closes with three trends: efficient attention alternatives, sparse activation, and dynamic adaptation. Readers will get a component-level framework for judging new architectures.

    1. Why Architecture Innovation Is Back in the Spotlight

    Over the past few years, progress in large language models has been driven mainly by scaling up: larger parameter counts, more training data, and longer training runs have produced steady gains in capability. However, the marginal returns from scaling are diminishing, while compute costs and inference overhead keep rising. As a result, the industry increasingly agrees that the standard Transformer architecture is not the final answer, and architectural innovation is a key variable for the next stage.

    Nearly all recent flagship models have modified their base architectures in targeted ways:

    • Attention efficiency: Grouped-query attention (GQA), multi-head latent attention (MLA), various sparse attention mechanisms, and related KV cache management strategies. The main goal is to reduce memory and compute costs for long-context inference.
    • Sequence modeling: Linear attention networks, represented by state space models and gated linear recurrent units, aim to escape the quadratic dependence of standard attention on sequence length.
    • Parameter utilization: Fine-grained mixture-of-experts (MoE) designs let a model have a very large total parameter count while activating only a small portion of those parameters in each forward pass.
    • Other directions: Multi-token parallel prediction, looped architectures, and new activation functions and normalization schemes are also being adopted in mainstream practice.

    At the same time, new requirements such as continual learning and test-time training are pushing architecture forward. These approaches expect a model to adjust its internal state, or even its weights, during inference as it encounters new tasks. This poses a fundamental challenge to the traditional static design, where a model is trained once and then runs inference with fixed weights.

    2. A Century of Language Model Architecture: Historical Context

    Understanding how architectures evolve requires a longer time frame. This history can be roughly divided into four stages.

    1. Origins of statistical language models (early 20th century)
    As early as 1913, a researcher used statistics on the arrangement of vowels and consonants in literary works to show, for the first time with data, that conditional dependencies exist in language. In 1948, foundational work in information theory treated language formally as a random source, providing a theoretical framework for later statistical modeling.

    2. n-grams and early neural networks (mid to late 20th century)
    In speech recognition, n-gram models approximated conditional probabilities using a limited history window, becoming the classic form of statistical language models. Later, recurrent neural networks were introduced into language modeling, giving models the ability to carry information across longer distances.

    3. The golden era of gated recurrent networks (1990s)
    Gated recurrent designs address long-range dependencies. The core idea behind LSTM had already been proposed in the early 1990s. These designs use input, forget, and output gates to control information flow. Their influence has been lasting, and their echoes can still be found in Transformer components today.

    4. The Transformer era (2017 to present)
    Architectures based entirely on attention replaced recurrent structures and became the de facto standard for large models. Their friendliness to parallel training and strong global modeling ability fit well with the demands of scaling up.

    The most valuable lesson from this long history is the cross-reuse of technical ideas. Several examples illustrate this:

    • An early idea called “fast weight programming” proposed an additive outer-product update rule. Viewed today, it is essentially a linear Transformer without normalization. Years later, kernelized linear attention methods were mathematically equivalent to adding a normalization step to this older path.
    • The multiplicative gating mechanism in recurrent networks was later transferred into feed-forward networks, evolving into the SwiGLU structure widely used in today’s mainstream models.
    • The additive pathway in recurrent networks, which keeps error signals stable during backpropagation, shares the same underlying idea as the residual connection in Transformers.

    These examples show that architectural innovation is rarely invented from nothing. It is usually a recombination and rediscovery of existing ideas under new constraints.

    3. Research Landscape of Six Architectural Components

    A recent survey from several universities and research institutions organizes related work by the components of the base Transformer. We can group this work into six directions.

    1. Efficient Attention and KV Cache Management

    The computational complexity of standard self-attention grows with the square of sequence length. Inference also requires caching the Key and Value tensors of all past tokens. Together, these make long-context inference the biggest bottleneck. Current mainstream optimizations include:

    • Compressed and shared KV caches: For example, grouped-query attention lets multiple query heads share one set of Key/Value heads. This shrinks the cache several times over with almost no loss in accuracy.
    • Latent compression: Keys and Values are projected into a low-dimensional latent space before storage, further reducing the cost of long sequences.
    • Sparsification and localization: Sliding window attention, block-sparse attention, and similar methods rely on the prior that attention distributions are naturally sparse. They compute only the high-value attention terms.

    2. Linear Language Models

    Linear attention and state space models rewrite attention in a recurrent form. At inference time, complexity drops from O(N²) to O(N), and the KV cache is compressed into a fixed-size state matrix. The trade-off is that these models recall precise long-range history less well than standard attention. For this reason, hybrid architectures that combine linear models with a few global attention layers have become a practical choice that balances efficiency and capability.

    A simplified linear attention example:

    1. # Standard attention: inference complexity O(N*d)
    2. # Linearization: change the computation order using associativity
    3. phi_q = phi(Q) # feature map
    4. phi_k = phi(K)
    5. state = phi_k.T @ V # recurrently maintained fixed-size state (d x d)
    6. out = phi_q @ state # O(d^2) per step, independent of sequence length

    3. Mixture-of-Experts (MoE)

    An MoE model uses a router to send each token to a small number of expert networks. This achieves a large total parameter count with a small number of active parameters. Research has shifted from coarse-grained experts to fine-grained expert partitioning. Using more, smaller experts with a larger routing pool can significantly increase the flexibility of expert combinations. Load-balancing losses, routing stability, and communication for expert parallelism are engineering problems that determine whether MoE delivers real gains in practice.

    4. Activation Functions and Feed-Forward Networks

    Feed-forward design has evolved steadily, from ReLU to GELU and then to the gated linear unit (GLU) family, represented by SwiGLU. Gating lets the network selectively pass information. Empirically, this yields better scaling curves, and gated designs have become the default choice for most flagship models.

    5. Normalization and Residual Connections

    The placement of normalization layers (Post-Norm or Pre-Norm) and their type (LayerNorm or RMSNorm) directly affect the training stability of deep networks. Residual connections provide direct paths for gradients and information. These components may seem minor, but they are the prerequisite for training models with hundreds of layers stably. Their underlying idea closely matches the error-preserving pathways in early recurrent networks.

    6. Positional Encoding and Tokenizers

    Positional encoding determines how a model perceives token order. It ranges from absolute positional encoding to rotary positional embedding (RoPE), and then to extrapolation and interpolation methods for long contexts. These are core variables for long-text capability. Tokenizers determine the granularity and efficiency of sequence representation, which directly affects how effectively the context window is used and the cost of inference. Vocabulary construction and byte-level encoding approaches continue to evolve.

    Current research points to three main directions for next-generation architectures:

    1. Efficiency-driven attention alternatives: Linear attention, sparse attention, and hybrid architectures are developing in parallel. The goal is to keep costs affordable at the scale of hundreds of billions of tokens and million-token contexts.
    2. Pushing sparse activation further: MoE is evolving from a substitute for dense models into an independent design paradigm. Research on routing strategies, expert granularity, and training stability is still far from settled.
    3. Dynamic adaptation and continual learning: Test-time training, fast weight updates, and recurrent depth aim to move models from static parameter snapshots toward systems capable of online adaptation. This would reshape the traditional boundaries between training, deployment, and inference.

    5. Conclusion

    From the discovery of statistical regularities a century ago, to gated recurrent networks, to the Transformer and its many variants, the history of language model architecture is one of repeatedly recombining classic ideas under the constraints of compute and expressive power. For developers and researchers, rather than chasing the hype around a single new architecture, it is more useful to build a component-level knowledge framework. Understand what problem each module solves, what trade-offs it involves, and which historical ideas it relates to. When the next generation of architectures arrives, you will be able to judge its value quickly instead of simply following changing terminology.