LLM Architecture Evolution: A Complete Map of Efficiency Bottlenecks and Emerging Turning Points

    Large language models have advanced rapidly, but capability gains depend as much on architecture as on scale — how attention is organized, how experts are routed, how positional information is injected, and how text is tokenized. As compute hardware evolution slows and long context and high concurrency become the norm, this article traces more than a century of language model architecture evolution and systematically breaks down efficient attention and KV cache management, linear language models and hybrid architectures, sparse mixture of experts, activation functions, normalization and residual connections, positional encoding, and tokenizers. Using a problem–mechanism–tradeoff–practice framework, it pairs each topic with working code examples, then closes with a selection checklist, common misconceptions, and an outlook on the three converging trends of sparsification, linearization, and updatability — helping readers build a complete map for choosing and understanding LLM architectures.

    In recent years, the leaps in LLM capability have often been attributed to “scale” — more parameters, more data, and more compute. But how a model actually computes depends on its architecture: how attention is organized, how experts are routed, how positional information is injected, and how text is tokenized. Scale merely amplifies the efficiency differences introduced by these structural choices. Almost every new generation of models makes substantive structural innovations on top of the previous architecture rather than simply scaling it up.

    Meanwhile, the pace of compute hardware evolution is slowing, long context and high concurrency have become the norm, and demands such as continual learning and test-time training keep emerging — some teams are even exploring models that can update their weights during inference. All signs indicate that the current attention architecture is not the endpoint, and this is gradually becoming a consensus in academia. Recently, a survey jointly completed by several well-known scholars together with researchers from multiple universities and companies systematically reviewed the more-than-a-century evolution of language model architectures, covering efficient attention and KV caching, linear language models, mixture of experts, activation functions, normalization and residual connections, positional encoding, and tokenizers. Drawing on its overall framework, this article breaks down each area in the order of “problem — mechanism — tradeoff — practice,” helping readers build a complete map of architectural evolution.

    1. A Century of Progress: Five Leaps in Language Model Architecture

    Architectural innovation rarely appears out of nowhere, and most of today’s designs have conceptual roots in history. Extending the timeline makes the underlying logic of evolution easier to see.

    Stage Approx. time Core idea Main bottleneck
    Statistical language models Since 1913 Conditional dependencies exist among language symbols and can be modeled by transition probabilities Data sparsity; state space grows exponentially with order
    n-gram models Mid-to-late 20th century Using the Markov assumption to characterize local dependencies Hard to generalize; insufficient capture of long-range dependencies
    Recurrent neural networks Around 1990 Compressing all history into a hidden state Vanishing gradients; time steps cannot be parallelized
    Gated recurrent networks Since 1991 Multiplicative gates control the writing and forgetting of information Inference still requires serial decoding
    Attention architecture 2017–present Pairwise interaction between any positions; highly parallel Computation grows quadratically with length; large caching overhead

    1. From conditional dependence to a stochastic source. In 1913, the mathematician Markov analyzed the arrangement of vowels and consonants in Pushkin’s novel-in-verse Eugene Onegin, using data to confirm for the first time that statistical dependencies exist among language symbols. Language modeling thus shifted from linguistic speculation to quantitative research. In 1948, Shannon modeled language as a stochastic source and estimated the entropy of English through an experiment of “guessing the next letter,” establishing the fundamental paradigm of “predicting the next symbol” that remains the core training objective of language models to this day.

    2. From lookup tables to state compression. Traditional n-gram models directly count the frequency of word sequences — simple, but faced with data sparsity. Recurrent neural networks instead summarize all history with a single hidden state, turning a dependence on fixed-length context into a conditional dependence on a variable state, yet introducing serial computation and vanishing gradients.

    3. From “no memory” to “selective memory.” The gated recurrent networks proposed around 1991 use multiplicative gates to control the writing, retention, and forgetting of information, easing the difficulty of training on long sequences. Around the same time, research on “fast weights” tried to let networks rapidly modify a set of weights at runtime. One rule that additively accumulated the outer product of “input–output” would, in today’s terms, be a linear attention without normalization. This clever idea will reappear several times later.

    4. From “compressing history” to “retrieving on demand.” In 2017, the attention architecture replaced hidden-state compression with “query–key–value” interactions, allowing any two positions to interact directly and training to be highly parallel, becoming the common foundation for later large models. Yet it never stopped evolving. The gating idea was brought into feedforward networks, forming the gated activation structures widely used today. The additive pathway in gated recurrent networks that was used to preserve gradient flow is, in principle, continuous with the later “residual connection.”

    5. From “passive compression” to “active memory.” Currently, architectures are beginning to explore reading and writing memory on demand during inference, and even updating memory, moving toward “active memory.”

    Three interwoven main lines are visible in architectural evolution:

    • Efficiency line: reducing the computation and storage overhead that comes with sequence length;
    • Capability line: enhancing long-range dependencies, precise retrieval, and reasoning ability;
    • Stability line: enabling deeper networks and larger models to be trained stably.

    Any new design is essentially a search for a new balance among the three.

    2. Driving Forces: Why Architectures Must Be Rebuilt

    2.1 Attention cost grows quadratically with sequence length

    The computational complexity of standard self-attention is O(n²·d), where n is the sequence length. Double the length and the attention computation quadruples. This is acceptable during training, but in long-context inference, every generated token must interact with all history, and the cost accumulates rapidly.

    2.2 KV cache becomes a memory and bandwidth bottleneck

    Autoregressive generation requires caching the keys (K) and values (V) of historical tokens to avoid recomputation. Its memory footprint can be approximated as:

    1. def kv_cache_bytes(n_layers, n_kv_heads, head_dim, seq_len, batch, dtype_bytes=2):
    2. """KV cache memory (bytes): 2 caches both K and V; dtype_bytes=2 corresponds to half precision"""
    3. return 2 * n_layers * n_kv_heads * head_dim * seq_len * batch * dtype_bytes
    4. # Example: 32 layers, 8 KV heads, head dim 128, half precision, sequence length 32K, batch size 8
    5. print(kv_cache_bytes(32, 8, 128, 32768, 8) / 1024**3) # about 32 GiB

    In this example, caching alone occupies about 32 GiB of memory, comparable to all the weights of many models. More critically, during token-by-token decoding, every step must re-read the entire cache. The main cost of the decoding phase is “moving the cache from memory into the compute units,” which is a bandwidth-bound problem. The larger the batch and the longer the sequence, the more likely you are to hit the memory wall and the bandwidth wall.

    2.3 Inference economics becomes the primary constraint

    From chat assistants to agents, to repository-level code understanding and document QA, real applications are extending toward both “longer context” and “higher concurrency” at the same time. Per-token cost is no longer just a reference metric after training is complete, but directly determines whether a business can scale.

    2.4 Training paradigms raise new demands

    Paradigms such as continual learning and test-time training require models to absorb new information during inference as well. The weights of current mainstream architectures are completely frozen after training ends, and can only “adapt temporarily” by relying on the limited information in context. Whether state can be updated at low cost at runtime is becoming a new dimension for measuring architectural advancement.

    3. Efficient Attention and KV Cache Management

    3.1 From “each head its own” to “multi-head sharing”: GQA

    In multi-head attention, each query head is paired with a separate set of key-value heads, so cache size grows in proportion to the number of heads. Grouped-query attention lets multiple query heads share one set of key-value heads; the fewer the groups, the more cache is saved, and in the extreme case it degenerates to all query heads sharing one set of keys and values (i.e., multi-query attention).

    1. def grouped_attention(q, k, v, n_kv_heads):
    2. """Grouped-query attention: n_kv_heads is the number of KV heads; equal to the number of query heads, it degenerates to multi-head attention"""
    3. B, H, N, D = q.shape
    4. g = H // n_kv_heads
    5. # Illustrates the grouping only; in real inference it is broadcast via mapping indices, without copying data
    6. k = k.repeat_interleave(g, dim=1)
    7. v = v.repeat_interleave(g, dim=1)
    8. scores = q @ k.transpose(-1, -2) / D**0.5
    9. return scores.softmax(dim=-1) @ v

    Tradeoff analysis: Reducing key-value heads to 1/8 theoretically cuts the cache to 1/8, but part of the key-value representation is shared across multiple query heads, usually accompanied by a slight quality loss. In engineering, one typically runs small-scale comparison experiments first, then chooses the smallest number of groups within an “acceptable quality range.”

    3.2 From “multi-head concatenation” to “low-rank compression”

    A more aggressive approach is to first compress each token’s keys and values into a low-dimensional “latent vector” before caching, and then approximately reconstruct them via an up-projection matrix when computing attention.

    • Advantage: The compressed dimension can be far smaller than “number of key-value heads × head dimension,” saving more cache and decoupling more thoroughly across different head-count configurations.
    • Cost: It adds up-projection computation and requires careful implementation to actually turn the bandwidth savings from compression into throughput gains.
    • Engineering detail: Some implementations adopt a “dimension decoupling” strategy, separating the position-related part from the compressed vector so that positional encoding and the compression mechanism do not interfere with each other.

    This kind of design shows that KV cache management has become a design variable as important as the attention structure itself.

    3.3 Sparse attention: from “seeing everything” to “seeing what should be seen”

    Not all tokens need to interact with all tokens. Sparse attention computes only a subset of attention scores, with common strategies including:

    • Sliding window: each token attends only to a few nearby positions, suitable for local semantics;
    • Global tokens: a few special tokens can be attended to by any position, acting as “information hubs”;
    • Block or routing sparsity: coarse filtering by block, or dynamically selecting relevant blocks by content.

    Sparsification can directly reduce computation, but it carries higher risk on precise-retrieval tasks: if the target information happens to fall in a skipped region, the model may “overlook” it. Many approaches therefore retain a few global layers, or use a “dense + sparse” hybrid arrangement.

    3.4 Cache management: the second optimization at the system level

    Even after compressing the cache at the algorithmic level, there is still much room at the system level:

    • Paged management: splitting the cache into fixed-size blocks allocated on demand, reducing fragmentation and reservation waste;
    • Prefix reuse: common prefixes such as system prompts and few-shot examples are computed only once and shared across multiple requests;
    • Cache quantization: storing keys and values in half precision or 8-bit integers to further compress size, which requires precision evaluation;
    • Scheduling coordination: incorporating cache occupancy into batch scheduling decisions to avoid memory overflow that causes requests to queue.

    For application developers, understanding these mechanisms helps optimize how prompts are organized — for example, placing stable content up front to benefit from prefix reuse.

    4. Linear Language Models and Hybrid Architectures

    4.1 Core idea: changing the order of computation

    Linear attention replaces Softmax with a feature map, so that attention of the form (Q·Kᵀ)·V can be rewritten as Q·(Kᵀ·V). The former order must explicitly construct an n×n matrix, while the latter first obtains a d×d “state matrix” that is independent of sequence length, reducing complexity from quadratic to linear. During inference it is equivalent to a loop that continuously updates a state, and this state can be computed blockwise in parallel during training.

    1. def linear_attention(q, k, v, eps=1e-6):
    2. """Un-normalized linear attention illustration: state = accumulated outer products (k, v), complexity O(n)
    3. Computed in one pass here; real implementations are usually blockwise and parallel, with decay gating added"""
    4. state = k.transpose(-1, -2) @ v # [..., d, d], independent of n
    5. out = q @ state # [..., n, d]
    6. z = q.sum(-1, keepdim=True) * k.sum(-2).unsqueeze(-2) + eps
    7. return out / z # normalization, approximating the role of Softmax's denominator

    Note that this structure is mathematically highly consistent with the additive outer-product rule of the 1991 “fast weights.” The difference is that linear attention introduces normalization, making it closer to the semantics of “averaging over all history with weights.”

    4.2 State space models and gating mechanisms

    Another representative idea is the selective state space model. It also maintains a fixed-size hidden state, but the key improvement lies in “gating”:

    • The way parameters are updated is dynamically determined by the current input;
    • The model can learn to “keep important information and forget irrelevant information”;
    • Selective gating solves the problem of pure linear recurrence “remembering everything but excelling at nothing.”

    In a sense, the gating idea from the recurrent network era reasserts itself under a new computational form.

    4.3 The boundaries of strengths

    Linearization is not without cost:

    • Advantages: constant memory footprint, sequence-length extrapolation friendliness, and stable decoding speed for long text;
    • Weaknesses: the fixed-size state has limited capacity, and on tasks such as precise localization, verbatim copying, and multi-cue joint retrieval, it often falls short of Softmax attention that can directly revisit history;
    • Practical approach: many solutions interleave a few full-attention layers with a majority of linear or state layers, balancing long-range capability and efficiency.

    4.4 The mutual shaping of implementation and hardware

    The two phases have different bottlenecks. During training, linear attention requires carefully designed parallel scans or blockwise algorithms to fully utilize compute. During inference decoding, the bottleneck of full attention is repeatedly reading the ever-growing cache, while the bottleneck of linear layers is reading the fixed-size weights and state at every step — the two pressures differ.

    Architectural choice is therefore deeply coupled with hardware characteristics. Only with software-hardware co-design can a theoretical complexity advantage turn into actual throughput.

    5. Sparse Mixture of Experts (MoE): Trading Conditional Computation for Capacity

    5.1 Basic idea: large total capacity, small per-invocation activation

    Feedforward networks usually account for the majority of a model’s parameters. Sparse mixture of experts splits a single feedforward network into multiple “experts,” and each token activates only a few of them via routing:

    1. def moe_forward(x, gate, experts, top_k=2):
    2. """Sparse MoE illustration: experts is a list of feedforward networks; real implementations batch-compute grouped by expert"""
    3. scores = gate(x).softmax(dim=-1) # each token's scores over all experts
    4. weights, idx = scores.topk(top_k, dim=-1) # keep only the top_k highest-scoring
    5. out = 0
    6. for slot in range(top_k): # illustrative; real implementations compute grouped by expert
    7. eid = idx[:, slot]
    8. out = out + weights[:, slot].unsqueeze(-1) * experts[eid](x)
    9. return out

    Its core value lies in “decoupling”: total parameter count determines the model’s knowledge capacity, while the activated parameter count determines the computational cost per token. Under the same training compute, sparsification lets a model hold more parameters, and in practice it has become a standard choice for many large models.

    5.2 Routing and load balancing

    Routing decides “who computes,” and is also the trickiest part of MoE engineering.

    Main risks:

    • If a few experts are selected too often, the remaining experts get no training (“routing collapse”);
    • Uneven expert load across devices creates a barrel effect, where overall throughput is dragged down by the slowest device.

    Common countermeasures:

    • Balanced auxiliary loss: encouraging tokens to be evenly distributed across experts;
    • Capacity factor: limiting the number of tokens each expert receives, with overflow dropped or routed to a fallback, trading off between efficiency and quality;
    • Communication optimization: working with expert parallelism to route tokens to the corresponding devices.

    5.3 Fine-grained experts and shared experts

    Fine-grained experts: cutting experts into finer pieces, with each token activating more but smaller experts. The combination space is larger and specialization higher, at the cost of more frequent routing and communication.

    Shared experts: setting up common experts that always participate in computation to carry general knowledge. The remaining routed experts focus on learning differentiated knowledge, reducing redundant learning of general capability across different experts.

    5.4 Engineering costs

    MoE is not a “free lunch”:

    • Large parameter count means weights stay resident in memory, demanding high memory capacity;
    • Cross-device token routing brings extra communication overhead;
    • Different tokens in a batch activate different experts, making clean batching difficult.

    Therefore, the actual gains of MoE depend heavily on the quality of distributed training and inference frameworks, and one should not judge by the parameter count alone.

    6. Activation Functions: The Revival of Gated Structures

    6.1 The evolution

    The activation functions of feedforward networks have evolved ReLU → GeLU → Swish/SiLU, with the common trend moving from “hard truncation” toward “smooth differentiability.” GeLU uses the cumulative distribution probability of the input itself as the gating strength, while SiLU uses Sigmoid as the gate — both embody the idea of “self-gating.”

    6.2 The Gated Linear Unit (GLU) family

    GLU splits one output path of the feedforward network into a “content” path and a “gate” path, multiplying them element-wise:

    $\text{FFN}_{\text{GLU}}(x) = \big(\sigma(xW)\big) \otimes (xV)$

    where σ can be Sigmoid, GeLU, or SiLU, corresponding to different variants. SwiGLU uses SiLU as the gating function:

    1. def swiglu(x, W, V, W2):
    2. """SwiGLU feedforward network illustration: W, V, W2 are linear projection weights"""
    3. return (F.silu(x @ W) * (x @ V)) @ W2

    Mechanism: The gated path lets the model dynamically adjust the amount of information passing through, per token and per channel, echoing gating in attention — both are “letting through on demand.” Because there is an extra projection path, the intermediate-layer dimension is usually scaled down proportionally to keep parameter count and computation roughly unchanged.

    Intellectual roots: Multiplicative gating was precisely one of the core means for combating vanishing gradients in the recurrent network era. Today’s gated feedforward networks can be seen as a “return” of this old idea in a new structure.

    7. Normalization and Residual Connections: The Foundation for Training Deep Networks

    7.1 Residual connections: gradient highways

    Residual connections make each layer’s output equal to “input + transformation,” providing a direct path for gradients and making it possible to train networks of hundreds of layers. From an information-flow perspective, this is essentially continuous with the additive pathway used to preserve the error flow in gated recurrent networks: a learnable transformation superimposed on a backbone that is not rewritten.

    7.2 The debate over the position of normalization layers

    Normalization stabilizes the distribution of each layer’s activations, making training smoother:

    • Layer Normalization (LayerNorm): standardizes all features of a single sample, independent of batch size;
    • Root Mean Square Normalization (RMSNorm): only scales by the root mean square, omitting the mean computation and bias parameters, with lower overhead — it has become the mainstream choice in large models.

    Placement: Early on, “post-normalization” was common — performing the sublayer operation first and then normalizing — but deep networks trained unstably. The mainstream approach today is “pre-normalization” — normalizing first and then entering the sublayer — often supplemented with:

    • Residual path scaling: shrinking the weights of the residual branch at initialization, or appropriately increasing the LayerNorm gain;
    • QK normalization: normalizing queries and keys to control the magnitude of attention scores and prevent numerical divergence.

    Fine-tuning by symbols: These tricks are often summarized as “subtract 1, divide by √N.” They may seem minor, but they significantly reduce the probability of loss spikes in ultra-large-scale training.

    7.3 Engineering points

    The stability of large-model training often depends on details:

    • Conventional measures such as learning-rate warmup, gradient clipping, and outlier detection are indispensable;
    • Under mixed precision, normalization statistics are sensitive to numerical range and usually need to be computed at higher precision;
    • When loss spikes appear, prioritize checking the learning rate, data anomalies, and normalization configuration, rather than blindly rolling back checkpoints.

    8. Positional Encoding: The Key to Length Generalization

    8.1 Why position must be injected explicitly

    Attention itself is “permutation-equivariant”: shuffling the input order only reorders the output accordingly. Without injected positional information, a model cannot distinguish word order. Mainstream approaches fall into roughly three categories:

    • Absolute position: learnable embeddings or sinusoidal function encoding;
    • Relative position: adding relative distance as a bias term into attention;
    • Rotary Position Embedding (RoPE): applying position-dependent rotations to queries and keys, so that attention scores depend only on relative distance, balancing effectiveness and efficiency — it has become the de facto mainstream choice today.
    1. import torch
    2. def apply_rope(x, base=10000.0):
    3. """Rotary position embedding illustration, using the first/second-half dimension pairing implementation; x shape is [sequence length, head dim]"""
    4. d = x.shape[-1]
    5. idx = torch.arange(0, d, 2, dtype=x.float32)
    6. inv_freq = 1.0 / base ** (idx / d) # each frequency component
    7. pos = torch.arange(x.shape[0], dtype=torch.float32)
    8. ang = torch.outer(pos, inv_freq) # [sequence length, head dim/2]
    9. x1, x2 = x[..., 0::2].float(), x[..., 1::2].float()
    10. cos, sin = ang.cos(), ang.sin()
    11. out = torch.empty_like(x)
    12. out[..., 0::2] = x1 * cos - x2 * sin
    13. out[..., 1::2] = x1 * sin + x2 * cos
    14. return out

    8.2 Length extrapolation: from training length to inference length

    A natural problem with rotary position embeddings is that when the inference length exceeds the training length, the rotation angles enter intervals not seen during training, and performance degrades markedly. Common countermeasures include:

    • Position interpolation: proportionally compressing position indices into the training interval;
    • Frequency scaling: raising the rotation base so that high-frequency components vary more gently;
    • Long-context continued training: doing a small amount of continued training on longer sequences.

    In practical systems, long-context capability is often determined jointly by the “positional encoding scheme + long-text training” — encoding tricks alone cannot conjure it from nothing.

    8.3 The positional problem in linear models

    Models with purely recurrent structures are inherently position-aware, but explicit positional encoding is still indispensable. Such structures often rely on decay mechanisms (letting the weight of earlier information decay exponentially over time) or explicit positional biases to compensate for insufficient ordering information. The differences between architectures in “how they handle position” often directly determine the kinds of tasks they excel at.

    9. The Tokenizer: The Most Underrated Architectural Component

    9.1 Mainstream algorithms

    The tokenizer decides how text is cut into tokens the model can read. Common algorithms include:

    • BPE (Byte Pair Encoding): starting from characters and repeatedly merging high-frequency adjacent pairs;
    • WordPiece: choosing merges by the criterion of language model likelihood;
    • Unigram: first building a large vocabulary, then pruning it step by step by probability;
    • SentencePiece: treating spaces themselves as symbols, naturally adapting to languages without spaces such as Chinese and Japanese;
    • Byte-level fallback: falling back to a byte representation when encountering out-of-vocabulary characters, ensuring any input can be encoded.

    9.2 The tradeoff of vocabulary size

    Vocab size Sequence length Parameter & compute overhead Typical issue
    Smaller Longer Lower embedding and output-layer overhead Little information per token; heavier burden per layer
    Moderate Moderate Balanced Mainstream approach; needs to balance multilingual fairness
    Larger Shorter Higher embedding and output-layer overhead Rare words over-split; low-frequency tokens under-trained

    Fertility is an important metric for measuring a tokenizer — the average number of tokens a word is cut into. In scenarios such as Chinese, code, and numbers, fertility is noticeably higher, meaning the same text needs more tokens to process, directly driving up training and inference costs.

    9.3 Engineering recommendations

    • Evaluate tokenizer quality: use fertility and coverage rather than subjective feel. Cover target languages, code, and specialized domains, and check whether out-of-vocabulary rates and high-frequency word splitting are reasonable;
    • Pay attention to downstream impact: how numbers and code are split affects arithmetic and precise-editing tasks; when necessary, do domain-specific vocabulary expansion and continued training;
    • Maintain inference consistency: tokenizers usually evolve independently of model weights, so when upgrading you must ensure training and serving use the same version, otherwise input-distribution mismatch occurs.

    10. An Architecture Selection Checklist and Outlook

    10.1 Selection evaluation checklist

    Faced with an endless stream of new architectures, rather than chasing the “strongest,” it is better to first answer the following questions:

    1. Workload characteristics: Are requests mostly long-context or mostly short? How long are the outputs? The former calls for attention to caching and positional encoding, the latter to decoding bandwidth;
    2. Latency and throughput goals: What are the requirements for time-to-first-token, per-token generation time, and overall throughput;
    3. Quality floor: How much retrieval precision do tasks such as long-document QA and repository-level code understanding need, and can you tolerate the slight loss from sparsity or compression;
    4. Deployment resources: Can memory capacity, interconnect bandwidth, and batch scheduling support the chosen architecture? Sparse expert models are especially sensitive to memory capacity;
    5. Ecosystem maturity: Are training frameworks, inference frameworks, and quantization schemes well-developed, and is community experience sufficient? This directly determines iteration speed.

    10.2 Common misconceptions

    • Parameter-count determinism: Sparse expert models have large total parameter counts, but the activated parameters and inference cost are what actually matter;
    • Complexity determinism: Standard attention has been optimized for years, with a mature implementation and toolchain, and in many scenarios remains the most cost-effective choice;
    • Ignoring the system layer: Throughput gains from prefix reuse, cache paging, and batch-scheduling optimization can sometimes rival an architecture upgrade;
    • Confusing length with understanding: A longer context window does not mean stronger long-text comprehension; you need to independently verify with “needle-in-a-haystack” style long-text retrieval tests.

    10.3 Outlook: three main lines continuing to converge

    Sparsification: From attention sparsity to expert sparsity, to fine-grained routing, “computing on demand” becomes the main line, with the core challenge being how to precisely judge “what to compute” without sacrificing quality.

    Linearization: State compression makes the cost of long-sequence inference controllable; how to compensate for the weakness in precise retrieval, and how to optimally mix with full-attention layers, remain open questions.

    Updatability: From test-time training to dynamically updating weights during inference, architecture is evolving from a “static function” toward a “system that can adapt at runtime.” This poses entirely new demands on training stability, weight security, and version management.

    It is worth emphasizing that these directions do not replace each other. Future flagship models are more likely to be hybrids: some layers use full attention to guarantee retrieval precision, some use linear structures to control long-sequence cost, feedforward layers use sparse experts to expand capacity, and these are combined with gated activations, rotary position embeddings, and carefully designed tokenizers. The real competition also shifts from “whether a single structural point is novel” to “the overall efficiency of software-hardware co-design.”