LLM Architecture Evolution: Attention Optimization and Sparse Expert Routing
This article reviews recent architecture-level improvements in large language models, focusing on attention cost reduction, residual-stream aggregation, sparse expert routing, latent-space expert design, KV cache offloading, and low-precision deployment. It also outlines how to interpret open-source model ecosystem data and benchmark results, and provides practical recommendations for evaluating and deploying new architectures under real engineering constraints.
Introduction
As large language model capabilities improve, inference cost, memory usage, and response latency also rise. When engineering teams select a model or deployment approach, the core question is often how to maintain model quality under limited compute while keeping the deployment plan maintainable over time. In recent years, architecture-level improvements have concentrated on three directions: reducing how quickly attention computation grows with sequence length; using sparse structures to expand parameter scale while controlling the actual computation per token; and managing cache and memory more efficiently during inference. This article covers these three directions and adds a method for evaluating information from the open-source model ecosystem, helping readers distinguish technical principles from market attention.
1. The Cost Problem of Attention Computation
Standard self-attention computation scales quadratically with sequence length. Each time the sequence length doubles, the attention computation increases by roughly four times. At the same time, inference requires storing the key-value cache (KV cache) for each layer, and its memory usage grows linearly with context length. For long documents, multi-turn dialogue, and repository-level code tasks, these two costs are often deployment bottlenecks.
Linear Attention and State Compression
The idea behind linear attention is to change the computation order and compress access to historical tokens into a fixed-size state. In this way, computation grows linearly with sequence length, and the cache no longer expands indefinitely with length. The trade-off is that the model’s ability to perform precise backtracking may decrease, which needs to be compensated for through hybrid structures or gating mechanisms.
Sparse Attention and Window Mechanisms
Sparse attention focuses only on selected positions, such as a local sliding window plus a small number of global positions. It is relatively straightforward to implement and is compatible with existing inference kernels. There are two engineering considerations: window size determines the range of retrievable information, and if the sparse pattern does not match the task structure, there can be clear degradation on long-range dependency tasks.
Trade-offs in Hybrid Architectures
In practice, more designs use a hybrid approach in which some layers use full attention and some layers use efficient variants. Selection can be evaluated along the following dimensions:
- Long-context accuracy: test with real long-document retrieval tasks rather than only short-text benchmarks.
- Cost per token: measure prefill and token-by-token decoding throughput on the target hardware.
- Peak memory: test memory usage separately under maximum batch size and longest context.
- Implementation maturity: confirm whether the inference framework already has stable kernel implementations.
2. Residual Streams and Cross-Layer Information Aggregation
In deep networks, the output of each layer is added to the main information stream through residual connections. As the number of layers increases, information from early layers may be diluted by later updates, making it difficult for the model to use deep representations stably. One improvement direction is to let later layers not only read the output of the previous layer but also aggregate representations from multiple layers with learnable weights.
The core of this type of method is a set of trainable aggregation coefficients that determine the share of each layer’s output in the final representation. Its benefits usually appear in training stability and deep gradient propagation, but it adds a small amount of parameters and memory access. During evaluation, attention should be paid to the convergence speed of training curves, the performance slope across different depths, and the impact of extra memory access on inference latency.
3. Sparse Expert Routing (MoE)
Mixture of Experts (MoE) splits the feed-forward network into multiple experts, and each token activates only a few of them. In this way, the total parameter count can be large, while the parameters actually involved in computation per token are only a small fraction.
Basic Structure
The router computes scores for each expert based on the token hidden state, selects the top-k experts with the highest scores, and performs a weighted sum of their outputs. Below is illustrative top-k routing pseudo-code:
Function top_k_route(hidden_states, router_weights, k):# hidden_states shape: [num_tokens, dimension]# router_weights shape: [dimension, num_experts]scores = matmul(hidden_states, router_weights)probs = softmax(scores, dim=experts)selected_probs, selected_indices = top_k(probs, k)normalized_weights = selected_probs / sum(selected_probs)return selected_indices, normalized_weights
Load Balancing
If the router always assigns tokens to a small number of experts, some experts become overloaded while others remain idle. A common approach is to add a load-balancing term to the training loss to encourage each expert to receive roughly the same number of tokens. During inference, uneven expert distribution can also lead to uneven communication load across devices, so actual expert load should be monitored during multi-GPU deployment.
Total Parameters and Activated Parameters
When evaluating an MoE model, record the total parameter count and the number of parameters activated per token separately. Their ratio determines the computational advantage relative to a dense model of comparable scale. However, memory usage is still related to the total parameter count, because the weights of all experts must reside in accessible storage.
4. Expert Design in Latent Space
Some approaches first project high-dimensional hidden states into a low-dimensional latent space before expert computation, perform routing or part of the computation in that latent space, and then project back to the original dimension. The motivation is to reduce communication data between experts and reduce computation overhead inside experts.
This design involves clear trade-offs. If the projection dimension is too low, information needed to distinguish different experts is lost; if the projection dimension is too high, the benefit becomes smaller. In engineering practice, ablation experiments can be used to determine the dimension, and the actual changes in communication volume and latency should be measured on the target hardware.
5. Cache Offloading During Inference
When context is long or concurrency is high, the KV cache can fill memory. The idea behind cache offloading is to migrate cache blocks that are not currently in use to host memory or a lower-speed storage tier, and load them back when needed.
Below is an illustrative cache management strategy using least recently used (LRU) eviction:
Class KVCacheManager:Initialize(VRAM_block_budget):VRAM_blocks = OrderedDict()host_blocks = Dict()budget = VRAM_block_budgetAccess(block_id):If block_id in VRAM_blocks:Move block_id to most recently used positionReturn "hit"If block_id in host_blocks:If VRAM is full, evict least recently used block to host memoryLoad host_blocks[block_id] into VRAMReturn "loaded from host memory"Otherwise:Allocate new block in VRAM, evicting old blocks if necessaryReturn "new"
The key metrics for offloading are swap bandwidth usage and extra latency. If cache access patterns show clear locality, such as repeated use of historical prefixes in multi-turn dialogue, offloading can bring significant benefits. If access patterns are random, frequent swapping can offset the benefits of memory savings.
6. Low-Precision Deployment
Low-precision representations of weights and activations can significantly reduce memory usage and bandwidth requirements. The 4-bit microscaling floating-point format is a low-precision format that shares scaling factors per small block, compressing storage while preserving dynamic range.
The following validation should be performed before deployment:
- Compare output differences between full-precision and low-precision models on representative tasks, including reasoning, code, and long-text tasks.
- Check whether the inference framework supports the full kernel path for the format, avoiding fallback to slow implementations.
- Measure throughput under the target batch size and context length, not only latency for a single request.
- Record version numbers and weight checksums to ensure that online and test environments use the same set of weights.
7. How to Read Open-Source Model Ecosystem Data
After a model is released, various platforms publish metrics such as download counts, rankings, and usage counts. These data are useful references, but their meanings differ, and direct comparisons can lead to incorrect conclusions.
Distribution Volume
Weight download counts reflect acquisition behavior; a single pull may be used for testing, mirroring, or automated pipelines. Growth in downloads indicates expanding reach, but it does not imply growth in online invocation volume.
Framework Support
Whether an inference framework includes a model in official support determines whether ordinary users can use it out of the box. An empty support field on a framework page, or code that is still in draft or unmerged status, indicates that the ecosystem is still adapting. When judging, distinguish among “released,” “merged,” and “under discussion.”
Real Usage
Token usage, session counts, and unique users on a platform are closer to actual usage than download counts. However, these data usually cover only a single platform, and the statistical window may be rolling. A decline in values within the window does not necessarily mean real usage has declined; it may simply mean that older data has left the statistical range.
Rankings and Share
Rankings and share are relative indicators that depend on the performance of other models on the same platform. Absolute growth with a declining ranking is common when the overall market expands and competitors grow faster. Analysis should consider absolute values, relative position, and time series together.
It is advisable to build an observation table that records the same set of metrics weekly, with data sources and statistical scope noted.
8. Credibility of Evaluation Benchmarks
Benchmarks are a common way to compare model capabilities, but the credibility of results depends on how transparent the methodology is. When evaluating a benchmark report, check the following questions:
- Were prompts frozen before testing, and were there any post-hoc adjustments?
- Was the test environment fixed, including repository state, random seeds, and tool versions?
- Were intervention counts, failure reasons, and retry rules recorded?
- Were raters independent of the system under test, and were scoring criteria public?
- Can the results be reproduced by third parties?
For coding agent tasks, a single successful result has limited persuasiveness. A more reliable approach is to run multiple times, report the distribution of success rates, and record the range of time and cost.
9. Recommendations for Engineering Deployment
Based on the analysis above, teams can follow these steps when introducing a new large model architecture or deployment plan:
- Define scenario constraints: context length, concurrency, latency targets, and budget ceiling.
- Run small-scale stress tests on the target hardware, recording prefill, decoding, peak memory, and long-context accuracy.
- Compare at least two candidate approaches, including dense and sparse models, and evaluate them on the same test set.
- Check inference framework support status and prioritize kernel implementations that have been officially released.
- Roll out gradually, monitor latency, error rate, cost, and output quality, and keep a fast rollback path.
- Use ecosystem data as supporting information, not as a replacement for your own test results.
Conclusion
Improvements in large model architecture mainly revolve around three questions: the computation and cache cost of attention, the balance between parameter scale and computation, and memory management during inference. Understanding the principles and trade-offs of these mechanisms is the foundation for making reliable technical choices. At the same time, maintaining a clear distinction between ecosystem data and benchmark results can help avoid misreading market attention as engineering capability. Teams should rely on measurements from their own scenarios and complete the process from evaluation to launch step by step.
