New Approaches to LLM Architecture: Linear Attention, History Aggregation, and Latent-Expert Routing Explained
As context windows grow from thousands to hundreds of thousands of tokens, architectural choices become critical to both compute cost and serving throughput. This article walks through three active lines of architecture research: linear attention with gated state updates, aggregation of representations from earlier layers along the residual path, and latent-space routing for sparse Mixture-of-Experts models. Each section explains the core idea, its trade-offs, and practical engineering considerations, with simplified code examples illustrating the underlying computation. A final note covers how to evaluate whether these mechanisms are actually working during training.
1. Background: Why Architecture Design Matters for Long Contexts
The compute cost of standard scaled dot-product attention grows roughly with the square of sequence length. When input grows from a few thousand tokens to hundreds of thousands, both attention computation and memory consumption rise sharply. During inference, models must also maintain the key-value cache (KV Cache) for every layer, which grows linearly with sequence length and frequently becomes the bottleneck for concurrent serving.
Architecture work over the past two years has largely followed three directions: reducing the computational complexity of attention, improving how information flows through deep networks, and increasing model capacity without significantly increasing the number of activated parameters. This article covers each direction in turn, and closes with methods for evaluation and deployment.
2. Linear Attention and Gated State Updates
Linear attention replaces the softmax normalization with a kernel function that can be factorized, allowing attention to be expressed as a recurrent state update. For each token at position t, the model maintains a single fixed-size state matrix. Computational complexity grows linearly with sequence length, and inference no longer requires a full key-value cache that scales with length.
The main weakness of pure linear attention is weaker retrieval. The state has limited capacity, so precise long-range matching typically underperforms standard attention. A common engineering response is a gating mechanism: input-dependent decay coefficients let the model decide which historical information to keep and which to forget. Another common design is a hybrid architecture, where most layers use linear attention while a small number of layers retain full attention, balancing throughput against retrieval accuracy.
Below is a simplified single-head recurrent example. It illustrates the computation form only and does not reflect the details of any specific implementation:
import torchdef gated_linear_attention(q, k, v, g):# q, k: [T, d_k]; v: [T, d_v]; g: [T, d_k], decay coefficients in (0, 1)T, d_k = k.shaped_v = v.shape[-1]state = torch.zeros(d_k, d_v)outputs = []for t in range(T):state = g[t].unsqueeze(-1) * state + torch.outer(k[t], v[t])outputs.append(q[t] @ state)return torch.stack(outputs)
Real implementations use chunked parallel algorithms to fully utilize GPUs. The step-by-step loop above exists only to express the underlying math.
3. History Aggregation Along the Residual Path
Standard residual connections add each layer’s output directly to the backbone. As depth increases, representations from earlier layers become progressively diluted, and deep layers can only read a mixed signal that has been repeatedly superimposed. One improvement is to let each layer aggregate representations from several previous layers with weighted combinations, where the weights are either determined dynamically by the input or provided by learnable parameters.
The benefit of this design is that deep layers can directly read features held in shallow layers—such as local lexical or positional information—without relying on that information being fully preserved in the backbone. The cost is extra storage and computation. In practice, implementations typically limit the aggregation window and normalize the weights to keep training stable.
def aggregate_history(history, weights):# history: a list where each item has shape [B, T, D], the outputs of previous layers# weights: learnable parameters of shape [len(history)], normalized via softmax before usew = torch.softmax(weights, dim=0)return sum(wi * h for wi, h in zip(w, history))
When training this kind of structure, watch for two issues: whether the per-layer weights collapse into dependence on a single layer, and whether gradients are distributed evenly across layers. If the weights remain concentrated on one layer over time, the aggregation mechanism is not doing its job—check the initialization and normalization scheme.
4. Sparse Mixture-of-Experts in Latent Space
Sparse Mixture-of-Experts (MoE) splits the feed-forward network into multiple experts, with a router selecting a small subset of experts for each token. The total parameter count can be very large, but each token only activates a fraction of the parameters, so training and inference costs are driven mainly by the number of activated parameters.
The main overhead in this structure comes from high-dimensional expert computation and communication between experts. The latent-space approach works as follows: a linear projection first compresses the hidden states to a lower dimension, routing and expert computation happen in the low-dimensional space, and a final projection maps results back to the original dimension. Expert parameter count and computation scale down with the latent dimension, and communication volume decreases accordingly.
The risk of this approach is information loss from compression. The latent dimension requires a trade-off between quality and efficiency, usually determined through ablation experiments. The example below shows only the forward logic and omits load-balancing losses and expert parallelism:
class LatentRoutedMoE(torch.nn.Module):def __init__(self, d_model, d_latent, n_experts, top_k):super().__init__()self.down = torch.nn.Linear(d_model, d_latent, bias=False)self.up = torch.nn.Linear(d_latent, d_model, bias=False)self.router = torch.nn.Linear(d_latent, n_experts, bias=False)self.experts = torch.nn.ModuleList([torch.nn.Sequential(torch.nn.Linear(d_latent, 4 * d_latent),torch.nn.GELU(),torch.nn.Linear(4 * d_latent, d_latent),)for _ in range(n_experts)])self.top_k = top_kdef forward(self, x): # x: [N, d_model]z = self.down(x) # project down to latent spaceprobs = torch.softmax(self.router(z), dim=-1)topv, topi = probs.topk(self.top_k, dim=-1)out = torch.zeros_like(z)for e, expert in enumerate(self.experts):mask = (topi == e) # [N
