Decoding Next-Gen LLM Architectures: Linear Attention, Residual Attention, and Latent MoE

1. Background: Why Flagship Models Are Swapping Their Engines

Over the past few years, mainstream large language models (LLMs) have predominantly relied on the standard Transformer recipe: softmax self-attention combined with dense feed-forward networks (FFN). However, as the push for longer contexts and higher knowledge density continues, this recipe has exposed two structural cost bottlenecks.

The first is attention cost. The computational complexity of self-attention grows quadratically with sequence length, and the KV cache expands linearly during inference. In scenarios like long-document QA and multi-turn Agent sessions, the KV cache often becomes the primary bottleneck for memory and throughput before compute does. Doubling the context length can halve the number of concurrent requests.

The second is parameter efficiency. Dense FFNs activate all parameters for every token, tightly coupling knowledge capacity with inference compute. To make a model “more knowledgeable,” you must pay proportional forward compute for every token, which contradicts the intuition that knowledge should be accessed on demand.

Therefore, the design focus of next-generation models has concentrated on three areas: introducing linear attention in attention layers (represented by gated delta attention), stacking cross-layer attention residuals on the residual pathway, and replacing FFN layers with latent Mixture of Experts (MoE). This article treats these three designs as general technical directions, dissecting their motivations, principles, and engineering implications layer by layer.

2. Gated Delta Attention: Equipping Linear Attention with Rewritable Memory

2.1 From Softmax to Linear Attention

Softmax attention can be viewed as “every token performing an exact retrieval against all historical tokens,” at the cost of quadratic computation and a linearly growing cache. Linear attention takes a different path: it maintains a fixed-size matrix state $S$ (shaped roughly $d_k \times d_v$). When a new token arrives, it updates this state at a constant cost and reads information from it using a query vector. This ensures that the “memory footprint” remains constant during inference, regardless of context length.

However, naive linear attention (like simple summation accumulation) has a fatal flaw: the state “only grows, never shrinks,” causing new and old information to interfere. As sequences get longer, early critical information is easily diluted by subsequent irrelevant updates, making exact retrieval significantly weaker than full attention.

2.2 Delta Rule: Retrieve-and-Overwrite Semantics

The Delta Rule provides a classic solution to this problem, with roots in associative memory and fast weight networks: treating context processing as an online learning problem. The state update can be written as:

$St = S{t-1}(I - \beta_t \cdot k_t \cdot k_t^T) + \beta_t \cdot v_t \cdot k_t^T$

Intuitively, $k_t$ is the “address,” $v_t$ is the “content,” and $\beta_t$ is the write gate. If the incoming key aligns with an existing address in the state, the update first “erases” the old value before writing the new one, achieving an overwrite-style modification. If the address is new, it is equivalent to inserting a new key-value pair. This upgrades the “accumulator that only grows” into an “addressable, rewritable memory,” significantly mitigating interference between old and new information. The single-head update logic is illustrated below:

S <- zero matrix (d_k x d_v)                 # Fixed-size recurrent state
for each token x_t:
    k_t <- W_k * x_t;  v_t <- W_v * x_t      # Address and content
    beta_t <- sigmoid(W_beta * x_t)          # Write gate
    old <- S * k_t                           # Retrieve old value at this address
    S <- S - beta_t * k_t * old^T + beta_t * k_t * v_t^T  # Overwrite
    y_t <- (W_q * x_t)^T * S                 # Query reads compressed memory

2.3 Fine-Grained Gating: From One Knob to Per-Channel Knobs

Early gated delta networks used scalar gates: the entire memory shared a single decay rate. Next-gen designs expand this to per-head, per-channel diagonal gating, effectively providing an independent forgetting knob for each feature dimension of the state. The result is a “multi-timescale memory”: some channels decay slowly, responsible for remembering slow variables like topic and style; others update rapidly, maintaining local references and temporary states. Per-channel decay is also believed to help mitigate the “memory over-squashing” problem in long sequences—where early critical information becomes unrecoverable after being washed out by subsequent updates.

2.4 Hybrid Stacking: Division of Labor

It is worth noting that models adopting these designs rarely replace all layers with linear attention. Instead, they interleave linear attention layers with a few full attention layers at a certain ratio (commonly ranging from 3:1 to 7:1 in the industry), sometimes穿插 local sliding window attention. The division of labor is clear:

• Linear attention layers handle “compressed memory”: Maintaining the summary state of long-range context, handling state tracking, and topic understanding.
• Full attention layers handle “exact retrieval”: Needle-in-a-haystack fact finding, symbol jumping in code, and exact citations in long documents. These tasks are highly sensitive to retrieval precision, which compressed memory cannot handle.

2.5 Engineering Benefits and Costs

On the benefit side, replacing a linearly growing KV cache with a constant state drastically reduces the memory footprint of long-context inference. This can be traded for larger concurrent batch sizes and higher throughput, which is particularly friendly for Agent-like applications where context only grows. On the cost side, the state update of the delta rule is sequential recurrence. Training requires specialized chunked parallel algorithms to fully utilize tensor cores, and the implementation threshold for backpropagation and operator fusion is high. On the inference side, mainstream frameworks need to adapt to new operators and state management paradigms. Ecosystem maturity usually lags behind model releases—a point that must be verified when evaluating production readiness.

3. Attention Residuals: Prior Knowledge from Previous Layers

3.1 Bottlenecks of the Standard Residual Stream

In a standard Transformer, each layer’s output is written as $x_{l+1} = x_l + Attn(x_l) + FFN(x_l)$, with all layers sharing a continuously accumulating residual stream. This provides a stable gradient pathway, but at depths of hundreds of layers, two hidden dangers emerge. First, information dilution: fine-grained information from early layers must pass through massive subsequent updates to reach deep layers, easily getting “averaged out.” Second, residual stream variance growth with depth, requiring meticulous initialization and scaling to maintain training stability.

3.2 Common Enhancement Techniques

Industry explorations on this pathway generally fall into three categories. Their commonality is minimal parameter overhead and primarily improving information flow:

1. Learnable residual scaling: Introducing learnable coefficients for each layer’s output to stabilize deep training.
2. Cross-layer attention residuals: Injecting the attention output from several layers prior into the current layer via gating. This allows the current attention to continue retrieving based on “existing retrieval results,” reducing redundant work and information loss.
3. Multi-residual streams: Maintaining multiple parallel residual channels, with layers exchanging information via learnable mixing coefficients, effectively widening the lanes of the “information highway.”

3.3 Chemical Reaction with Linear Attention

The combination of cross-layer residuals and linear attention is particularly intriguing. The state of linear attention is essentially lossy compression; “not remembering everything” is its nature. Cross-layer residuals provide a bypass, allowing precise information to skip the compression phase and reach deep layers directly, acting as a “fidelity channel” alongside the “compressed memory.” Furthermore, a smoother residual gradient flow alleviates the optimization difficulties introduced by new operators—this is a common combination punch in the training stability design of new architectures.

4. Latent Mixture of Experts: Continuous Combination Over Discrete Selection

4.1 Review of MoE Pain Points

Mixture of Experts (MoE) replaces dense FFNs with $N$ experts plus a router, activating only top-k experts per token. This expands parameter scale without proportionally increasing inference compute. However, traditional discrete routing has persistent issues: uneven expert utilization, load balancing heavily reliant on auxiliary loss tuning, memory footprint growing linearly with the number of experts, and noticeable routing decision jitter during early training.

4.2 Core Idea of Latent Experts

The idea of Latent MoE is to introduce a “latent expert bank”—a set of learnable low-rank latent vectors. A token’s effective expert is no longer “picking an off-the-shelf one” from the bank, but is generated on-the-fly by combining several latent vectors according to routing weights. This can be understood as a two-tier structure: shared experts handle general knowledge to guarantee a baseline; the latent combination part synthesizes an expert tailored to the current token via linear combinations of low-rank bases, trading combinatorial explosion for expressiveness:

# Form A: Routing weights combine latent bases to synthesize effective expert on the fly
w <- softmax(router(x))
A <- Sum w_i * bankA_i          # Combine low-rank left factor
B <- Sum w_i * bankB_i          # Combine low-rank right factor
y <- act(x * A) * B
# Form B: Convex combination of outputs from a few selected experts
{idx, w} <- topk(router(x), k)
y <- Sum w_i * Expert_idx_i(x)

4.3 Benefits and Costs

On the benefit side: a small number of latent vectors can combine to create effective experts far exceeding the bank’s capacity, yielding higher parameter efficiency. Combinatorial routing is smoother than hard discrete selection, helping to mitigate expert idling and routing jitter. On the cost side: training stability is more sensitive, typically requiring load balancing auxiliary losses, routing regularization, and cautious warmup strategies. On the inference side, expert combination introduces extra compute, or requires specialized kernel and communication optimizations. The all-to-all communication pattern under expert parallelism is also more complex. When evaluating such models, engineering teams should focus on expert load balance and inference kernel maturity.

4.4 Synergy of the Three Designs

Viewing these three elements together paints a clear picture of “memory layering”: full attention layers handle exact retrieval, linear attention states handle compressed long-range memory, and MoE’s parameterized knowledge handles solidified world knowledge. The memory saved on the attention side can be allocated to larger expert banks and longer contexts; MoE increases the knowledge capacity per unit of activated compute, compensating for the detail loss of compressed memory. Together, they shift the capacity-cost curve of “context memory + parameter knowledge” to the right.

5. A Rational Interpretation of Ecosystem Signals

Following the release of new architecture models, the community often sees a flurry of “data snapshot” discussions. Based on general industry experience, we recommend using the following framework to filter the noise:

1. Distribution reach is not production usage. Download counts and favorites on model hosting platforms reflect the reach of weight distribution. This is distinct from online API inference traffic and cannot be extrapolated between the two.
2. Do not over-extrapolate windowed metrics. Weekly or daily rolling token consumption and ranking fluctuations are heavily influenced by statistical methodologies. A drop in a windowed value does not equate to a drop in actual usage.
3. Framework support: “From proposal to GA.” A new operator or quantization format often takes weeks to go from community proposal, review, and merge to an official inference engine release. A pending proposal cannot be treated as an available capability.
4. Community benchmarks: methodology first. Fixed environments, identical prompts, traceable interventions, and independent evaluators are key elements for determining benchmark credibility. “First successful case” narratives require multi-source verification and should not be directly accepted.
5. Enterprise selection checklist. Before introducing a model into production, at least verify: quantization format maturity, mainstream inference engine adaptation status, real-world performance in long-context scenarios, cost per token, and security/compliance evaluation results.

6. Conclusion and Actionable Advice

Linear attention, attention residuals, and latent MoE represent the three main evolutionary paths for current LLM architectures: making memory cheaper, making information pathways more faithful, and making knowledge capacity more efficient. For developers, it is advisable to closely track metrics like KV cache footprint, long-text throughput, and expert load balance, and introduce them into production only after inference framework adaptation is mature. For enterprise users, the most reliable approach is to conduct small-scale POCs using real business workloads (long-document QA, multi-turn Agents, code tasks) and compare “cost per unit quality” rather than just looking at leaderboard scores. When building an evaluation environment on the cloud, you can leverage general cloud capabilities like GPU compute, container platforms, and logging/monitoring tools—such as the managed compute and observability toolchains provided by platforms like Baidu AI Cloud—to quickly establish a reproducible benchmarking process, grounding architectural decisions in measurable evidence.