Deep Dive into Next-Gen LLM Architectures: Delta Attention and Latent MoE

I. Background: Three Bottlenecks of Traditional Architectures in the Long-Context Era

In recent years, long-text understanding, multi-turn agents, and coding tasks have become the primary workloads for Large Language Models (LLMs). The traditional Transformer architecture exposes three structural issues in these scenarios:

1. Linear KV Cache Expansion: Standard softmax attention requires caching Key/Value vectors for every historical token. The memory footprint grows linearly with context length. When the context reaches hundreds of thousands of tokens, the cache for a single request can consume most of an accelerator’s memory, directly lowering concurrency.
2. Quadratic Complexity of Attention Computation: During the prefill phase, compute grows quadratically with context length, leading to high Time-To-First-Token (TTFT) for long documents and significantly dragging down overall throughput.
3. The MoE Memory Paradox: Mixture-of-Experts (MoE) reduces per-token compute via sparse activation, but all expert parameters must reside in memory. This creates a contradiction: larger nominal parameters save compute but consume more memory.

Recently discussed next-generation model architectures provide a combined answer to these bottlenecks: reconstructing attention layers with delta linear attention, reorganizing inter-layer structure with attention residuals (interleaving a few full-attention layers), and compressing expert parameters with Latent MoE. These three techniques target the attention layer, inter-layer organization, and FFN sparsification, respectively. Let’s break them down one by one.

II. Delta Linear Attention: From “Step-by-Step Retrieval” to “State Maintenance”

2.1 Basic Concept

The core shift in linear attention is treating historical information not as a “detailed list retrieved item by item,” but as a fixed-size state matrix maintained continuously:

• Naive form: $St = S{t-1} + k_t \cdot v_t^T$. For each incoming token, the key-value outer product is accumulated into the state. Retrieval is done via the inner product of the query and the state. The update cost per token is $O(d_k \cdot d_v)$, independent of sequence length.
• Obvious benefit: The KV cache shrinks from $O(n)$ to an $O(1)$ constant state, directly eliminating the major memory bottleneck in long-context inference.

2.2 The Delta Rule: Solving State Pollution

Naive accumulation has a clear flaw: repeated writes of similar keys degrade the state into a “blurred average,” diluting earlier information. The key improvement in Delta Attention is introducing the Delta rule, treating the state as a differentiable dictionary that “erases” old values before writing:

$St = S{t-1} \cdot (I - \beta_t \cdot k_t \cdot k_t^T) + \beta_t \cdot k_t \cdot v_t^T$

The intuition is to first project the old state using the current key $k_t$ ($S \cdot k_t$) to estimate the existing value at that key position, then subtract and add according to the write strength $\beta_t$, achieving a “key-aligned deduplication update.” By stacking a per-timestep exponential decay gate $\gamma_t$, older information naturally fades at a learnable rate, forming a gated delta memory mechanism. This concept traces back to fast-weight programming and associative memory in academia and is a recognized technical route.

The following pseudo-code illustrates the update for a single timestep:

# State update for Delta Attention (pseudo-code, gating and numerical details omitted)
S = torch.zeros(d_k, d_v)          # State matrix: size independent of sequence length
for t in range(seq_len):
    q_t, k_t, v_t = Q[t], K[t], V[t]
    beta_t = beta_proj(x[t])       # Write strength, value between 0 and 1
    S = S - beta_t * torch.outer(S @ k_t, k_t)   # "Erase" old value based on new/old key alignment
    S = S + beta_t * torch.outer(k_t, v_t)       # Write new key-value pair
    y[t] = q_t @ S                 # Read out: inner product of query and state

2.3 Training Parallelism

Although the recurrent form appears strictly sequential, engineering relies on chunked parallelism: the sequence is split into fixed-length chunks. Within a chunk, the recurrence is unfolded into parallelizable matrix multiplications using matrix algebra. Between chunks, only a small amount of state is passed sequentially. Combined with fused kernels, training throughput can be brought to a usable level. This is a critical engineering threshold for deploying such architectures—without efficient kernels, theoretical advantages cannot be realized.

2.4 Weakness: Weak Exact Retrieval

The state is essentially a fixed-size, lossy compression, naturally unfriendly to “needle in a haystack” exact recall. The model struggles to verbatim recall specific details from hundreds of thousands of tokens ago. This means delta attention is unsuitable as a standalone architecture and must be combined with full-attention layers—exactly the topic of the next section.

III. Attention Residuals: Inter-Layer Organization in Hybrid Architectures

3.1 “Filling the Gap” with Sparse Full-Attention Layers

The mainstream hybrid approach uses delta linear attention for the vast majority of layers, responsible for “compression and patterned memory.” A few standard full-attention layers are interleaved periodically, responsible for “exact retrieval and global alignment.” From the perspective of the residual stream, the main trunk is the continuous writing of linear memory, while full-attention layers act as “corrective overwrites” at critical depths. Thus, this is often referred to as attention residual design.

3.2 Three Design Key Points

• Ratio: The lower the proportion of full-attention layers, the greater the KV cache savings, but the weaker the retrieval capability. A rough estimation: for a 64-layer model, arranging 1 full-attention layer every 8 layers results in a KV cache roughly 1/8 that of a pure full-attention architecture. This can be further compressed using Grouped-Query Attention (GQA/MQA).
• Layout: Uniform interleaving is the most common approach. Some designs concentrate full-attention layers more in the shallow and middle layers to establish global alignment early.
• Positional Encoding Coordination: Linear layers typically rely on decay and gating for temporal information, while full-attention layers often use Rotary Position Embedding (RoPE). These two mechanisms must be coordinated under a unified training objective, which is a primary difficulty in training stability for hybrid architectures.

3.3 Benefits and Costs

The benefits are straightforward: the major memory overhead for long-context inference is removed; prefix states for multi-turn sessions can be reused at a fixed size, significantly reducing the marginal cost in Agent scenarios. The cost is increased architectural complexity: initialization, learning rates, and regularization strategies for both types of attention layers require independent tuning. Training instability is not uncommon in early experiments. The differences between the three forms are summarized below:

Dimension	Pure Full Attention	Pure Linear Attention	Hybrid Architecture + Latent MoE
KV Cache	$O(n)$, expands with length	$O(1)$ constant state	Few layers $O(n)$ + Majority $O(1)$
Exact Retrieval	Strong	Weak	Medium to Strong, depends on full-attention density
Long-Context Cost	High	Extremely Low	Significantly lower than pure full attention
Engineering Maturity	Very mature	Kernel and framework support still improving	Requires specialized adaptation

IV. Latent MoE: Fitting Experts into a Low-Rank Space

4.1 Pain Points of Standard MoE

MoE activates only a small number of FFN experts per token via a router, achieving “large parameters, low compute.” However, two issues persist: first, all expert parameters must reside in memory, limiting scale by memory constraints; second, unbalanced routing loads cause some experts to overheat while others are undertrained.

4.2 The Latent Expert Approach

Latent MoE applies low-rank decomposition to expert matrices: different experts share a set of basis matrices, and each expert only stores a low-dimensional latent vector or low-rank increment. During computation, the hidden state is projected into the latent space, interacts with the expert vector, and is projected back to the original dimension. Pseudo-code is as follows:

# Latent Expert FFN (pseudo-code)
h = x @ W_down                    # Project down to latent dimension r (r << d_ff)
z = routed_latents[expert_ids]    # Each routed expert only stores a low-dim latent vector
h = h * z                         # Lightweight expert transformation in latent space
y = h @ W_up                      # Project back to model dimension

The effect is that, without changing the routing structure, the memory increment required to expand “nominal expert capacity” drops from $O(N \cdot d_{ff} \cdot d)$ to $O(N \cdot r)$. In practice, a small number of shared experts are often paired to carry general knowledge, allowing routed experts to focus on specialization and alleviating expression conflicts caused by parameter coupling.

4.3 Routing and Load Balancing

Regardless of expert compression, load balancing is critical to training success. Auxiliary balancing losses, capacity factor limits, and routing consistency between training and inference are engineering details that must be handled properly. Because latent experts have tighter parameter coupling, balancing strategies must be more cautious to prevent the shared basis from being “hijacked” by a few high-frequency patterns, leading to insufficient expert differentiation.

V. Engineering Implementation and Ecosystem Signals

5.1 Inference Framework Adaptation is the First Hurdle

Custom operators required by new architectures (chunk scanning, fused gating, state management) often first appear as open-source PRs or draft branches, taking time before official merging and General Availability (GA). Before deployment, verify operator support in your inference engine version, rather than just checking if weight files are available. Furthermore, cache offloading (swapping attention states or KV cache to main memory or even SSD) is a common complementary direction to alleviate long-context memory pressure, but its maturity also needs verification. Whether building an in-house inference cluster or using GPU containers and model services from cloud platforms like Baidu AI Cloud, this step cannot be skipped.

5.2 Quantization Strategies

Low-bit weight compression (e.g., 4-bit floating-point formats) is generally applicable to the static weights of new architectures. However, the state matrix of linear attention is dynamically updated during inference, making its quantization sensitivity higher than that of static weights. It is recommended to empirically test the accuracy vs. throughput trade-off curve on target workloads, rather than directly adopting the quantization config of full-attention models.

5.3 Rational Interpretation of Adoption Signals

Common quantitative signals around new architectures include download counts on weight distribution platforms, distribution volumes on mirror platforms, and user/session counts in AI coding tools. When interpreting, maintain three boundaries:

• Distribution reach does not equal production usage: Download counts reflect weight acquisition popularity, which is different from actual online traffic.
• Window fluctuations are not directly causal: Rolling window ups and downs are influenced by statistical criteria and should not be interpreted directly as real usage shifts.
• Community benchmarks require methodology checks: Whether elements like fixed prompts, traceable intervention counts, and evaluator independence are present is more important than conclusive statements like “only successful” or “lowest cost.”

A more reliable approach is to build three types of evaluations internally: long-text exact retrieval hit rate, multi-turn Agent task success rate, and end-to-end cost per token. Let your own workload data speak for itself.

VI. Selection Recommendations and Conclusion

Scenarios suited for prioritizing hybrid architecture models:

• Contexts frequently exceeding hundreds of thousands of tokens, or Agent applications with long conversation histories;
• Online services with high concurrency, memory constraints, and sensitivity to per-token costs;
• Summarization and analysis tasks where “remembering patterns” is more important than “verbatim retrieval.”

Scenarios requiring cautious evaluation:

• Applications heavily reliant on verbatim exact recall (e.g., fine-grained citation comparison in long documents); verify that the full-attention layer density is sufficient first;
• Production environments sensitive to inference framework versions and unable to wait for ecosystem maturity.

Conclusion: Delta linear attention compresses the attention state from $O(n)$ to $O(1)$; attention residuals use a few full-attention layers to preserve exact retrieval; Latent MoE uses low-rank decomposition to reduce expert memory footprint. These three technologies target the attention layer, inter-layer organization, and FFN sparsification, respectively, all pointing to the same goal: flattening the cost curve for long-context and Agent workloads. Understanding their principles, benefits, and boundaries is essential coursework for evaluating next-generation model architectures.