Decoding Next-Gen LLM Architectures: Delta Rule Linear Attention and Latent MoE

I. Background: Why Attention and MoE Are the Frontlines of Architectural Evolution

Over the past two years, the context length of large language models has expanded from thousands of tokens to hundreds of thousands or more. Meanwhile, agentic applications require models to maintain stable memory across multi-turn tool calls and extended sessions. Consequently, traditional softmax self-attention faces two hard constraints:

• Quadratic compute growth with sequence length: Prefill and decoding costs spiral out of control under long contexts.
• Linear KV Cache expansion with context: The memory bottleneck for long-context inference is primarily concentrated here.

On the other hand, the Mixture-of-Experts (MoE) architecture expands parameter capacity through sparse activation, but shifts the cost from “compute” to “memory and I/O”. Expert weights constitute the bulk of the model size, making downloading, loading, and multi-GPU deployment significantly heavier.

Thus, next-generation architectural improvements focus on three directions: compressing attention history into a fixed-size state (linear attention), using a few full attention layers to retain exact retrieval capabilities (attention residual mixing), and compressing the expert parameters themselves (Latent MoE). Let’s break these down one by one.

II. Delta Rule Linear Attention: Turning “Memory” into a Maintainable State

2.1 From “Reviewing All History” to “Maintaining a State”

Standard attention computes similarity against all historical tokens at every time step, essentially acting as “retrieval-based memory.” Linear attention takes a different approach: compressing historical information into a fixed-size state matrix $S$. When a new token arrives, it only requires a single state update and a single read, reducing computational complexity from $O(n^2d)$ to $O(nd^2)$. During inference, the state’s memory footprint is independent of the sequence length.

The most basic form is “outer product accumulation”: $St = S{t-1} + k_t \cdot v_t^T$. However, simple summation accumulates indefinitely—old information is never diluted, and new information struggles to be written. Over long sequences, the state quickly becomes “blurred.”

2.2 Decay Gating and the Delta Rule

Two types of improvements address this issue:

• Forget Gate: Applies decay to the state at each step, allowing older information to fade out as needed and preventing infinite state expansion.
• Delta Rule: Before writing a new key-value pair, it “erases” the old association in the state that is similar to the current key, then writes the new value. This is equivalent to changing “append-only write” to “overwrite”:

$St = S{t-1} - \betat \cdot k_t \cdot (S{t-1}^T k_t)^T + \beta_t \cdot k_t \cdot v_t^T$

Intuitively, $S^T k$ uses the old state to “recall” the key $k$. Subtracting its projection in the $k$ direction deletes the old value at that position and writes the new one. This gives the state the ability to “update by same key,” which is significantly more friendly for tasks requiring precise memory, such as code variable tracking and dialog state maintenance.

Schematic pseudocode (single-head; actual training uses equivalent chunked parallel implementations):

S = zeros(d_k, d_v)                        # Fixed-size state matrix
for t in range(seq_len):
    q, k, v = Wq(x[t]), Wk(x[t]), Wv(x[t])
    g     = sigmoid(Wg(x[t]))              # Channel-wise decay gate
    beta  = sigmoid(Wb(x[t]))              # Write strength
    S     = S * g                          # Forget: decay state by channel
    S     = S - beta * outer(k, S.T @ k)   # Erase: weaken old associations similar to current key
    S     = S + beta * outer(k, v)         # Write: establish new key-value association
    y[t]  = S.T @ q                        # Read: query the current state

2.3 Fine-Grained Gating: From Scalar to Channel-Wise

Early gating used a single scalar per head, where state transition approximated “global scaling + low-rank update.” Next-gen designs scale gating to channel-wise or even dimension-wise, introducing non-linearity so the state transition matrix possesses the finer expressive power of “diagonal + low-rank.” Each feature channel can forget and write at its own pace. This significantly improves the capacity and selectivity of state memory, at the cost of more complex operator implementation.

2.4 Advantages and Inherent Limitations

• Advantages: Constant-sized state, near-linear compute complexity, and high throughput for long-context prefill.
• Limitations: State is ultimately lossy compression. For tasks like “ultra-long-distance word-for-word copying or exact citation of original text,” pure linear attention is still weaker than full attention. This shortcoming is exactly the starting point for the hybrid design in the next section.

III. Attention Residual Mixing: Using a Few Full Attention Layers to Secure Exact Retrieval

3.1 Not a Binary Choice

The mainstream approach is not “replace everything with linear attention,” but rather letting the vast majority of layers use linear attention for efficient information aggregation, and inserting a standard full attention layer every few layers (common ratios range from 1:3 to 1:8, varying by model). This is the so-called attention residual: the full attention layer is not the backbone, but a “precise retrieval pathway” attached to it.

3.2 The Semantics of “Residual”: Periodic Global Calibration

This can be analogized to a synchronization frame in a communication system: linear layers continuously perform low-cost approximate accumulation, while full attention layers periodically rebuild a global index, correcting accumulated errors and supplementing information that requires verbatim precise access. For retrieval-augmented generation (RAG) and long-document Q&A tasks, a handful of full attention layers can often pull recall precision back to a level close to full-attention models.

3.3 The Cost-Benefit Account

Assume 1 full attention layer for every $r+1$ layers. The KV Cache is roughly $1/(r+1)$ of a similarly sized full-attention model. By stacking latent compressed KV caching (projecting KV into a lower-dimensional vector before caching, a mature practice in the industry), the memory footprint for long-context scenarios can be reduced to a fraction of the original or even less. The speed advantage during the prefill phase will be significantly amplified as context length grows.

3.4 Engineering Essentials

• Positional Encoding: Linear and full attention layers depend on Rotary Position Embedding (RoPE) differently; they must be handled separately during implementation.
• Operator Adaptation: Chunked parallel kernels for linear attention and the scheduling of mixed layers constitute the bulk of inference stack adaptation work.
• Training Stability: Differences in gradient scales across different layers require coordination between normalization and initialization.
• Serving: General optimization techniques like prefix caching, KV quantization, and hierarchical offloading remain applicable and yield more pronounced benefits.

IV. Latent MoE: Compressing Experts into a Low-Dimensional Space

4.1 The Old Debt of MoE

MoE trades “sparse activation” for capacity: the total parameter count is huge, but each token only activates a small fraction. However, expert weights exist statically—regardless of whether they are activated, they occupy memory. Model size, weight loading time, and the communication and replication costs of multi-node deployment all grow with the number of experts.

4.2 Latentization: Synthesizing Expert Weights via Shared Structures

The Latent MoE approach no longer allows each expert to independently hold complete FFN weights. Instead, expert weights are synthesized from a “shared base + low-rank projection,” or generated from a set of shared latent dictionaries. The router no longer outputs a simple expert ID; it can select top-k latent codes in the latent space, which then synthesize the actual expert used for the current forward pass:

# Conventional MoE: Experts independently hold full weights
y = sum(g[i] * FFN_i(x) for i in topk(router(x)))
# Latent MoE (schematic): Expert weights = shared base + low-rank factor synthesis
# W_i = W_shared + U_i @ V_i, where rank r is much smaller than d_ff
for i in topk(router(x)):
    W_i = W_shared + U[i] @ V[i]     # Dynamically synthesize expert from latent factors
    y  += g[i] * (x @ W_i.T)

(The above is a schematic representation; real implementations fuse computations at the kernel level to avoid explicitly synthesizing large matrices.)

4.3 Benefits

• Parameter count and memory footprint decrease; weight downloading, loading, and quantization are lighter.
• The shared expert base effectively enforces knowledge sharing, helping to mitigate “expert degradation” and “routing collapse.”
• More friendly to single-node and edge deployment, offering a viable path to “pack large capacity into small memory.”

4.4 Risks and Trade-offs

• Low-rank constraints may limit expert diversity; ablation studies are needed to verify quality under the same activated parameter count.
• Routing and low-rank factors require joint training, which may cause instability in early training stages.
• During quantization, precision allocation between the shared base and low-rank factors needs independent verification. The impact of low-bit formats on the error of synthesized weights is not intuitively additive.

V. The Overall Logic of Combining the Three

Looking at the three components together, the division of labor is very clear:

• Delta Rule Linear Attention: Responsible for “state-compressed global scanning,” providing memory footprint and high throughput independent of sequence length.
• Attention Residual: Responsible for “precise retrieval,” using a very small number of full attention layers to secure retrieval and citation accuracy.
• Latent MoE: Responsible for “low-cost capacity expansion,” expanding parameter capacity without linearly increasing memory.

A simplified complexity account:

• Full Attention Model: KV Cache $\propto$ layers $\times$ sequence length $\times$ KV dimensions.
• Hybrid Model (1:r ratio): KV Cache $\approx 1/(r+1)$ of the full attention model.
• Linear Layers: State is fixed at $O(d^2)$, independent of sequence length.
• Latent MoE: Total parameters $\approx$ shared base + number of experts $\times$ low-rank factors, significantly smaller than independent experts.

Their goals are aligned: redistributing the limited “precision budget.” Routine information aggregation uses cheap state updates; precise access utilizes expensive full attention; capacity expansion amortizes costs via shared structures. For developers, this means the same memory budget can serve longer contexts and larger concurrent batches.

VI. Evaluation and Deployment Recommendations for Developers

6.1 Distribution Metrics $\neq$ Production Readiness

After new architecture models are released, metrics on model hosting platforms like download counts, image sync numbers, and tool-side user counts are “reach signals.” They only indicate breadth of distribution, not actual production invocation volumes. Statistics on sessions and tokens within a rolling window are also affected by the windowing criteria and cannot be directly interpreted as fluctuations in real usage. Model selection should be based on internal business regression sets and empirical stress testing.

6.2 Monitor Framework Support Maturity

Hybrid attention and Latent MoE impose new requirements on the inference stack: chunked kernels for linear attention, scheduling for mixed layers, all-to-all communication for MoE, and operator support for specialized quantization formats. Early on, a common window exists where “weights are public, but framework support is not officially ready.” The community will continuously submit adaptation work like cache offloading and memory tiering—until these are merged and released, they should not be considered available capabilities.

6.3 Evaluation Dimension Checklist

• Quality: Long-context recall and citation benchmarks + internal business regression sets.
• Performance: Throughput-memory curves, batch inference cost, time-to-first-token (TTFT), and decoding speed.
• Stability: Routing balance, long-session state consistency.
• Methodology: When referencing community benchmarks, check evaluation independence. Fixed prompts and reproducible environments are pros, but if the evaluators lack independence from the evaluated parties, conclusions should be treated as clues rather than evidence.

6.4 Scenario Recommendations

• Advantageous Scenarios: Long-session agents, repository-level code understanding, batch long-document processing, high-concurrency multi-turn services.
• Scenarios Requiring Empirical Testing: Tasks demanding ultra-long verbatim citations or complex multi-hop retrieval. A/B testing against a full-attention baseline is recommended.

Whether building an in-house inference cluster or using general cloud compute and model services (such as container platforms, GPU compute, and model hosting capabilities provided by Baidu AI Cloud and others), the above evaluation dimensions apply equally.

VII. Conclusion

The three keywords of next-generation architectures—Delta Rule Linear Attention, Attention Residual, and Latent MoE—essentially answer the same question: within the triangle of precision, memory, and throughput, where is it most cost-effective to spend the budget? For developers, understanding the mechanism is more valuable than memorizing the terminology. By grasping “how the state updates, who guarantees precise retrieval, and how capacity amortizes costs,” you can evaluate every newly released architecture using a unified framework, rather than being led by promotional metrics.