Next-Gen LLM Architectures: Delta Attention and Latent MoE

I. Background: The Two Walls of the Classic Transformer

The standard “Self-Attention + Feed-Forward Network (FFN) + Residual Connection” architecture has been the default backbone for mainstream LLMs for years. However, as context lengths push toward hundreds of thousands or even millions of tokens, and task paradigms shift toward multi-step Agents and long-document processing, two distinct walls have become increasingly prominent:

• Compute and VRAM Wall: The computational complexity of softmax self-attention grows quadratically with sequence length. During inference, the KV cache expands linearly with context size, often exceeding the model weights themselves in long-context scenarios, becoming the largest consumer of VRAM.
• Parameter Efficiency Wall: Dense FFNs activate all parameters for every token. Because the knowledge required by different tokens is highly uneven, vast amounts of compute power are consumed inefficiently, driving up both training and inference costs.

To overcome these two walls, three key concepts have repeatedly emerged in recent architectural discussions: Incremental Linear Attention (Delta Attention), Attention Residuals (AttnRes), and Latent Mixture of Experts (LatentMoE). These target the three fundamental contradictions of “memory cost, training stability, and knowledge cost,” respectively. This article breaks down their principles, motivations, and engineering implications, offering evaluation and deployment recommendations.

II. Incremental Linear Attention: Turning the KV Cache into a Fixed-Size State

2.1 Basic Paradigm: From “Per-Token Retrieval” to “State Read/Write”

Softmax attention is expensive because generating each token requires computing the similarity against all historical tokens. Linear attention takes a different path: it compresses “reading history” into a fixed-size state matrix $S$. As tokens flow through, only a single write and a single read operation are performed:

S_t = S_{t-1} + φ(k_t)·v_tᵀ
y_t = φ(q_t)ᵀ·S_t

Complexity drops from $O(n^2)$ to $O(n)$, and inference VRAM becomes independent of context length. The per-layer KV cache that previously grew with sequence length is replaced by a constant state matrix of size $d_k \times d_v$.

2.2 Delta Rule: Error-Corrective Writing

Naive linear attention only performs addition; historical information goes in but never comes out. As the context grows, the interference (dilution) of irrelevant old content on new writes becomes severe. Borrowing from associative memory concepts, the Delta Rule reads out the old value at a given key position and subtracts it before writing the new value—effectively acting as an “overwrite”:

S_t = S_{t-1} + β_t · k_t·(v_t − S_{t-1}ᵀ·k_t)ᵀ

Here, $\beta_t$ is a learnable write-strength gate, and some designs also introduce a forget gate for global decay. Intuitively, the state acts like an overwritable key-value store: when the same key appears repeatedly, the new value overwrites the old one rather than accumulating infinitely.

# Incremental linear attention inference illustration (single head)
# S ∈ R^{d_k × d_v}: Fixed-size persistent state, replacing the linearly growing KV cache
for t in range(seq_len):
    k = W_k(x[t])                          # [d_k]
    v = W_v(x[t])                          # [d_v]
    q = W_q(x[t])                          # [d_k]
    r = S.T @ k                            # Read: previous old write for this key
    S = S + beta * torch.outer(k, v - r)   # Error-corrective write
    y[t] = S.T @ q                         # Read current output

2.3 Training Parallelization and Hybridization

The recurrent form is not GPU-friendly for parallelization. The standard industry approach is a chunk-wise algorithm: computing in parallel within blocks using attention-like methods, and passing states between blocks. This balances training efficiency with linear complexity.

However, it is crucial to recognize that a linear state is inherently lossy compression. Tasks requiring “exact per-character copying or precise positioning” are weaker under linear attention compared to softmax attention. Therefore, mainstream approaches do not completely replace softmax but rather use hybrid stacking: a small number of global attention layers handle precise retrieval, while a large number of linear attention layers manage long-term memory.

III. Attention Residuals (AttnRes): Making Deep Stacking More Stable and Accurate

3.1 The Crowding Problem of a Single Residual Stream

In standard architectures, all sublayers share a single residual stream. As depth increases, the outputs of each layer continuously accumulate. The magnitude of the attention branch’s output is relatively small and easily “drowned out” by the FFN branch. Additionally, gradient paths become increasingly fragile with depth, raising the risk of training instability.

3.2 Three Common Techniques

1. Learnable Scaling/Gating: Introducing learnable per-layer or even per-head coefficients for the attention branch, allowing the network to decide whether information should bypass via a shortcut or pass through the branch.
2. Normalization Adjustments: Such as sandwich normalization, placing normalization both before and after the residual summation to suppress magnitude drift.
3. Attention as a Sparse Correction Term: In hybrid architectures where linear attention is the backbone, a few global attention layers can be treated as “residual corrections” to the backbone representation. The backbone maintains long-term memory, while global attention intervenes only when precise alignment is necessary.

3.3 Benefits

Training becomes more stable (loss curves for deep stacks are smoother), effective depth increases, and long-range retrieval capabilities are preserved. For models with hundreds of layers, these seemingly minor structural tweaks often determine whether training can successfully scale.

IV. Latent Mixture of Experts (LatentMoE): Compressing the VRAM Cost of Sparsification

4.1 The Fundamental Contradiction of MoE

MoE uses a router to activate only a few expert FFNs per token, achieving “many parameters, little compute.” However, the bottleneck lies in VRAM and communication: more experts mean larger weight storage and cross-GPU communication overhead; fewer experts result in insufficient capacity.

4.2 The Latent Approach

• Shared Basis + Lightweight Combination: All experts share a set of latent basis vectors. An individual expert is merely a combination of coefficients on this basis, meaning expanding the number of experts no longer linearly increases VRAM.
• Compressed-Domain Computation: The low-rank latent representations produced by the attention side serve directly as the input/output space for the expert layers. The expert transformation occurs in the $d{latent}$ dimension rather than $d{model}$, simultaneously reducing compute and memory access.
• Fine-Grained + Shared Experts: A large number of small experts improves routing precision, while a few shared experts cover common knowledge as a fallback.

# Latent MoE illustration: Compressed-domain routing + sparse activation
z = W_down(h)                             # [B, d_latent] dimension reduction
logits = router(z)                        # [B, E]
w, idx = torch.topk(logits, k=2)          # Activate only a few experts
g = torch.softmax(w, dim=-1)
out = torch.zeros_like(z)
for i in range(2):
    out += g[:, i:i+1] * experts[idx[:, i]](z)
h = h + W_up(out)                         # Return to model dimension and apply residual

4.3 Trade-offs and Regularization

The coupling between routing and experts is tighter, making load balancing more sensitive. This typically requires balance losses, routing entropy regularization, and other techniques. The communication patterns under expert parallelism also require specific optimization within the inference engine.

V. The Combination: A System View for Long Contexts

Piecing these three directions together reveals the complete logic of next-generation architectures: Incremental linear attention turns the VRAM cost of “memory” into a constant; attention residuals ensure that stacks of hundreds of layers can be trained stably; Latent MoE compresses the VRAM cost of “knowledge” to a minimum.

The cascading effects on the inference side require close attention from engineers:

1. State Management Replaces KV Cache Management: The linear layer state is a request-level constant, but the state lifecycle under concurrency, preemption recovery, and multi-turn dialogue breakpoint resumption (saving the state at the end of each turn) present new engineering challenges.
2. Prefix Reuse Becomes Difficult: Linear states depend on the complete prefix. Cross-request reuse requires explicit state checkpoints, and the benefits model differs from the KV cache era.
3. Chunk-wise Prefill Becomes Standard: Prefill advances block by block, passing states between blocks.
4. Tiered Expert Caching: Expert weights are tiered between VRAM and main memory based on routing heat, and can be coordinated with low-bit quantization formats like microscaling floating-point to further compress footprint.

# Hybrid stacking illustration: Periodically insert global attention (interval varies by model)
for i in range(num_layers):
    if i % N == 0:
        h = GlobalAttention(h)             # Precise retrieval
    else:
        h = DeltaLinearAttention(h)        # Long-term memory
    h = LatentMoE(h)

VI. How to Rationally Evaluate and Deploy

Advanced architecture does not mean out-of-the-box usability. Evaluation should be based on four dimensions:

1. Capability: Multi-hop retrieval (needle-in-a-haystack and variants), long-document QA, repository-level code understanding. Focus on whether “exact copying” tasks degrade due to a high proportion of linear layers.
2. Performance: Throughput, time-to-first-token (TTFT), and peak VRAM across different context lengths. The advantages of linear layers are concentrated in the decode phase and ultra-long contexts.
3. Ecosystem: Whether inference frameworks natively support new operators, whether quantization schemes are complete, and whether distributed state management is mature. The engineering ecosystem for new architectures often lags behind standard attention by months.
4. Signal Filtering: Weight distribution downloads, dev tool usage statistics, and community benchmarks are only indirect signals. Community evaluations may lack independence. Ultimately, you must return to offline testing on your own business workloads.

Regarding deployment paths, whether building your own inference cluster or using cloud platforms like Baidu AI Cloud (which provides GPU compute, model serving, and monitoring/alerting capabilities), it is recommended to first conduct a small-traffic canary test to verify the real benefits of long-context scenarios before gradually scaling up.

VII. Conclusion

Incremental linear attention, attention residuals, and Latent MoE address the three fundamental contradictions of “memory cost, training stability, and knowledge cost,” respectively. Their combination outlines the contour of next-generation LLM architectures: longer contexts, lower per-token costs, and more stable large-scale training. For developers, understanding the trade-off logic behind these structures is more important than memorizing the terms themselves—it determines whether you can accurately judge where a new model is strong, where it saves resources, and where the pitfalls lie during selection, deployment, and tuning.