New-Generation LLM Architectures: Delta Attention, Attention Residuals, and Latent MoE

1. Why Architecture Innovation Is Back in Focus

Over the past two years, the center of gravity in LLM competition has shifted from “scaling up parameters” to “rearranging computation.” Three structural reasons sit behind this:

First, long context has become standard. Standard Transformer self-attention costs O(n²) compute during prefill, and the KV cache grows linearly with sequence length during inference. As context windows move from tens of thousands of tokens toward the million level, memory footprint and bandwidth cost become the first walls you hit.

Second, MoE has become the mainstream path to sparsity. Mixture-of-experts structures that activate only a subset of parameters per token make “large total parameter count, controllable compute” possible—but expert weights resident in memory, routing instability, and load imbalance shift the burden to the deployment side.

Third, the open-weight ecosystem amplifies the engineering gap. When an open-source model built on a new architecture is released, weight distribution volume often grows quickly—hosting platform downloads, mirror synchronization, and session counts in coding tools all rise first; but support in mainstream inference frameworks usually lags: merge requests are still under review, and backend support flags remain empty. This is a reminder that understanding a new architecture requires looking beyond the paper narrative to how it lands in a real inference stack.

The delta attention, attention residual, and latent mixture-of-experts (Latent MoE) designs discussed here are three of the most representative directions in recent public discussion.

2. Delta Attention: Replacing “Full Recall” with “Correction-Based Writes”

2.1 The Two Costs of Softmax Attention

Self-attention must retain key/value vectors for every historical token, which creates two costs: quadratic compute during training and prefill, and a KV cache memory cost that grows linearly during inference. Long documents, multi-turn dialogue, and repository-scale context all make these costs spiral quickly.

2.2 The Recurrent Formulation of Linear Attention

The idea behind linear attention is to rewrite attention as a state recurrence: maintain a fixed-size state matrix S, and each new token performs one read and one update. Memory relative to sequence length becomes constant, and throughput advantages on long sequences are obvious.

But naive accumulation introduces “state pollution”: old information continuously overwrites new information, or new writes crudely overwrite history that is still needed, causing degradation on precise retrieval tasks (finding details in long documents, long-range references in code).

2.3 The Key Step of the Delta Rule

The delta rule proposed in the literature provides a correction: before writing, first “recall”—use the current key to retrieve an estimate from the old state, subtract it from the true value, and correct the state only by that error. Mathematically this is equivalent to an online least-squares update with a rank-one correction, driving the state progressively toward “the optimal compression of historical key-value pairs.” Illustration:

def delta_attn_step(S, k, v, q, decay, beta):
    # S: (d_k, d_v) state matrix maintained across time steps
    S = decay * S                              # 1. Gated decay to prevent unbounded state growth
    recall = S.t() @ k                         # 2. Use old state to "recall" the value for this key
    S = S + beta * torch.outer(k, v - recall)  # 3. Apply outer-product correction based only on recall error
    return S, S.t() @ q                        # 4. Read out the current output

2.4 Gating and Hybrid Layouts

Related designs in public discussion typically include two points: first, fine-grained gating, refining decay control from a scalar to the channel or even feature dimension, so different information channels have different forgetting rates; second, hybrid ratios—pure linear attention still has weaknesses in precise lookup, so a common industry practice is to let linear layers handle most of the “state maintenance” work while inserting a small number of full-attention layers at fixed ratios for “precise retrieval,” for example one full-attention layer per several linear layers.

2.5 Engineering Impact

• Constant-size state makes long-context memory controllable, and prefill throughput benefits significantly;
• Training requires chunked parallel algorithms that unroll the recurrence into matrix operations to saturate compute;
• Inference requires frameworks to support “recurrent state” as a new cache form—which is the root reason why inference backends often lag after a new structure is released.

3. Attention Residuals: A Safety Net for Deep Stacks

3.1 Motivation

Residual connections are the cornerstone of stable training in deep networks. But when recurrent attention layers are mixed into the backbone, combined with more aggressive matrix-based/second-order optimization methods, the scale and distribution differences among sublayer outputs are amplified, and deep stacks become prone to training instability or loss spikes.

3.2 Common Techniques

Designs referred to in community discussion as “attention residual” typically mean a combination of the following general techniques: normalizing or scaling the attention branch output before residual addition to avoid magnitude mismatch; adding learnable gates to the residual path so the network decides “how much to trust”; and feeding upper-layer attention outputs back to lower layers at a ratio to mitigate deep-layer information decay. Note that these are general industry stability engineering techniques; specific implementation details for different models should be based on official technical reports.

3.3 Synergy with Hybrid Attention

In a backbone of “mostly linear layers + a few full-attention layers,” the output distributions of the two layer types naturally differ, and the design of the residual path directly determines whether the hybrid is stable. This explains why discussions of such architectures always pair stability design with inter-layer ratios.

4. Latent MoE: Moving Experts into a Low-Rank Space

4.1 The Old Problems of MoE

Mixture-of-experts structures have three long-standing pain points: all expert parameters must be resident in memory, so the larger the total parameter count, the higher the deployment barrier; routing suffers from jitter and load imbalance, making “winner-take-all” common during training; and coarse-grained experts have limited expressiveness while fine-grained experts amplify memory and scheduling pressure.

4.2 The Latent Approach

The direction referred to as “latent mixture-of-experts” is to change experts from “independent full matrices” to “constructed in a low-rank latent space”: maintain a set of shared low-rank latent bases, express each expert as a combination coefficient over the bases, with finer storage and quantization granularity; routing can also be performed on the compressed representation, reducing compute and communication overhead. Illustration:

def latent_moe(x, basis, coefs, router, top_k=2):
    # basis: shared low-rank latent bases; coefs: combination coefficients of each expert in latent space
    idx, w = top_k_select(router(x))        # Sparse routing
    experts = coefs[idx] @ basis            # Combine latent bases into the selected experts
    return sum(wi * e(x) for wi, e in zip(w, experts))

(The above is illustrative pseudocode for clarity and does not represent the implementation of any specific model.)

4.3 Relationship to Existing Techniques and the Deployment View

This can be seen as a continuation of industry ideas such as shared experts, fine-grained experts, and low-rank compression: shared experts solve “not storing common capabilities redundantly,” and latent bases further compress “inter-expert redundancy.” For deployment, more regular expert weights are friendlier to low-bit compression such as 4-bit microscaling floating point (open formats like MXFP4); sharded weight files facilitate resumable downloads and on-demand loading; and the maturity of expert parallelism and all-to-all communication determines the actual throughput ceiling.

5. Putting the Three Together: A Sketch of a Hybrid Backbone

class HybridBlock(nn.Module):
    # Sketch: most layers use delta attention, a few use full attention, FFN uniformly goes through latent MoE
    def forward(self, x, state=None, kv_cache=None):
        if self.layer_type == "delta_attn":    # Constant-size recurrent state
            x, state = self.delta_attn(x, state)
        else:                                  # A few precise-retrieval layers
            x = self.full_attn(x, kv_cache)
        x = x + self.gate * self.latent_moe(self.norm(x))  # Gated residual
        return x

The division of labor is now clear: delta attention handles “state cost,” residual design handles “training stability,” and latent MoE handles “expert memory.”

6. How to Evaluate a New Architecture Model Calmly

Given recent community activity, a verification-oriented approach is recommended:

1. Distribution signals ≠ usage signals. Hosting platform download counts and mirror sync numbers only indicate reach; session counts and token share in coding tools are relative in-platform measures and cannot be extrapolated to global production traffic; a decline in rolling window values does not necessarily mean a real usage decline.
2. Look at absolute growth and relative position together. Absolute adoption may grow modestly while relative share comes under pressure; only the combination reveals the ecological niche.
3. Ask about methodology in evaluations. Agent evaluations with fixed environments, identical prompts, and traceable interventions are more informative, but whether evaluator independence holds and whether sample sizes are sufficient determines whether conclusions generalize; community summaries of the “only success, a few minutes, a few dollars” variety are clues at best.
4. Safety evaluations also need verification. Results such as refusal benchmarks depend heavily on evaluation protocols and reproduction details.
5. Check framework support maturity. Merge requests not yet merged and empty backend support flags mean the capability is not yet officially available; whether you run your own inference cluster or use a cloud platform’s managed LLM service such as Baidu AI Cloud, the officially supported backend version for the new architecture should be your reference.

7. Conclusion

The value of architecture innovation ultimately comes down to three things: whether training is stable, whether inference is cost-effective, and whether capabilities are verifiable. Delta attention compresses the state cost of long context, residual design safeguards stable training of deep hybrid backbones, and latent MoE lowers the expert memory barrier—all three point to the same goal: under the same compute budget, spend every unit of memory and bandwidth where it matters more. For engineers, more important than memorizing the terms is understanding the bottleneck each design responds to, and confirming before deployment that the inference stack, quantization scheme, and evaluation methodology have caught up.