Decoding Next-Gen LLM Architectures: Linear Attention and Latent MoE

I. Redesigning Attention: The “Quadratic Tax” of Long Contexts

The core of the Transformer architecture is softmax attention, which calculates the similarity between every pair of tokens. This provides powerful global modeling capabilities but incurs two costs that scale with context length:

• Computational Cost: During the prefill stage, attention computation scales at O(N²·d). Doubling the context length quadruples the attention overhead.
• Memory Cost: During inference, the KV cache scales at O(layers×N×d), expanding linearly with sequence length. In long-context scenarios, memory capacity and memory bandwidth become the primary bottlenecks.

The industry commonly follows two paths to address this. The first retains exact attention but applies sparsification, sliding windows, or retrieval-based compression. The second rewrites attention itself to recurse via a “fixed-size recurrent state,” as seen in linear attention and state space models (SSMs). Recent architectural models over the past year rarely choose just one; instead, they adopt a hybrid approach: the backbone uses linear attention to process the vast majority of tokens, while a small proportion of full attention layers are inserted as a fallback. The following sections break down the complete stack: linear backbone, full attention residual, latent MoE, low-bit quantization, and maturity assessment.

II. Delta Linear Attention: Rewriting Attention as “Memory with Error Correction”

2.1 From Cumulative Outer Products to the Delta Rule

The most basic form of linear attention compresses history into a matrix state S. For each incoming token, it performs an outer product accumulation: S ← S + v·kᵀ, and reads via y = S·q. The problem with pure accumulation is the lack of an “overwrite” mechanism. When the same key reappears, the old value is not replaced, causing the state to blur together.

The Delta rule draws inspiration from online least squares. Before writing a new key-value pair, it estimates the “readout of the old state in the direction of the current key,” cancels it out from the state, and then writes the new value:

S_t = S_{t-1}·(I − β_t·k_t·k_tᵀ) + β_t·v_t·k_tᵀ

Here, β_t is a learnable write strength. Intuitively, when key k_t collides with historical records, it first subtracts the old association and then adds the new one. The state transitions from “infinite accumulation” to “overwriting updates,” significantly improving memory capacity utilization and fault tolerance.

Pseudo-code (recursive form, omitting parallelization details):

S = torch.zeros(d_v, d_k)                # Fixed-size associative memory state
for t in range(T):
    q, k, v = inputs[t]                  # q,k ∈ R^d_k，v ∈ R^d_v
    beta = write_strength[t]             # Learnable write strength
    S = channel_gate[t] * S              # Per-channel decay (forget gate)
    S = S @ (torch.eye(d_k) - beta * torch.outer(k, k))  # Cancel old value
    S = S + beta * torch.outer(v, k)     # Write new value
    y[t] = S @ q                         # Linear complexity readout

2.2 Gating and Per-Channel Decay

Building on the Delta rule, newer implementations commonly introduce fine-grained gating. Instead of applying a single scalar forget rate to the entire state, they apply diagonalized decay (element-wise multiplicative gating) across the state’s channel dimension. This allows the model to learn differentiated strategies for “how long to keep specific memory channels”—syntactic and local reference information decays quickly, while entity and style information has a longer shelf life. Functionally, this unifies the dual needs of “sliding window” and “global compression” into a single set of learnable state dynamics.

2.3 Parallel Training and Complexity

The recursive form appears inherently non-parallelizable, but engineering has mature chunked parallel algorithms. Sequences are divided into chunks; within a chunk, WY-style transformations express multiple Delta updates as matrix multiplications, and only states are passed between chunks. Consequently, the training side achieves hardware utilization on par with standard attention, while the inference side maintains O(N·d²) linear complexity and a constant-size state. For million-token contexts, the memory profile flattens from “linear growth with length” to an “almost horizontal line.”

2.4 Capability Boundaries

The linear state is essentially “lossy compression.” When facing exact retrieval tasks—such as verbatim quoting a specific line from page 37—a fixed-size state inherently falls short. This is not an implementation flaw but an information-theoretic trade-off, which is exactly why the hybrid design in the next section exists.

III. Full Attention Residual Layers: Fallback with Sparse Exact Retrieval

3.1 Design Space for Hybrid Ratios

The mainstream approach is to insert full attention layers at fixed intervals. For example, 1 full attention layer for every 3 to 7 linear attention layers. Recent public research mostly falls in the 3:1 to 7:1 range, with around 3:1 showing more stable performance on retrieval benchmarks. Because full attention layers are in the minority, the total KV cache remains an order of magnitude lower than in pure full-attention models, while these layers guarantee retrieval capabilities.

3.2 The Dual Meaning of “Residual”

This design is often summarized as “full attention residual,” where “residual” carries two meanings:

• Engineering meaning: The inserted full attention layers follow standard residual connections and pre-normalization with the backbone, ensuring clean gradient paths and training stability consistent with a pure Transformer.
• Functional meaning: Full attention layers act as an “exact recall channel” overlaid on the linear backbone. The vast majority of token-level interactions (local syntax, semantic smoothing) are handled by the linear state, while a small amount of information requiring exact alignment (quotes, code symbols, long-range entity binding) is supplemented by the full attention layers. The two pathways naturally fuse in the residual stream without explicit routing.

3.3 Supporting Stability Details

Several common practices exist in hybrid stacks: normalizing Q/K before attention scoring to suppress numerical spikes; scaling the output of full attention layers to prevent a few layers from dominating the residual stream; and using initialization compatible with gating for linear attention layers to ensure controllable state update magnitudes early in training. These individual details are not complex, but together they determine whether a hybrid architecture can be trained out-of-the-box.

3.4 Layer Scheduling Example

class HybridBlock(nn.Module):
    def forward(self, x):
        for i, layer in enumerate(self.layers):
            if i % self.full_every == 0:      # Sparse insertion
                x = x + layer.full_attn(x)    # Exact retrieval channel
            x = x + layer.delta_attn(x)       # Linear backbone
            x = x + layer.moe_ffn(x)          # Latent MoE feed-forward
        return x

IV. Latent Mixture of Experts: Compressing Experts into Low-Rank Spaces

4.1 The Two Costs of MoE

Mixture of Experts (MoE) uses a router to send each token to only the top-k experts, achieving sparsity of “many parameters, low computation.” However, in traditional implementations, each expert is a complete feed-forward network. The ledger still carries two overheads: parameter memory scales linearly with the number of experts, and expert weight loading is limited by memory bandwidth, making weight搬运 a potential new bottleneck during large-batch inference.

4.2 The Latent Approach

The starting point for “latent experts” is to change expert parameters from independent full matrices to low-rank latent representations. Common approaches include:

• Shared Base + Low-Rank Increment: All experts share a base matrix. Each expert only holds a low-rank increment W_e = W_base + B_e·A_e, where the rank is much smaller than the hidden dimension.
• Latent Bottleneck Projection: Experts first pass through a shared projection to compress into a latent dimension, perform expert-specific transformations in the latent space, and then project back.
• Shared vs. Routed Expert Division: A small number of shared experts are permanently active to process general knowledge, while routed experts handle conditional computation only.

Combined, these methods significantly reduce parameter count and memory footprint for the same number of experts while retaining the compute advantages of sparse activation. For inference systems, this means smaller weight transfer volumes and higher batch limits.

4.3 Training Stability and Inference Costs

General MoE engineering principles apply here: auxiliary load balancing losses and router entropy regularization prevent “winner-takes-all” scenarios; expert capacity and token dropping policies control memory peaks; and router computations require attention to numerical precision. On the inference side, systems must handle all-to-all communication brought by expert parallelism, as well as scenarios where sparse activation is disadvantageous at low batch sizes (in which case, shared experts can bear more of the load).

V. Block-Scaled FP4 Quantization: Fitting Large Parameters into Memory

Even with sparse activation, trillion-level parameters require full weights to reside in memory. New models commonly provide 4-bit weights directly on the distribution side. The mainstream choice is the FP4 format from the microscaling family: sharing a scaling factor per block (e.g., every 16 to 64 elements), with elements stored in 4-bit encoding. Compared to per-tensor scaling, block-level scaling preserves the dynamic range of outliers with significantly less precision loss. Compared to BF16, weight volume is reduced to about a quarter, allowing a single 8-GPU node to accommodate full weights with cache space to spare.

Engineering considerations include three points: first, the storage and dequantization access patterns of the scaling factors themselves require dedicated operator support; second, precision-sensitive operators like routing and normalization should remain in high precision; third, quantization should be viewed as a “distribution format” rather than a “capability promise,” and validation should still be based on self-tested accuracy.

VI. More Important Than Download Counts: Assessing New Architecture Maturity

Common “ecosystem tracking” reports in the community essentially answer one question: Can a structurally novel model be used in production? This can be abstracted into a reusable evaluation framework:

1. Distribution Signals ≠ Production Usage. Model hosting platform downloads, likes, and AI coding tool session counts reflect “reach and interest,” not online production traffic. Window fluctuations should not be directly interpreted as usage declines. Real usage signals come from the inference service side, which external observers cannot access. Therefore, any “relative ranking” conclusions should be labeled with uncertainty.
2. Framework Support Matrix Exposes Issues Earlier Than Benchmarks. New architectures mean new operators: chunked parallel kernels for linear attention, Delta state management, latent MoE routing, and hybrid layer KV/state caching. The merge state of these operators in inference frameworks directly determines whether stable deployment is possible. Note that open/draft state contributions (like memory offloading or prefix caching optimizations) are “directional attempts,” not released capabilities.
3. Value and Limits of Community Benchmarks. Community challenges with fixed tasks, fixed prompts, and traceable intervention counts are methodologically valuable. However, summary conclusions like “single success, cost X, time Y” serve only as clues when evaluator independence is insufficient. The correct approach is to integrate them into an in-house regression suite: long-document QA, repository-level code tasks, and multi-turn tool calling, each with set thresholds, rerun on version upgrades.
4. Deployment Checklist. First, validate throughput and latency by replaying real traffic in a canary environment. Run dedicated regressions for long-context scenarios (the performance of linear attention models on “needle in a haystack” tasks must be tested practically rather than relying on marketing claims). Design fallback paths. Monitoring and alerting must cover the memory watermarks of both the KV cache and the linear state cache. If using cloud hosting, general capabilities such as container platforms, GPU resource scheduling, logging, and monitoring can be reused to build the evaluation environment (for example, the general infrastructure categories provided by Baidu AI Cloud).

VII. Conclusion

Putting the puzzle together: Delta linear attention settles the “complexity bill” of attention; sparse full attention residual layers secure the “baseline of exact retrieval”; latent MoE lowers the “capacity and memory bill”; and block-scaled FP4 quantization paves the “last mile of deployment.” This complete chain of “efficiency—precision—capacity—deployment” is highly likely to become the universal paradigm for the next generation of open-weight foundation models. For engineering teams, beyond architecture papers, what truly dictates model selection remains a few pragmatic factors: whether operator support is merged, whether long-context regressions pass, and whether fallback paths are clear.