Next-Gen LLM Architectures: Decaying Linear Attention and Latent MoE

I. Background: Three Drivers for Architectural Redesign

Over the past few years, the backbone architecture of mainstream Large Language Models (LLMs) has largely remained a combination of Softmax full attention and dense Feed-Forward Networks (FFNs). However, as context lengths expand from thousands of tokens to hundreds of thousands or even millions, three structural bottlenecks of this combination have become increasingly prominent:

1. Linear KV cache expansion with context: Softmax attention must retain a key-value cache for every historical token during inference. Memory consumption scales linearly with sequence length, meaning long-context throughput is bottlenecked by cache bandwidth rather than compute.
2. Low parameter efficiency in dense FFNs: Every token must pass through the entire set of feed-forward parameters. Since the “knowledge subset” required by different tokens varies greatly, a significant amount of computation is inherently redundant.
3. Training stability challenges in ultra-deep networks: As depth, width, and the number of experts increase simultaneously, managing the scale of the residual stream, normalization placement, and routing balance significantly impacts convergence quality.

Recent community discussions around next-generation model architectures provide three distinct responses to these bottlenecks: restructuring the cost of attention using Decaying Linear Attention, stabilizing deep network training and inference via Attention-Residual Coordination, and improving parameter and memory access efficiency in sparse models using Latent Compressed Mixture-of-Experts (MoE). Let’s break down each component.

II. Decaying Linear Attention: Packing History into a Fixed-Size State

2.1 The Cost Structure of Softmax Attention

Standard attention computes the similarity for every pair of queries and keys, followed by a Softmax weighted sum. During training, the computational complexity is O(L²) (where L is the sequence length), and during inference, the KV cache scales as O(L). This quadratic curve is negligible for short contexts, but once L reaches hundreds of thousands, prefill latency and cache memory become hard constraints.

2.2 Core Idea of Linear Attention

Linear attention changes the computation order of the “weighted sum.” Instead of rescanning all history for each query, it incrementally accumulates history into a fixed-size matrix state S, with shape (d_k, d_v), independent of sequence length. During inference, reading a new token requires only one state update and one state read. The per-step cost is O(d_k·d_v), completely decoupling memory footprint from context length.

2.3 Incremental Rules: From “Accumulation” to “Overwriting”

Naive linear attention only performs outer product accumulation. The problem is that once old information is written, it cannot be overwritten, causing the state to “saturate” quickly and leading to interference between old and new keys. A widely adopted improvement borrows incremental rules from online learning: when a new observation arrives, first subtract the projection of the old value in the direction of the key from the state, then write the new value. The step-by-step inference process is illustrated in the following code:

# Step-by-step inference perspective: state size is independent of sequence length
S = torch.zeros(d_k, d_v)                      # Matrix state maintained across time steps
for t in range(seq_len):
    q, k, v = Q[t], K[t], V[t]
    decay = torch.exp(-softplus(log_decay[t])) # Data-dependent decay coefficient ∈ (0,1)
    beta  = torch.sigmoid(write_gate[t])       # Write strength ∈ (0,1)
    # Erase old value projection first, then write new value residual
    S = decay * S + beta * torch.outer(k, v - S.T @ k)
    y[t] = q @ S

2.4 Decay Gating: Fine-Grained Forgetting Mechanism

While the incremental rule solves “writability,” decay gating addresses “what should be forgotten.” It multiplies the state by a decay coefficient in the range (0,1) before the state update. This coefficient is determined by the current input data and can be finely controlled—at the per-key, per-head, or even per-channel granularity. This shares its origins with gating mechanisms in recurrent networks: allowing the network to learn which historical contexts need long-term retention and which can be quickly forgotten, thereby packing more valuable information into a limited state capacity.

2.5 Engineering Considerations and Trade-offs

Implementing this mechanism involves three key points:

• Parallel training requires chunked scan kernels: The state is defined recursively, and naive implementations cannot be parallelized. Engineering efforts rely on chunked parallel algorithms and custom kernels, which is the fundamental reason why adapting inference frameworks to new architectures takes a long time.
• Usually mixed and stacked with full attention: Pure linear attention underperforms Softmax attention in precise retrieval tasks (e.g., needle-in-a-haystack). A common industry practice is to use linear attention in the vast majority of layers for local and mid-range dependencies, inserting a full attention layer every few layers for precise retrieval, balancing cache efficiency with recall capability.
• Benefits are realized in long-context inference: A constant state means that memory footprint and bandwidth no longer grow with conversation turns during decoding, a structural advantage for multi-turn Agent and long-document processing workloads.

III. Attention-Residual Coordination: Traffic Management for Deep Networks

3.1 The Residual Stream Perspective

The residual stream can be understood as an “information highway” in the backbone network, where each sublayer simply reads from and writes to it. When a network reaches dozens of layers, managing the scale of writes becomes critical. If early layers write too strongly to the residual stream, it compresses the expressive space of subsequent layers, easily leading to loss spikes during training.

3.2 Common Stabilization Components

Mature industry practices include three categories. First, the choice of normalization placement—from classic Pre-Norm and Post-Norm to sandwich normalization (placing normalization at both the input and output of a sublayer)—each with its trade-offs. Second, applying key-value side normalization to the attention scores themselves to suppress logit scale drift. Third, multiplying an overall scaling factor before the attention output is written back to the residual stream, suppressing the contribution of the attention branch in the early training phases and allowing the backbone to establish stable representations first.

3.3 Gated Injection

A further step is to add a learnable gating vector to each write, allowing the network to decide how much content to write to the residual stream for each layer and channel. An illustration is as follows:

class Block(nn.Module):
    def forward(self, x):
        h = self.norm_attn(x)
        a = self.attn(h) * self.res_scale     # Output scaling: suppress strong early writes
        x = x + self.gate_a(a)                # Write back to residual stream after gating
        h = self.norm_moe(x)
        x = x + self.gate_m(self.moe(h))      # Expert branch also writes back via gating
        return x

This design is also inference-friendly: once normalization and gating placements are fixed, operator fusion becomes easier, reducing memory access round-trips during long-context decoding. Numerically, it is also less prone to amplifying errors during low-precision inference (e.g., 8-bit or 4-bit weights).

IV. Latent Compressed MoE: Making Sparse Models More Efficient

4.1 The Baseline and Bottlenecks of MoE

Mixture-of-Experts (MoE) achieves “more parameters, less computation” by using a router to activate only a few FFN experts per token. However, it has three recognized bottlenecks: the large total number of experts creates a high memory threshold; routing introduces significant cross-card communication overhead under tensor parallelism; and load imbalance can turn some experts into hotspots.

4.2 Fine-Grained and Shared Experts

There are two common optimization approaches in the industry. First, splitting large experts into more small experts and increasing the top-k count, giving the router finer-grained combinatorial freedom. Second, setting up shared experts that all tokens must pass through to carry general knowledge, allowing routed experts to focus on differentiated knowledge and reduce redundant work.

4.3 Latent Compression: Low-Rank Bottleneck into Experts

The core idea of “latentization” is consistent with latent compressed attention: compress first, then compute. A low-rank bottleneck is introduced into the expert structure, projecting the input down to a latent space much smaller than the model width before expanding it to a wide intermediate layer:

class LatentFFN(nn.Module):
    """Low-rank bottleneck: compress to latent space first, then expand to wide intermediate layer"""
    def __init__(self, d_model, d_latent, d_ff):
        self.down = nn.Linear(d_model, d_latent)
        self.up   = nn.Linear(d_latent, d_ff)
        self.out  = nn.Linear(d_ff, d_model)
    def forward(self, x):
        return self.out(act(self.up(self.down(x))))

The benefits are twofold: the effective parameter count and memory access footprint of experts decrease, and the cross-card transmission involves low-dimensional latent representations rather than full-width activations, alleviating communication pressure. Combined with standard routing scoring and sparse activation logic:

def moe_forward(x, gate, experts, shared, top_k=2):
    y = shared(x)                              # Shared expert: mandatory for all tokens
    w, idx = topk_softmax(gate(x), top_k)      # Activate only top-k routed experts
    for j in range(top_k):
        y = y + w[..., j:j+1] * experts[idx[..., j]](x)
    return y

4.4 Stability Companion Mechanisms

Sparse models typically require auxiliary mechanisms like load balancing loss, routing jitter control, and expert-level dropout to prevent routing collapse to a few experts during early training. These are industry consensus. When evaluating a new MoE architecture, the focus should be on its balancing strategy and expert utilization curves, rather than just total parameter count.

V. The Synergy of the Three Components

Component	Problem Solved	Trade-offs
Decaying Linear Attention	Decouples KV cache from context length	Requires custom kernels; precise retrieval relies on a few full attention layers
Attention-Residual Coordination	Deep network training stability and low-precision robustness	Normalization/gating adds a few small serial operators
Latent Compressed MoE	Expert memory footprint and routing communication pressure	Low-rank bottleneck may limit single expert capacity; balancing strategy is more complex

Together, these three threads outline a clear design philosophy: localize “precise but expensive” components, and make “high-frequency general” components cheaper. Full attention appears only in a few layers, while linear attention handles daily throughput; sparse experts handle knowledge capacity, while low-rank compression handles memory access efficiency; residual management ensures all of this doesn’t collapse under deep stacking.

VI. Engineering Implementation and Ecosystem Evaluation

An advanced architecture does not equate to usability. For teams planning to adopt new architectures, it is recommended to evaluate ecosystem maturity using the following methodology to avoid being misled by surface-level metrics.

6.1 Weight Distribution and Quantization Formats

Current large weighted models are generally distributed in low-bit microscaling floating-point (4-bit level) compressed tensor formats, and download volumes in the millions are not uncommon. However, be clear-headed: weight distribution reach is merely a signal of “availability,” not production readiness. Sync volumes on mirror sites and increasing star counts are distribution-side metrics.

6.2 The General Path of Inference Framework Adaptation

New operators (chunked scan, gated decay, latent routing) require kernel-level support for actual throughput. Observation points include: whether the support matrix for various inference backends is still empty, whether relevant support code has been merged into the main branch, and whether the official sources provide directly runnable inference examples. Unmerged community attempts are directional clues, not released capabilities.

6.3 Three Pitfalls in Metric Interpretation

• The token consumption displayed on aggregate platforms is a rolling window value. A short-term decline might be due to statistical caliber or window drift, and cannot be directly interpreted as a drop in actual usage.
• Rankings and share percentages have small sample sizes, amplifying fluctuations; only long-term trends should be considered.
• For community benchmarks to be credible, they must meet conditions like fixed environments, frozen prompts, traceable intervention counts, and independent evaluators. Isolated conclusions of “unique success, extremely low cost, extremely short time” can only be treated as clues, not evidence.

6.4 Implementation Checklist

• Confirm the target inference backend’s support status for mixed attention and latent MoE operators.
• Reproduce benchmarks in a controlled environment rather than citing second-hand summaries.
• Conduct a quality-cost-latency triangle evaluation using your own business workloads before deciding on a gradual rollout strategy.
• If leaning towards building an in-house inference cluster, you must bear the kernel adaptation and memory planning yourself; if you wish to reduce engineering burden, you can choose a cloud-managed path—for example, using general model services and compute resources provided by cloud platforms like Baidu AI Cloud for rapid validation, allowing you to focus your energy on business-side evaluation.

VII. Conclusion

Decaying linear attention, attention-residual coordination, and latent compressed MoE are essentially systematic responses to the triple pressure of “long context, massive parameters, and deep networks.” They do not change the fundamental paradigm of the Transformer but perform delicate surgery on the cost structure. For engineers, understanding “what each component saves and what it costs” is more important than remembering the terminology; for teams, whether architectural dividends can be realized ultimately depends on the degree of inference ecosystem adaptation and how well it matches their own workload characteristics. It is advisable to start with small-scale controlled evaluations and let data, rather than community hype, drive migration decisions.