Why Attention and FFN Became the Main Battleground for Architecture Evolution

Each layer of a standard Transformer consists of two blocks: multi-head self-attention handles information exchange between tokens, while the feed-forward network (FFN) performs per-token nonlinear transformation. This structure is general enough, but it exposes clear cost problems in two scenarios:

1. Quadratic complexity and KV cache in attention. Self-attention computation grows quadratically with sequence length. During inference, the key (K) and value (V) matrices for every layer must be cached so that historical tokens remain accessible. The longer the context, the more GPU memory the KV cache consumes, which directly reduces batch size and concurrency. For inference serving, this often becomes a bottleneck earlier than compute itself—inference is typically a memory-bound workload, and every generated token requires moving the cache through memory again.
2. Dense parameters in the FFN. A large share of model capability comes from FFN parameters, but a dense FFN means every token pays for all parameters, regardless of whether that token actually needs that specific knowledge.

As a result, architecture evolution in recent open-source LLMs has moved along three complementary directions: using linear attention to reduce attention state cost to constant scale; keeping a small number of full-attention layers as residual compensation to preserve precise retrieval; and using sparse mixture-of-experts (MoE) to expand parameter capacity without increasing per-token computation. The following sections break down each direction and show how they fit together into a complete efficient long-context architecture.

Delta-Rule Linear Attention: Rewriting Attention as Readable and Writable Memory

From Softmax to Linear Attention

Softmax attention can be viewed as a weighted query over all historical K/V pairs: y_t = Σ_s softmax(q_t·k_s)·v_s. Its precision comes from pairwise comparison, at the cost of retaining the entire history. Linear attention takes a different approach to memory: it maintains a fixed-size state matrix S, accumulates the outer product k·vᵀ recursively token by token, and produces the output with y_t = q_tᵀS. The state size is independent of sequence length, and per-step inference cost becomes constant. This is a fundamental reduction in complexity.

Why Naive Accumulation Fails

Naive accumulation is essentially an associative memory that only adds and never deletes: values associated with similar keys keep stacking up and interfering with each other, while old information can never be overwritten. Once the sequence becomes long, the state saturates quickly, and the model cannot distinguish which memories are new and which should be replaced.

The Delta Rule: Writing with Error Correction

The delta rule borrows from classical adaptive filtering and changes the state update from accumulation to a read-before-write operation that first erases conflicting information:

S_t = (I − β_t·k_t·k_tᵀ)·S_{t−1} + β_t·k_t·v_tᵀ

Intuitively, this is an upsert operation for key-value storage: β_t controls the write strength, and k_tᵀS_{t−1} reads the old value currently stored under that key. The model erases the conflicting part before writing the new value. Similar keys no longer accumulate without limit; later writes overwrite earlier ones, making the memory behavior closer to a real cache.

# q, k ∈ R^{d_k}, v ∈ R^{d_v}, state S ∈ R^{d_k × d_v}
S = np.zeros((d_k, d_v))
for t in range(T):
    beta  = sigmoid(w_beta @ x[t])          # write strength
    gamma = exp(-softplus(w_g @ x[t]))      # forget gate
    S = gamma * S                           # (1) decay historical memory
    old = k[t] @ S                          # (2) read old value for this key
    S -= beta * np.outer(k[t], old)         # (3) erase conflicting old value
    S += beta * np.outer(k[t], v[t])        # (4) write new value
    y[t] = q[t] @ S                         # (5) query output

Gating and Fine-Grained Decay

Adding a data-dependent forget gate on top of the delta rule—optionally fine-grained down to the channel level—lets the model learn which memory dimensions to keep and which to decay faster. This mitigates state saturation on long sequences and is a key reason these structures perform more reliably on language modeling than earlier linear attention variants.

Dual Forms for Training and Inference

The recurrent form is naturally suited for inference, with constant cost per step. For training, it can be rewritten as a block-parallel matrix form, turning attention over the whole sequence back into large matrix multiplications that use hardware efficiently. This dual-form property—recurrent for inference, block-parallel for training—is the engineering prerequisite that makes these structures practical.

Acknowledging the Shortcomings

A fixed-size state means lossy information: the model cannot losslessly retrieve any arbitrarily distant individual token. For tasks like exact copying or locating details in long documents, purely linear attention is clearly weaker than full attention. This weakness is exactly what the next section addresses.

Attention Residuals: Using a Small Amount of Full Attention for Precise Retrieval

Two Meanings

In engineering contexts, “attention residual” usually covers two complementary designs:

1. Structural residuals. Residual connections inside and around attention blocks, pre-normalization, QK normalization, and residual scaling are stability techniques that keep deep stacks trainable and prevent information flow collapse. These are widely adopted general practices.
2. Hybrid compensation. A small number of full-attention layers are interleaved among many linear-attention layers, serving as a residual channel for precise retrieval. Most token interactions are handled by linear attention, while full attention covers cases that require exact per-token access.

Mixing Ratio and KV Cache Savings

A common pattern is to insert one full-attention layer every 3–7 linear-attention layers, with the exact ratio determined by ablation studies. The benefit can be estimated directly: as a rough rule, the proportion of full-attention layers largely determines the proportion of the KV cache that grows with sequence length. For example, if only one layer in four uses full attention, the length-dependent cache overhead is roughly a quarter of a pure-attention model. The remaining linear layers only maintain fixed-size states independent of context length.

def build_layers(n_layers, full_every=4):
    layers = []
    for i in range(n_layers):
        if (i + 1) % full_every == 0:
            layers.append(FullAttention())        # a few layers: precise retrieval
        else:
            layers.append(DeltaLinearAttention()) # most layers: constant state
        layers.append(LatentMoE())                # sparse FFN
    return layers

Layer Position Is Also a Hyperparameter

Where to place full-attention layers matters too. Even spacing is a conservative and common strategy. Some implementations concentrate full attention in the higher layers near the output, serving the generation stage that needs to pull details from long context. Like the ratio, the position should be determined through retrieval-oriented evaluation.

Latent Mixture-of-Experts: Sparse Capacity with Latent Compression

The MoE Baseline

Replacing a dense FFN with a router plus N experts—where each token activates only the top-k experts—scales parameter capacity while keeping per-token computation roughly unchanged. There are three costs: memory usage rises because all experts must reside on device, routing introduces training instability, and inference adds scheduling and communication complexity.

Fine-Grained Experts and Shared Experts

Two mainstream improvements have emerged in recent years. One is to split experts more finely and activate more small experts per token, which significantly improves combinatorial expressiveness. The other is to add shared experts that every token passes through, carrying common knowledge while routed experts focus on long-tail and specialized patterns. Together they deliver better results at the same activation budget.

What “Latent” Means Here

The core idea of latent compression shares the same origin as low-rank caching on the attention side: project a high-dimensional representation into a low-dimensional latent space, perform scoring or storage in that space, then project back. Applied to MoE, a typical approach is to perform routing scores on a down-projected latent representation, reducing routing computation and communication overhead. Applied to attention, it means compressing K/V into low-rank latent vector caches, substantially reducing memory usage. Both are fundamentally the same idea: approximate high-dimensional information with low-dimensional latent variables, and trade the saved overhead for capacity and context.

class LatentMoE(nn.Module):
    def forward(self, x):                     # x: [batch, seq, d]
        h = self.down_proj(x)                 # project down to latent space
        logits = self.gate(h)                 # score in latent space, lighter routing
        w, idx = topk(softmax(logits), k=8)   # sparse activation
        out = self.shared_expert(x)           # shared expert: all tokens pass through
        for w_i, e_i in zip(w, idx):
            out = out + w_i * self.experts[e_i](x)
        return out

Training Stability and Inference Engineering

Known MoE difficulties need to be addressed directly. Load imbalance can cause expert collapse and is usually constrained with auxiliary balancing losses. Routing instability affects convergence, so smoother gating and warm-up strategies are common. On the inference side, expert parallelism, cross-device communication, and memory fragmentation are the main challenges. Larger batches yield more amortized benefit from sparse activation, so these architectures are most beneficial in high-concurrency serving scenarios.

How the Three Components Fit Together

Combining the three directions produces the typical skeleton of recent efficient long-context models:

• Attention backbone. Delta-rule linear attention as the main component, with state cost independent of sequence length.
• Retrieval fallback. A small number of full-attention layers inserted at fixed intervals, forming a residual channel for precise retrieval, while retaining residual connections, normalization, and other stability designs.
• FFN layer. Replaced with a fine-grained latent mixture-of-experts with shared experts, offering large capacity, low per-token compute, and reduced cache and communication overhead.

The benefits of the three components are orthogonal: linear attention solves state growth with sequence length, hybrid attention preserves precise retrieval, and latent MoE decouples capacity from compute. Together, they make long context, large parameter counts, and acceptable inference cost achievable at the same time.

How to Verify and Evaluate These Models

For developers deciding whether to adopt a model with a new architecture, a verifiable process is more reliable than relying on isolated claims:

1. Check openness. Whether the weights are public, whether the license allows commercial use, and whether the technical report provides ablation studies rather than conclusions alone.
2. Check ecosystem support. Whether mainstream inference engines have adapted the core operators, such as block-wise linear attention kernels and MoE routing with expert parallelism. Early in a new architecture’s lifecycle, framework support often lags behind weight releases, which directly affects production readiness.
3. Check measurement methodology. Use business samples, fix the input, and run repeated tests to compare throughput, time-to-first-token, long-context recall, and memory usage. Community reports like “the only model that succeeded on a task, along with time and cost” should be treated as clues rather than conclusions when the evaluation methodology lacks independence.
4. Distinguish distribution popularity from real usage. Downloads, likes, and leaderboard rankings reflect reach and attention, not production traffic. Production decisions should be based on first-hand metrics from small-scale canary testing.

Teams that do not want to build their own inference stack can also choose managed model services on cloud platforms, leaving operator adaptation and scaling to the provider while focusing on evaluation and business integration.

Conclusion

Linear attention, attention residuals, and latent mixture-of-experts correspond to three goals: making state cheaper, making retrieval more reliable, and making capacity more elastic. Understanding their motivations and trade-offs matters more than memorizing any specific model name. This design language is becoming the shared foundation of a new generation of long-context models, and it will continue to shape inference serving choices and optimization directions in the coming years.