Hybrid Linear Attention Architecture Design Explained: From DeltaNet to an Efficient Chunkwise Algorithm

1. Background: Why Hybrid Attention Architectures Are Needed

Since the Transformer architecture was introduced, self-attention has remained the core mechanism for sequence modeling. However, standard self-attention has quadratic computational and memory costs with respect to sequence length, making training and inference on long sequences a significant bottleneck. Among the approaches explored to address this problem, linear attention has become an important direction for improving training and inference efficiency because it reformulates attention computation as a recurrent state update, reducing complexity to linear in sequence length.

That said, pure linear attention still lags behind full attention in expressive power and memory precision. A practical engineering approach is therefore to mix a small number of full-attention layers with a majority of linear-attention layers, preserving model quality while substantially reducing overall cost. The architecture discussed in this article follows this approach. It combines multi-head latent attention (MLA) with a gated linear attention module referred to here as KDA in a 3:1 ratio. KDA handles the bulk of the computation, while MLA retains the key capabilities of full attention. This article focuses on the design of KDA and its associated efficient chunkwise algorithm.

2. Overall Structure: Design Considerations Behind the 3:1 Ratio

The central question for a hybrid architecture is how much to mix and how to arrange the mixture. A 3:1 ratio means that linear-attention layers dominate while full-attention layers are inserted periodically. The main benefits of this design include:

• Efficiency gains: Most layers use linear complexity, greatly reducing computation and memory pressure on long sequences.
• Quality preservation: Full-attention layers retain precise token-level interaction, mitigating degradation on tasks that require retrieval or exact recall.
• Engineering manageability: A fixed ratio makes layer scheduling and parallel partitioning easier to handle in a training framework and easier to adjust according to available hardware.

It is important to note that the ratio is not arbitrary. It represents a trade-off on the efficiency-quality curve. Too many linear layers can weaken the model’s ability to capture long-range exact dependencies, while too few linear layers reduce the efficiency benefit. In practice, 3:1 has proven to be a reasonable balance.

3. KDA Structure: Combining Strengths of Linear Attention Variants

3.1 Core Design: DeltaNet with Gating

The KDA design can be seen as combining strengths from several earlier structures. Its main body follows the DeltaNet state update form and introduces a gating mechanism. To make the design easier to understand, here are the recurrent update rules for several representative structures, where S_t denotes the memory state matrix at time t and v_t k_t^T is the write term:

Pure Linear Attention:  S_t = S_{t-1} + v_t k_t^T
DeltaNet:               S_t = (I - β_t k_t k_t^T) S_{t-1} + β_t v_t k_t^T
Gated DeltaNet:         S_t = α_t (I - β_t k_t k_t^T) S_{t-1} + β_t v_t k_t^T
GLA / RWKV-6 style:     S_t = S_{t-1} Diag(α_t) + v_t k_t^T
RWKV-7 style:           S_t = S_{t-1} (Diag(α_t) + a_t b_t^T) + v_t k_t^T
KDA:                    S_t = (I - β_t k_t k_t^T) Diag(α_t) S_{t-1} + β_t v_t k_t^T

The formula comparison clearly shows KDA’s hybrid nature: it retains both the rank-one correction term (I - β_t k_t k_t^T) from DeltaNet and the diagonal scaling term Diag(α_t) from gated structures, combining them in a first-forget-then-correct order.

3.2 Two Key Parameters: The Roles of β and α

Understanding KDA requires clarifying what these two scalar parameters do:

β_t: Memory update intensity. In DeltaNet, the term (I - β_t k_t k_t^T) performs an erase-before-write operation. It attenuates old memory along the direction of the current key k_t and then writes the new value v_t. The parameter β_t controls the strength of this update. A larger β_t overwrites old information more thoroughly, while a smaller β_t makes the update more gradual. Conceptually, β_t adjusts how much weight is assigned to old versus new values in the memory matrix.

α_t: Forget rate. A structural limitation of pure DeltaNet is that fully refreshing the memory matrix requires at least d steps, where d is the state dimension. In other words, forgetting is passive. By introducing diagonal scaling with α_t, the model gains the ability to forget actively and quickly. For this reason, KDA-style structures are also referred to as DeltaNet with a forget gate. The finer the granularity of α_t, the more flexible the forget control.

3.3 Finer-Grained Gating: Parallels with RWKV-7-Style Designs

Compared with a basic gated DeltaNet, KDA refines the granularity of α_t further. This direction is similar in spirit to RWKV-7-style structures, which introduce an additional low-rank term a_t b_t^T to enhance the expressive power of the state transition. KDA instead refines the combination of diagonal gating and rank-one correction. Both approaches share the same goal: making memory writes and erasures less constrained by a single coarse-grained gate and enabling finer adjustment by channel or direction.

In the formula, KDA places Diag(αt) before the correction term, creating a decay-first path through Diag(α_t) S{t-1}, followed by directional erasure and rewriting via (I - β_t k_t k_t^T). This ordering allows forgetting and updating to be analyzed separately and also simplifies the derivation of the chunkwise parallel algorithm.

4. Hardware-Friendly Chunkwise Parallel Algorithm

The recurrent form of linear attention is mathematically elegant, but token-by-token sequential computation cannot fully utilize GPU parallelism. Another key technical aspect of KDA is its hardware-efficient chunkwise algorithm. The sequence is divided into chunks of length C. Matrix operations are parallelized within each chunk, while state recurrence is performed across chunks. This approach builds on publicly available research on chunkwise algorithms for DeltaNet.

Let K[t], U[t], and W[t] denote the key, value, and weight matrices within chunk [t]. Let Γ[t]^{i→C} denote the cumulative decay within the chunk, and let γ_[t]^C denote the chunk-level decay. The cross-chunk state update and intra-chunk output can be written as:

S[t+1] = Diag(γ_[t]^C) S[t]
         + (Γ_[t]^{i→C} ⊙ K_[t])^T (U_[t] − W_[t] S[t])      ∈ R^{d_k×d_v}
O_[t]  = (Γ_[t]^{i→C} ⊙ Q_[t]) S[t]
         + Tril( (Γ_[t]^{i→C} ⊙ Q_[t]) (K_[t] / Γ_[t]^{i→C})^T )
           (U_[t] − W_[t] S[t])                                ∈ R^{C×d_v}

The structure can be understood as follows:

• Cross-chunk recurrence: S[t+1] is obtained by applying chunk-level decay Diag(γ[t]^C) to the previous chunk state and then adding the incremental update from the current chunk. The term (U[t] − W[t] S_[t]) reflects DeltaNet’s correction idea: first compute the residual of the current chunk against the old state, then write it using the transposed keys.
• Intra-chunk parallelism: The output O_[t] consists of two parts. The first part reads the historical state using the queries. The second part uses a lower-triangular mask Tril to perform causal attention-style computation within the chunk, converting sequential dependencies inside the chunk into matrix multiplication.
• Numerical stability: The division form (K[t] / Γ[t]^{i→C}) requires overflow protection in implementation, typically through log-domain accumulation or segmented normalization.

The value of this chunkwise design is that it combines linear-attention recurrence with attention-style matrix parallelism. During training, it achieves high hardware utilization. During inference, it can proceed chunk by chunk with a constant-size state cache, addressing the efficiency needs of both scenarios.

5. Design Takeaways and Practical Considerations

Overall, this architecture provides several useful engineering lessons:

1. Mix rather than replace: Linear attention does not need to fully replace full attention. A proportional mix is currently a more reliable path for production use.
2. Gate granularity is a key design choice: The granularity of α_t directly affects forget control and therefore long-sequence modeling quality. It should be tuned based on the target task.
3. Co-design the algorithm with hardware: The value of the chunkwise algorithm is not only mathematical equivalence. It lies in mapping the recurrent structure to high-utilization matrix operations. Implementation details such as overflow prevention and chunk size selection also determine the actual benefit.
4. Use formulas as design documentation: Comparing recurrent formulas across different structures reveals the design trade-offs clearly. When evaluating similar architectures, it is useful to analyze the formulas first before moving to implementation and experiments.

For teams building long-context capabilities, this type of design, combining a mostly linear architecture with a small number of full-attention layers, fine-grained gating, and chunkwise parallelism, offers a practical reference for balancing efficiency and quality.