Attention Residuals: Rethinking Transformer Depth Connections
This article examines attention residuals, a retrieval-based alternative to fixed residual connections in Transformers. It explains the PreNorm dilution problem, the time-depth duality behind the idea, the core attention-residual mechanism, and the engineering challenges around compute, memory, training stability, inference compatibility, and orthogonality with existing optimizations. It also discusses a broader design method: revisiting long-standing architectural components through duality.
Attention Residuals: Rethinking Transformer Depth Connections
1. The Connection That Was Overlooked for a Decade
Over the past few years, the internal structure of the Transformer has been taken apart almost piece by piece. Attention has evolved from standard multi-head attention to grouped-query attention, multi-query attention, and even linear attention. Feed-forward networks have moved from dense MLPs to sparse mixture-of-experts. Positional encoding has shifted from absolute encoding to rotary position embeddings. Normalization has moved from LayerNorm to RMSNorm, along with layouts such as PreNorm and PostNorm.
The one component connecting all of these pieces—the residual connection—has remained almost unchanged since the ResNet era. It simply adds the previous layer’s output to the current sublayer’s output:
h_l = h_{l-1} + f_l(h_{l-1})
This “fixed, weight-free, direct addition” structure has remained stable for a long time because it is simple, numerically friendly, and naturally mitigates vanishing gradients. But “simple and effective” does not mean “optimal under current conditions.” As model depth scales from a dozen layers to dozens or even hundreds, and as the training objective shifts from classification to autoregressive language modeling over hundreds of billions of tokens, this decade-old design begins to expose new structural tension.
2. The Dilution Problem Under PreNorm
Modern large language models almost universally adopt PreNorm: normalize before entering a sublayer, then add the sublayer output back to the main branch.
h_l = h_{l-1} + f_l(Norm(h_{l-1}))
Expanding this layer by layer gives:
h_L = h_0 + Σ_{i=1..L} f_i(Norm(h_{i-1}))
This expression reveals a fact that is easy to overlook: the backbone hidden state is essentially an equal-weight accumulation of all historical layer outputs. The contribution of h_0 is constant, and the contribution of each newly added f_i is also constant. As the number of layers increases, the magnitude of the hidden state grows statistically, while the share of newly injected information from a single layer in the total continues to decline.
An analogy: if someone deposits a fixed amount into the same account every day, the larger the account balance, the smaller the proportion that day’s deposit represents. Along the depth direction, early-layer outputs are “drowned out” by later equal-weight accumulation, and a single layer struggles to have a decisive influence on the final representation. This is what industry discussions often refer to as PreNorm dilution.
Its consequences are not limited to magnitude expansion at the numerical level. More fundamentally, it affects the model’s expressive capacity: local features and syntactic cues captured by shallow layers are diluted by a large number of unrelated accumulation terms as they propagate to deeper layers. When deep layers want to reuse early information, they lack a path for “selective lookback.”
3. Time-Depth Duality
To understand the solution to this problem, one can look for an analogy in history.
How did the Transformer beat the RNN? The core lay in how information is aggregated. The RNN indiscriminately compresses all history over the time dimension into a fixed-size hidden state, so long-range dependencies inevitably decay. The Transformer uses attention to achieve “on-demand retrieval”—the current position actively selects which historical positions to attend to via a query, rather than receiving information passively.
Move this duality to the depth dimension, and a striking symmetry appears: a standard residual connection is essentially compressing information from all previous layers indiscriminately into a gradually expanding hidden state. Since the time dimension has already been revolutionized by attention, the “fixed residual” in the depth dimension likewise has room to be transformed by attention mechanisms. This is the core insight of the so-called time-depth duality—it is not a minor fix involving a different normalization position or stacking a few more layers, but a reexamination of the basic assumption of how information flows between layers.
4. Core Mechanism of Attention Residuals
The idea behind attention residuals can be summarized in one sentence: instead of letting the current layer passively receive an equal-weight sum of all historical layer outputs, use a query to actively retrieve previous layer outputs.
Specifically, a standard residual constructs the input to layer l as “previous layer output + current sublayer output.” An attention residual instead treats each previous layer’s output as a retrievable “memory entry.” The current layer generates a query and performs attention-weighted aggregation over the keys/values of all historical layers to obtain the input representation for that layer. A conceptual illustration is as follows:
# Standard residual (PreNorm)def forward(h, layers):for layer in layers:h = h + layer(norm(h))return h# Attention residual (conceptual illustration, not a concrete implementation)def forward(h, layers):memory = [h] # Historical layer outputs as retrievable memoryfor layer in layers:q = project_q(norm(h)) # Current hidden state generates a querykv = [project_kv(m) for m in memory]h = attend(q, kv) # On-demand weighted aggregation replaces fixed additionh = h + layer(norm(h)) # Sublayer computation is retainedmemory.append(h)return h
Compared with a standard residual, this design brings three key changes:
- Weights change from fixed to learnable. Which historical layers are more important to the current layer is no longer determined by the prior of “distance,” but is dynamically allocated through data-driven attention weights.
- Aggregation changes from accumulation to retrieval. Historical information is no longer passively added to the main branch; it serves as candidate memory to be queried. Deep layers can precisely look back at key features from early layers and avoid being diluted by a large number of irrelevant terms.
- The depth direction gains the ability to model long-range dependencies. This is functionally highly isomorphic to attention over the time dimension: sparse, selective, and content-related, rather than dense, indiscriminate, and position-related.
5. Engineering Barriers to Deployment
From concept to usability, attention residuals need to solve several practical engineering problems. This is also an important observation point for judging whether a new architecture can truly be deployed.
First, compute and memory overhead. The sequence length of attention over the time dimension is usually thousands to tens of thousands of tokens, while the number of layers in the depth dimension is between dozens and hundreds. If full attention were performed for every layer over all previous layers, the combinatorial overhead would rise significantly. Common engineering responses in the industry include block aggregation, low-rank projection, cross-layer shared key/value, and window- or group-based sparse retrieval. The goal is to bring the cost of depth attention to the same order of magnitude as a standard residual.
Second, training stability. One hidden value of residual connections is that they provide an identity mapping path, allowing gradients to propagate stably. Once attention residuals replace this path, initialization, normalization, and scaling strategies must be redesigned to ensure that values do not diverge and gradients do not vanish during deep propagation. This is often the first threshold a new architecture must cross to move from a paper to production.
Third, inference-side compatibility. A standard residual can be computed layer by layer sequentially during inference, with low memory usage. Attention residuals need to retain each layer’s output as “memory,” which imposes new requirements on inference caching and parallel strategies. Whether it can be compatible with existing KV Cache, pipeline parallelism, and tensor parallelism mechanisms directly determines deployment cost.
Fourth, orthogonality with existing optimizations. This mechanism is not mutually exclusive with grouped-query attention, sparse experts, rotary position embeddings, RMSNorm, and similar techniques. In theory, they can be stacked. But in a composite architecture, the numerical interactions among components need systematic validation to avoid local optima undermining global stability.
6. Broader Lesson: Reexamining Old Components Through Duality
The value of attention residuals lies not only in proposing a new connection method, but also in demonstrating a structural way of thinking: when a foundational component in a field has long maintained a certain form, it is often worth asking—“Why was it designed this way in the first place? Have today’s constraints changed?”
Looking back at the history of large-model architecture evolution, many breakthroughs have come from this kind of “symmetry transfer”: moving sequence-modeling ideas from recurrence to attention, introducing mixture-of-experts from shallow layers to deep layers, and bringing the local inductive bias of convolution into vision Transformers. Time-depth duality is just a new facet of this kind of thinking. Along this line, one can also ask: Is there room to make normalization retrievable? Should routing between layers change from static to dynamic? Can information aggregation in feed-forward networks along the depth direction also introduce sparse retrieval?
For teams working on model R&D, the significance of such exploration is that it provides alternative paths for “non-incremental improvements.” When the marginal returns of pure parameter scaling and data scaling decline, reexamining foundational components can often open new efficiency space. To achieve this, what is needed is not just reproducing a paper, but building a complete evaluation pipeline from principle analysis, small-scale ablation, and numerical stability validation to large-scale training—using engineering to turn “conceptually feasible” into “production-ready.”
Conclusion: The fact that residual connections have remained unchanged for a decade does not mean they are already optimal; it only means they are sufficiently usable and costly to change. Attention residuals transform “indiscriminate accumulation” in the depth dimension into “on-demand retrieval,” which in spirit follows the same lineage as attention replacing recurrence. Whether it ultimately becomes part of the next mainstream architecture depends on whether engineering can reduce the extra overhead to an acceptable range and make training stability solid. But regardless of the outcome, this approach of reexamining foundational components through duality has already provided a direction for architecture innovation that deserves serious consideration.
