Are Residual Connections Underestimated? Exploring Attention Residuals in LLM Architectures

1. Residual Connections: The Overlooked “Default Configuration”

Since the inception of the Transformer, the industry has modified almost every key model component. Attention mechanisms have evolved from standard Multi-Head Attention to various efficient variants; Feed-Forward Networks (FFNs) have developed from simple fully connected layers to Mixture of Experts (MoE) architectures; position encoding has progressed from absolute and relative encoding to Rotary Position Embedding (RoPE); and normalization layers have iterated from LayerNorm to RMSNorm. However, the residual connections linking these components have long retained their original design.

Since the introduction of ResNet, the core formula for residual connections has remained almost unchanged:

h_l = h_{l-1} + f_l(h_{l-1})

The meaning of this formula is straightforward: the output of the current layer equals the output of the previous layer plus the increment computed by the current layer. This “identity mapping + increment” design is widely considered the key to solving the vanishing gradient problem in deep networks, making it a standard configuration in almost all modern deep neural networks.

But a critical question remains: is this fixed-weight accumulation approach truly optimal? When model depth scales from dozens to hundreds of layers, does this simple addition strategy introduce new bottlenecks?

2. The Hidden Bottleneck of Fixed Residuals

To understand the potential issues with fixed residual connections, we must first review the architectural pattern adopted by mainstream Large Language Models (LLMs). Most modern LLMs use the PreNorm architecture, which applies normalization before entering a sublayer and then adds the input to the output via a residual connection.

Under this architecture, the computation at each layer can be described as:

h_l = h_{l-1} + f_l(Norm(h_{l-1}))

Here, f_l represents the current layer’s computation function (either an attention or FFN layer), and Norm represents the normalization operation. The essence of this formula is that each layer “accumulates” new information into the existing hidden state, with a fixed accumulation weight always equal to 1.

As network depth increases, this fixed-weight accumulation introduces two interconnected problems:

The first problem is continuous magnitude growth. Because each layer adds a new vector to the hidden state, and there is no guarantee that these new vectors are orthogonal to the existing ones, the norm of the hidden state tends to grow continuously as layers increase. Although the PreNorm structure normalizes the input before each layer begins—ensuring the input to the computation function f_l remains stable—it cannot prevent the overall magnitude expansion caused by residual accumulation. As depth increases, the total energy of the hidden state keeps climbing, meaning subsequent layers operate in an increasingly “inflated” representation space.

The second, more critical problem is information dilution. Suppose layer l generates a new information increment Δ_l = f_l(Norm(h_{l-1})). The proportion of this increment within the final hidden state will continuously decrease as subsequent layers are added. In other words, important features extracted by earlier layers still exist in the hidden state, but their “signal strength” is continually diluted relative to the expanding overall representation. This phenomenon can be termed “fixed residual dilution”—the earlier the layer, the more easily its contributions are buried beneath the continuously accumulated information of later layers.

This bears a striking resemblance to the dilemma faced by Recurrent Neural Networks (RNNs) across the time dimension. Traditional RNNs compress all historical information into a fixed-size hidden state. As time steps grow, early information is inevitably diluted and forgotten. Transformers solved this issue in the time dimension through attention mechanisms, allowing the model to “retrieve on demand” from any position in history at any given moment, rather than indiscriminately compressing all information together.

However, in the depth dimension, the behavior of standard residual connections is fundamentally similar to RNN’s information compression: it indiscriminately accumulates information from all preceding layers into an increasingly inflated representation. It lacks any mechanism to distinguish the relative importance of information from different layers, nor can it “retrieve on demand” key features from early layers during deep stages.

3. Core Insight: The Duality of Time and Depth

The above analysis reveals a profound architectural analogy: the way information propagates across the time dimension and the depth dimension faces isomorphic challenges.

Looking back at how Transformers replaced RNNs as the mainstream sequence modeling solution, their core advantage was substituting fixed-size hidden state compression with attention mechanisms. In an RNN, the output at any time step depends on a constantly overwritten hidden state, making information decay structural. The Transformer’s attention mechanism, through global key-value retrieval, allows each output position to directly access any position in the input sequence, fundamentally circumventing long-distance information decay.

So, can the same logic be applied to the depth dimension?

The hidden state h_l in standard residual connections essentially plays a role similar to an RNN’s hidden state—it is a “continuously accumulated” representation carrying all historical information from layer 0 to layer l. The fixed-weight accumulation method deprives this representation of the ability to “selectively access” information from different layers.

Based on this observation, a natural question arises: if attention mechanisms can achieve “on-demand retrieval” in the time dimension, can a similar mechanism be introduced in the depth dimension? This would allow each layer to dynamically and on-demand access information from any preceding layer, rather than passively receiving a fixed accumulation result.

This is the core motivation behind the concept of “Attention Residuals.” It is not a modification to the attention mechanism itself, nor an adjustment to FFNs or normalization layers, but rather a re-examination and upgrade of the residual connection infrastructure.

4. Design Philosophy of Attention Residuals

With the motivation clear, the design direction for Attention Residuals becomes apparent. In standard residual connections, the current layer’s hidden state is simply the sum of the previous hidden state and the current increment. This “blind addition” means that each layer’s “access method” to preceding information is fixed—it can only see the accumulated result of the previous layer and cannot skip intermediate layers to directly access earlier information.

The core idea of Attention Residuals is to replace fixed-weight accumulation with a learnable, attention-based dynamic integration.

Specifically, when computing the output of layer l, instead of merely adding h_{l-1} and the current increment, a cross-depth attention operation is introduced. The input to this operation might include the outputs of multiple preceding layers (or even all preceding layers). It dynamically calculates the weight of each layer’s information via an attention mechanism and then performs a weighted aggregation to form the new representation of the current layer.

This concept can be expanded from two dimensions:

Key-Value Retrieval in the Depth Dimension. Treating each layer’s output as a “key-value pair,” the current layer uses a query vector to perform attention retrieval over the outputs of all preceding layers, selectively integrating the historical information most relevant to the current computation. This transforms the deep network from a passive accumulation process into an active retrieval and dynamic integration process.

Dynamization of Residual Weights. In standard residual connections, the weight of preceding information entering the current layer is always 1. Attention Residuals turn this weight into a dynamically computed result, allowing the model to autonomously decide “how much information to retrieve from each preceding layer and which information to retrieve” based on the current layer’s needs.

It is important to note that this design does not entirely discard residual connections. The gradient highway provided by residual connections remains a crucial guarantee for training deep networks. A more probable form for Attention Residuals is to retain the basic residual path while adding a learnable depth attention module, upgrading information integration from “fixed accumulation” to a combination of “dynamic retrieval and fixed accumulation.”

5. Implementation Considerations

From an engineering perspective, the design of Attention Residuals introduces factors that require careful trade-offs.

Computational Overhead. If every layer needs to perform attention computation over all preceding layers, the total computational complexity will grow from linear to quadratic. In practical implementations, strategies such as windowing (focusing only on a recent few layers), sharing query vectors across layers, or using linearized attention approximations may be required to reduce computational costs.

Memory Footprint. Depth attention requires storing the outputs of preceding layers as a key-value cache. For large models with hundreds of layers, this means a significant memory increase. Whether to store the outputs of all layers or only store dimensionality-reduced summary representations is a design problem that needs careful consideration.

Compatibility with Existing Architectures. Attention Residuals can be viewed as a plug-and-play module, theoretically capable of coexisting with existing technologies like attention variants, MoE FFNs, and RoPE. However, how to avoid adverse interactions with these existing optimizations requires verification through extensive experimentation.

6. Significance and Implications

From a broader perspective, the proposal of Attention Residuals highlights an easily overlooked principle: a component long considered a “default setting” within a technology system may precisely hide the entry point for the next breakthrough.

The Transformer itself was a revolution in how information is processed across the time dimension compared to RNNs, and this exploration into residual connections attempts to extend the same logic to the depth dimension. The duality between these two dimensions provides a highly valuable analytical framework—it forces researchers to re-examine “taken-for-granted” design choices and consider whether there are hidden assumptions and limitations.

For tech practitioners focused on LLM architectural evolution, this direction offers at least threefold implications:

First, expanding model depth is not without cost. Simply stacking more layers and relying on fixed residual connections may not linearly improve a model’s representation capacity. The benefits of depth might be offset by information dilution.

Second, fine-tuning foundational components can yield systemic improvements. Compared to local optimizations on attention mechanisms or FFNs, modifying a global component like residual connections has a broader impact and potentially larger returns.

Third, cross-dimensional analogical thinking is a crucial source of innovation. Migrating solutions from the time dimension to the depth dimension is a powerful methodology in itself. Many seemingly novel architectural innovations essentially stem from the reapplication of existing concepts in new dimensions.

Currently, LLM architecture is in a new cycle of exploration. From the popularization of MoE architectures and the revival of State Space Models (SSMs) to the re-examination of foundational components like residual connections, all trends point to the same conclusion: the fundamental Transformer architecture is far from “set in stone.” Beneath every seemingly stable component, there remains design space worth exploring. For tech practitioners, maintaining a spirit of questioning foundational assumptions is of greater long-term value than simply keeping up with the latest model parameters.