Should Deep Residual Connections Be Replaced by Attention Mechanisms?

Over the past few years, Transformer architecture modifications have rarely slowed down. Attention mechanisms have evolved from standard multi-head attention into multi-query, grouped-query, and even linear attention variants. Feed-forward networks have moved from dense structures toward mixture-of-experts designs. Positional encodings have shifted from absolute positions to rotary position embeddings. Normalization layers have also been repeatedly revisited and adjusted.

However, one component has remained almost unquestioned for nearly a decade: the residual connection.

Since the rise of residual networks, adding the previous layer’s output directly to the current layer has become something close to an engineering law. But recent foundational architecture research raises a thought-provoking argument: this fixed-weight accumulation may be limiting the expressive capacity of very deep models. The core idea is framed through a perspective called time-depth duality.

The Revolution in the Time Dimension

To understand the value of this discussion, it helps to review how attention mechanisms replaced recurrent neural networks.

Recurrent neural networks compress all historical information into a fixed-size hidden state across the time dimension. No matter how long the sequence is, all information must pass through this capacity-limited bottleneck. Attention mechanisms take a different approach: at each step, the model retrieves relevant information from history on demand rather than passively accepting a repeatedly overwritten summary.

This comparison reveals a key pattern: replacing on-demand retrieval with fixed compression tends to fail quickly as the amount of information grows.

What happens if we apply the same logic to the depth dimension?

Standard residual connections do exactly that: they accumulate outputs from all previous layers all the way to the final layer with fixed weights. As the network gets deeper, the magnitude of the hidden state keeps growing, while the proportion of new information contributed by a single layer becomes smaller and smaller. In other words, the genuinely new information contributed by each layer is being diluted by accumulated history.

The PreNorm Dilution Effect

Modern large language models commonly adopt a PreNorm structure, meaning normalization is placed before each sublayer. Combined with unit-weight residual accumulation, this design has clear advantages for training stability, but it also introduces a side effect.

Let the hidden state at layer l be h_l and the sublayer transformation be f_l. The standard residual update is:

h_l = h_{l-1} + f_l(h_{l-1})

Under PreNorm, the input to fl is normalized first, so its output magnitude remains relatively stable. But h{l-1} itself keeps accumulating as the network deepens. As a result, in deeper layers, the proportion of h_l originating from earlier layers keeps increasing, while new information from the current layer becomes overwhelmed.

Researchers call this the PreNorm dilution effect. It helps explain why simply adding more depth does not always produce proportional performance gains. Newly added layers are numerically diluted, and their expressive capacity cannot be fully released.

Applying Attention to the Depth Dimension

If the bottleneck in the time dimension can be addressed with attention, can the bottleneck in the depth dimension be addressed the same way?

This is the central argument of the research: replace fixed residual connections in the depth dimension with attention mechanisms.

The analogy is straightforward:

• Time dimension: recurrent neural networks compress history into a fixed state → attention retrieves history on demand
• Depth dimension: fixed residual connections accumulate layers with fixed weights → depth-wise attention aggregates layers on demand

In practical terms, each layer would no longer simply add the previous layer’s output to itself. Instead, it would compute a weighted aggregation of outputs from all previous layers. The weights are dynamically calculated based on the current layer’s state, similar to attention scores.

This means deeper layers are no longer dominated by the accumulated magnitude of earlier layers. Instead, they can choose which layer representations to focus on. For tasks requiring shallow features, the network can retain more early information. For tasks requiring high-level semantics, it can focus more on recent outputs.

Implications for Architecture Design

This idea has several broad implications.

The gradient propagation path changes. Traditional residual connections provide an identity-mapping gradient path, which is key to training very deep networks. With depth-wise attention, gradients must flow back through attention weights, placing new demands on initialization and normalization strategies.

Training stability needs to be reassessed. One implicit benefit of fixed-weight accumulation is predictable numerical behavior. Once dynamic weights are introduced, the magnitude distribution of hidden states may become more complex and require additional stabilization techniques.

Parameter and computational overhead. Depth-wise attention introduces extra parameters and computation. It is necessary to balance improved expressive capacity against efficiency. One possible compromise is to use it only between selected layers or to adopt lightweight weight computation methods.

Compatibility with existing techniques. This change sits at the residual connection level, so it can theoretically be combined with orthogonal techniques such as mixture-of-experts, linear attention, and rotary position embeddings. However, the interaction effects still need experimental validation.

Considerations from an Engineering Perspective

From an engineering standpoint, this type of low-level architecture change faces several practical challenges.

The first is training cost. Any approach that changes the residual structure requires training from scratch or large-scale continued training to validate its effectiveness, which demands substantial resources.

The second is inference efficiency. If depth-wise attention introduces additional computation, it may affect inference latency and require targeted optimization on the serving side.

The third is ecosystem compatibility. Most existing model parallelism and pipeline parallelism schemes assume a standard residual structure. A new architecture may require corresponding adaptations in distributed training.

Finally, there is the evaluation framework. Improvements in deep expressive capacity often become visible mainly on tasks involving long-range dependencies or complex reasoning, so more targeted evaluation methods may be needed.

Summary

The question raised by this research may be more valuable than the answer it provides: if we have already accepted attention as a replacement for recurrence in the time dimension, why do we still hold on to fixed residual connections in the depth dimension?

Of course, fixed residual connections are not without merit. Their simplicity, stability, and efficiency have been part of the foundation for scaling deep learning over the past decade. Whether depth-wise attention can fully replace them remains an open question. But this discussion reminds us that in an era of rapid large-model architecture evolution, even components long treated as infrastructure may contain room for improvement.

For developers and researchers, what deserves attention is not one specific solution, but the mindset of re-examining default assumptions. Whether in training stability, deep expressive capacity, or architecture efficiency, new breakthroughs may emerge from this kind of low-level reconsideration.