When Residual Connections Meet Attention: Rethinking the Depth Dimension of Transformers

1. The Overlooked “Founding Member”

Over the past few years, researchers have overhauled nearly every component of the Transformer to make it stronger:

• Attention modules: evolving from standard multi-head attention to multi-query/grouped-query attention, and further to various linear and sparse attention variants, with the core goals of reducing inference cost and extending context length;
• Feed-forward networks: moving from dense FFNs to Mixture-of-Experts (MoE), trading sparse activation for larger parameter capacity;
• Positional encoding: progressing from absolute positional encoding to relative schemes such as rotary positional encoding, to better extrapolate to long sequences;
• Normalization: the debate between LayerNorm and RMSNorm, and the placement trade-offs between PreNorm and PostNorm, have both been repeatedly refined.

Yet one component has remained virtually untouched for nearly a decade since the residual connection was proposed: hl = h{l-1} + f(h_{l-1}) — simply adding the previous layer’s output directly to the current layer’s transformation result. It is so simple, stable, and effective that it has gradually evolved from an engineering trick into an “unquestionable physical law.”

But recent research has raised systematic doubts about this default setting: a fixed, undifferentiated accumulation residual mechanism may be quietly dragging down the expressive capacity of deep models.

2. Diagnosing the Problem: PreNorm Dilution

1. The Status Quo: PreNorm + Unit-Weight Accumulation

Modern large language models overwhelmingly adopt the PreNorm structure, whose core computation can be abstracted as:

h_{l+1} = h_l + Attn(Norm(h_l))
h_{l+2} = h_{l+1} + FFN(Norm(h_{l+1}))

After recursive expansion, the output of layer L is equivalent to:

h_L = h_0 + f_1(h_0) + f_2(h_1) + ... + f_L(h_{L-1})

In other words, the outputs of all layers are accumulated forward with a constant weight of 1, with no mechanism to distinguish which layer’s contribution matters more.

2. Two Chained Problems

Problem one: Unbounded norm growth. As depth increases, the norm of the hidden state keeps growing. To maintain numerical stability, normalization layers must continuously “suppress” the magnitude, which itself consumes model expressive capacity.

Problem two: Individual layer information gets diluted. Assuming each layer’s newly added information has a similar magnitude, the contribution of layer l to the total sum is roughly on the order of 1/l. The deeper the network, the weaker any single layer’s “incremental correction” becomes in the overall signal. Researchers call this phenomenon PreNorm dilution.

An intuitive analogy: a choir where new singers keep joining, each singing at the same volume. By the hundredth layer, any new singer’s voice accounts for only one percent of the whole — even if they are carrying the most critical correction, the model struggles to “hear” it.

3. Potential Impact on Model Behavior

• Diminishing returns from depth: when stacking more layers, the marginal benefit of new layers may be offset by the dilution effect, offering an explanation for the engineering rule of thumb that “going wider beats going deeper”;
• Gradient propagation path: during backpropagation, gradients also flow along this fixed-weight accumulation path, making it hard for deep supervision signals to stand out;
• Connection to scaling laws: if information utilization in the depth dimension is poor, then simply stacking more layers is not an efficient scaling path.

3. The Core Idea: Time-Depth Duality

The most valuable part of this research is that it offers an elegant thinking framework, not merely a new module.

A look back: why was the RNN replaced by Attention?

In the time dimension, the RNN compresses all historical information indiscriminately into a fixed-size hidden state: with each new token, old information gets mixed and washed out once more. The longer the sequence, the harder it is to fully preserve early information, making long-range dependencies a persistent challenge. The Transformer uses attention to achieve “on-demand retrieval” in the time dimension — when generating the representation at position t, the model can dynamically decide which historical positions to attend to and by how much, thereby bypassing the fixed-compression bottleneck.

Now rotate this perspective 90 degrees and map it onto the depth dimension:

Isn’t the standard residual connection doing exactly the same thing — accumulating the outputs of all previous layers with a fixed weight into an ever-expanding hidden state? With each new layer output, old information gets further diluted. The deeper the layer, the harder it is for any single layer’s contribution to stand out — structurally isomorphic to the RNN’s predicament on long sequences.

This yields a natural corollary: just as “fixed compression” in the time dimension was revolutionized by attention, “fixed accumulation” in the depth dimension deserves to be attention-ized as well — letting the network learn to retrieve information from historical layers on demand, rather than treating all layers alike. This is the essence of the so-called “time-depth duality.”

4. Technical Paths: How to Attention-ize Residual Connections

1. From Fixed Summation to Dynamic Aggregation

Formally, the residual update can be generalized from

h_{l+1} = h_l + f_l(h_l)

to

h_{l+1} = Σ_{i≤l} α_i · h_i + f_l(·)

where the aggregation coefficients α are no longer a constant 1, but dynamic weights computed jointly from the current state and the historical layer outputs — the most natural implementation being inter-layer attention: using the current layer as the query and the historical layer outputs as keys and values.

2. A Spectrum: From Lightweight to Full

Ordered by degree of “dynamism” from low to high, a technical spectrum emerges:

1. Learnable scalar residual: multiply each layer’s residual branch by a learnable scalar coefficient, initialized near 1 or 0 — an almost zero-cost modification;
2. Gated residual: use a small network to output per-dimension or per-channel gate values based on the current state, turning “how much to add” into a function of the state;
3. Inter-layer attention aggregation: perform full attention in the depth dimension, so that any layer can dynamically weight and retrieve the outputs of all historical layers — the most thorough realization of the duality idea.

3. Illustrative Implementation

# Paradigm 1: standard PreNorm residual (fixed-weight accumulation)
H = [x0]
for block in blocks:
    h = H[-1] + block(norm(H[-1]))   # each layer's contribution has a constant weight of 1
    H.append(h)
# Paradigm 2: attention-ized residual (dynamic aggregation in the depth dimension)
H = [x0]
for block in blocks:
    h_prev = H[-1]
    ctx = depth_attn(query=h_prev, keys=stack(H), values=stack(H))
    h = ctx + block(norm(ctx))
    H.append(h)

It is worth emphasizing that the convolutional network era also saw attempts at cross-layer dense connections, but these mostly relied on concatenation followed by a fixed transformation; the key difference here is the use of attention to achieve “input-dependent dynamic weighting,” deeply integrated with the PreNorm system so that aggregation behavior can adapt to input content.

5. Engineering Challenges and Trade-offs

Any architectural innovation must pass the engineering test, and attention-ized residuals bring at least four sets of trade-offs:

1. Activation memory: retaining historical layer outputs for attention retrieval significantly increases activation memory during training. Possible mitigations include keeping only the most recent few layers (sliding window), applying low-rank projections to historical layers, or downsampling summaries;
2. Compute overhead: the cost of inter-layer attention grows quadratically with the number of layers. Control methods are isomorphic to attention optimization in the time dimension: windowing, chunking, linear approximations, and so on;
3. Training stability: the distributional properties of softmax aggregation during initialization, temperature settings, and how it interacts with PreNorm all need careful design to avoid disrupting already-validated optimization properties; progressive introduction is also a common strategy;
4. Parallel training compatibility: cross-layer dependencies change the communication patterns and pipeline bubble distribution under pipeline parallelism and tensor parallelism, requiring a re-evaluation of distributed strategies.

In addition, evaluation criteria need to be upgraded: it is not enough to look only at validation perplexity — scaling curves under different depth/width configurations, downstream task performance, and inference cost should also be examined to determine whether it constitutes a genuine Pareto improvement.

6. A Rational View: The Value and Lessons of Architectural Innovation

For practitioners and enterprises, the lessons from this line of work may matter more than the specific structure itself:

• First-principles review: most “default settings” deserve periodic re-examination. The power of the duality argument lies in the fact that it is not empirical tuning, but rather extracts a structural pattern from an already-successful case (attention in the time dimension) and generalizes it to a dimension not yet revolutionized (depth);
• Transfer value: even if inter-layer attention does not ultimately become mainstream, the idea of “adaptive aggregation in the depth dimension” can be transferred to adjacent directions such as conditional computation, early exit between layers, and dynamic routing;
• Infrastructure coupling: if improvements in depth utilization efficiency are validated, it means deeper models can be trained more efficiently under the same compute budget. This will impose new requirements on the cloud engineering capabilities that large-scale training relies on — compute scheduling, memory optimization, distributed communication, monitoring, and alerting — and architectural research and infrastructure evolution need to advance in tandem.

7. Conclusion

The residual connection is one of the most successful “default settings” in deep learning history. It made hundred-layer networks possible and kept “deeper is stronger” valid for a long time. But default does not mean optimal: as model depth moves from dozens of layers to hundreds, the dilution effect of fixed-weight accumulation begins to enter a non-negligible range.

Time-depth duality offers a concise yet profound perspective: attention once liberated information retrieval in the time dimension; now it is the depth dimension’s turn to receive the same scrutiny. Regardless of the specific form this direction ultimately takes, the mindset of “re-problematizing fundamental components” is worth every large-model practitioner adding to their toolkit.