Rethinking Transformer Residual Connections: Solving Signal Dilution in Deep PreNorm Architectures

The Overlooked Building Block: Residual Connections

Over the past few years, most core Transformer components have gone through at least one round of redesign. Attention has evolved from multi-head attention to grouped-query and multi-query variants; feed-forward networks have moved from dense structures to mixture-of-experts; positional encoding has progressed from absolute and relative schemes to rotary position embeddings; and normalization has been repeatedly debated between LayerNorm and RMSNorm. The residual connections that tie these components together, however, have barely changed at a fundamental level.

Since ResNet introduced them, residual connections have kept the same basic form:

h_l = h_{l-1} + f_l(h_{l-1})

Here f_l is the transformation function of layer l, and h_{l-1} is the output of the previous layer. This “add the input directly to the current layer’s output” pattern is widely treated as a self-evident default. It addresses the vanishing gradient problem in deep network training: when backpropagation passes through the identity branch, gradients can flow back without going through the transformation function, avoiding the decay caused by repeated multiplication.

But is this unconditional identity accumulation truly optimal at every depth and for every task? As model scale grows rapidly, this question has become increasingly pressing.

Residual Dilution in PreNorm Architectures

Modern large language models generally adopt the PreNorm structure: each sublayer is normalized before its input is passed to the attention or feed-forward module. The computation flow can be summarized as:

x = x + MultiHeadAttention(Norm(x))
x = x + FeedForward(Norm(x))

Compared with PostNorm, this design is easier to train, but it exposes a structural flaw as depth increases: every layer’s output is added to the global hidden state with a fixed weight of 1. If Δ_l denotes the transformation output of layer l, then after L layers the final hidden state can be written as:

h_L = h_0 + Δ_1 + Δ_2 + ... + Δ_L

All layer increments are summed with equal weight. As L grows, the magnitude of the hidden state trends upward overall, while the relative share of any single layer’s new information in the total keeps shrinking. This has two direct consequences:

First, deep-layer information gets drowned out. Each new representation Δ_l that a layer computes contributes only a tiny increment relative to the large state accumulated by all preceding layers. The deeper the network, the weaker a single layer’s actual influence on the final representation.

Second, the effective representation space gets compressed. As the hidden state magnitude keeps growing, normalization layers continually rescale it—but rescaling cannot recover structural information that has already been diluted. Deep networks effectively make small patches on an ever-growing body of “historical baggage” rather than performing high-quality transformations in a refined representation space.

In short: residual connections accumulate information across layers with fixed weights, which is essentially a form of “indiscriminate information compression” along the depth dimension. Each layer’s new information is forced to share a single representation channel with all historical information, with no mechanism to selectively preserve, strengthen, or weaken specific historical components as needed.

Rethinking Depth-Wise Propagation Through a Time-Duality Lens

One intuitive way to understand this problem is through an analogy: sequence modeling along the time dimension and layer-by-layer propagation along the depth dimension are structurally dual to each other.

In sequence modeling, early RNNs compressed the entire history into a fixed-size hidden state. The problem with this mechanism is that when sequences get long, information from earlier time steps gets overwritten or diluted by later ones. Transformers changed this with attention: instead of compressing all history into a fixed vector, the model builds, for each current position, the ability to retrieve from all time steps on demand. Query vectors match against key vectors, attention weights are assigned by relevance, and historical information is aggregated selectively.

Back on the depth dimension, standard residual connections behave very similarly: they indiscriminately accumulate the outputs of all preceding layers into a single hidden state. As the number of layers grows, this state gradually inflates, and each layer’s new information occupies only a tiny share of it. This is highly isomorphic to the “mindless compression” problem RNNs face along the time dimension.

If attention has already proven along the time dimension that “selective retrieval” beats “indiscriminate compression,” can a similar idea apply along the depth dimension? This is the key perspective for understanding how residual mechanisms might improve.

Put differently, the real question is not whether residual connections should exist, but whether they must use a fixed, equal-weight, unselective accumulation scheme. A natural extension is to give each layer more autonomy in deciding what to write into the global state, while making that write policy learnable and adjustable.

Design Ideas and Implementation Principles for Attention-Style Residuals

Within this framework, the core goals of improving residual connections can be summarized in three points:

1. Suppress history dominance: prevent the accumulated output of earlier layers from overwhelmingly shaping the final representation.
2. Strengthen deep-layer increments: ensure that information produced by deep transformations is not easily diluted.
3. Preserve training stability: introduce a learnable modulation mechanism without destroying the gradient propagation benefits of the identity mapping.

This leads to an architectural improvement direction: change depth-wise information aggregation from “fixed identity accumulation” to “learnable weighted aggregation.” The general form can be expressed as:

h_l = Aggregate(h_{l-1}, f_l(h_{l-1}), α_l)

where Aggregate is a differentiable aggregation function and α_l is a modulation parameter for layer l. In practice, Aggregate can be a scalar gate:

g = sigmoid(W_g · x + b_g)
h = (1 - g) * h_prev + g * f_l(x)

Alternatively, it can adopt a more structured, attention-style design: treat the accumulated state of the residual stream as the “query,” treat historical increments from each layer as “key-value pairs,” and use attention weights to decide how much information to extract from which historical layers.

It is worth noting that the goal of these designs is not to fully replace identity connections, but to add a “controllable information valve” to the residual stream. In shallow models, simple identity accumulation may be sufficient; but in deep models with dozens or even hundreds of layers, unselective equal-weight accumulation causes severe imbalance in how much each layer’s information contributes. Learnable gating or attention-style aggregation lets the model dynamically adjust the influence of historical information based on the current layer’s representation needs.

From an engineering standpoint, several key principles matter:

• Keep the normalization position unchanged: apply the improvement at the aggregation step of the residual branch rather than reintroducing PostNorm. PreNorm’s training stability advantages remain important.
• Control the added parameter count: gating parameters should mostly consist of a single linear transform over the input dimension—negligible relative to the whole model.
• Pay attention to initialization: the gate’s initial bias should keep the output close to standard residual behavior, so the model retains most of the identity mapping’s benefits at the start of training and gradually learns to modulate.
• Monitor gradients: in deep configurations, track the gradient norms of the residual stream and the magnitude changes of hidden states to avoid introducing new numerical instability.

Practical Value and Application Directions

This line of thinking has direct practical value. Today, as large language models scale in depth, several phenomena recur: the returns from adding more layers keep shrinking, deep-layer representation quality degrades, and context forgetting worsens on long-sequence tasks. These problems cannot all be attributed to attention window size or positional encoding—the residual mechanism along the depth dimension is also an important variable.

For teams building deep models, evaluating and improving residual aggregation may raise model quality without a large increase in compute. Specific scenarios worth trying first:

• Very deep models: when the layer count exceeds a few dozen, residual dilution becomes more pronounced, and introducing learnable depth-wise attention typically yields more stable training curves.
• Long-document understanding and reasoning: these tasks are more sensitive to deep-layer representation quality. The model needs to retain and strengthen key information across layers while weakening irrelevant historical accumulation.
• Multi-step complex reasoning: the model must keep referencing key early information throughout the reasoning chain. Selective propagation in the deep residual stream helps mitigate the problem of information being drowned out.

In addition, on general cloud platforms for model training and inference, this direction is closely related to engineering topics such as performance tuning and inference stability. When training large language models in the cloud, teams tend to focus on distributed training efficiency, communication optimization, and memory management—but the residual propagation strategy at the architecture level also affects final quality. Combining architectural improvements with a cloud platform’s distributed training, monitoring and alerting, and log analysis capabilities makes it possible to evaluate improvements more systematically.

Summary and Outlook

Residual connections have provided the fundamental guarantee for deep network training for nearly a decade, but fixed equal-weight accumulation is not the optimal solution. Along the depth dimension, information from all historical layers is indiscriminately accumulated into an ever-growing hidden state, diluting new information from deep layers and degrading representation quality. Introducing learnable, attention-style depth-wise aggregation—letting the model selectively choose and strengthen representation information from different layers—is a direction worth watching for improving the expressiveness of deep models.

Going forward, the evolution of residual mechanisms may further combine with sparsification, gating strategies, and dynamic routing to form a more flexible system for depth-wise information propagation. For developers, breaking out of the “residual connections are untouchable” mindset and re-examining model architecture from an information flow perspective may be exactly the key to unlocking the next stage of capability gains.