From Self-Attention to Sliding Windows: A Complete Guide to Attention Mechanisms and How to Choose One

    A practical walkthrough of how attention works and why it dominates long-context cost. It covers why attention existed before the Transformer, a step-by-step derivation of self-attention, multi-head attention (MHA), grouped-query attention (GQA) and the KV Cache math behind it, sliding-window and sparse attention, and cross-attention for sequence-to-sequence and multimodal models. It also explains how to read a model architecture card (layers, hidden size, KV heads, attention variant, positional encoding, normalization, FFN, context length) and turns those fields into capacity-planning, serving, and benchmarking decisions, including the most common selection mistakes.

    1. Why It’s Worth Revisiting Attention from First Principles

    In large-model engineering, one class of “non-code asset” is routinely undervalued: architecture maps. Taking the layer count, hidden dimension, attention variant, positional encoding scheme, and context window of mainstream models and rendering them into model cards with a uniform template measurably reduces the communication cost of selection reviews, failure attribution, and capacity planning. These maps are popular not because they look good, but because they compress hyperparameters scattered across configuration files into a single structured table that can be compared side by side.

    But a map is only an index. What actually drives engineering judgment is an understanding of the underlying mechanism. Attention is the clearest case: it is both the core computation of the Transformer and the main consumer of memory and bandwidth, and it directly sets the ceiling on long-context capability. Understanding how it is derived is worth more than memorizing the head count of any particular model.

    2. Attention Predates the Transformer

    Attention was not invented for the Transformer. Its direct ancestor was the encoder–decoder recurrent neural network for machine translation. In those systems, the encoder RNN reads the source sentence word by word, compressing the whole sentence into a series of hidden states — in the simplest implementations, keeping only the final state vector. The decoder RNN then generates the target sentence word by word, starting from that limited summary.

    The problem becomes obvious quickly: the fixed-length state vector is an information bottleneck. The longer the source sentence, the more easily information carried by earlier words is overwritten by subsequent updates, and translation quality degrades noticeably as sentence length grows. At the same time, when the decoder generates the t-th target word, what it actually needs is to attend to the few source words aligned with it, not the averaged semantics of the entire sentence.

    The original motivation for attention was to break exactly this constraint: let the decoder compute dynamically, at every step, where to look in the source sentence, assign weights to different source positions, and take a weighted sum over the encoder states to obtain a context vector strongly correlated with the current decoding step. Note that three ingredients — weighting, normalization, and generating weights dynamically from the current state — were all established before the Transformer appeared. The Transformer’s contribution lies in two things. First, it generalized the mechanism from cross-sequence alignment to intra-sequence self-alignment, i.e. self-attention. Second, it replaced the sequential time-step dependency of recurrent networks with a fully parallel structure, so that training could fully exploit parallel compute.

    3. Self-Attention: Building Context-Aware Representations in Three Steps

    The goal of self-attention can be stated in one sentence: let every token in the sequence gather information from the other visible tokens according to relevance, producing a new representation that is context-aware. This gathering step is exactly why the same token should have different representations in “Apple released a new phone” and “I bought an apple.”

    The computation splits into three steps, implemented in practice with three sets of linear projections, conventionally denoted Q, K, and V:

    1. Produce query, key, and value: use three separate weight matrices to project the input representation into a query vector (what I am looking for), a key vector (what identifier I can offer), and a value vector (the content I actually carry).
    2. Score and normalize: take the dot product of the query with all keys, divide by the square root of the key dimension, then normalize. The scaling factor suppresses the effect where, as dimensionality grows, dot-product variance becomes too large, the normalized output tends toward uniform, and gradients shrink.
    3. Weighted aggregation: apply the normalized weights to the corresponding value vectors and sum them to obtain the output for that position. In autoregressive decoding, a causal mask must also be applied to block future positions, keeping training and inference consistent.
    1. import math
    2. def softmax(xs):
    3. m = max(xs)
    4. ex = [math.exp(x - m) for x in xs]
    5. s = sum(ex)
    6. return [e / s for e in ex]
    7. def attention(q, k_rows, v_rows, pos, window=None):
    8. """Attention of a single query vector against a set of K/V. Illustrative implementation."""
    9. d = len(q)
    10. scale = 1.0 / math.sqrt(d)
    11. scores = [sum(a * b for a, b in zip(q, k)) * scale for k in k_rows]
    12. if window is not None:
    13. # Sliding window: keep only tokens visible in the (pos - window, pos] range
    14. scores = [s if (pos - window < j <= pos) else -math.inf
    15. for j, s in enumerate(scores)]
    16. w = softmax(scores)
    17. out = [0.0] * d
    18. for wi, v in zip(w, v_rows):
    19. for j in range(d):
    20. out[j] += wi * v[j]
    21. return out

    The complexity is worth emphasizing. For a sequence of length n and model dimension d, both the time and memory cost of a single attention layer grow with n². This quadratic term is the root cause of runaway long-context cost, and the core problem every derived variant is trying to solve.

    4. Multi-Head Attention (MHA): The Standard Shape and Its Cost

    Multi-head attention is the standard form of self-attention inside the Transformer. It runs several attention heads in parallel, each with its own independently learned projection matrices, performs the three steps above in its own subspace, and finally concatenates the head outputs and fuses them into a unified representation through an output projection matrix.

    The point of multiple heads is a division of labor across representation subspaces: different heads can spontaneously learn different relational patterns, such as local adjacency, syntactic pairing, or long-distance coreference. A single head is limited to one scoring function and can express only one notion of relatedness; multiple heads expand the model’s expressive capacity at a modest implementation cost, and since the heads are fully parallel computationally, they add almost no serial depth.

    In a standard configuration, the number of heads h and the model dimension d satisfy the relation that each head has dimension d/h, so after concatenation the dimension is still d, and the parameter count of the output projection does not inflate as the head count grows.

    5. From MHA to Grouped-Query Attention: Inference Memory Is the Real Constraint

    The drawback of MHA only surfaces at inference time. During autoregressive decoding, to avoid recomputing the keys and values of historical tokens, engineering practice caches them — the KV Cache. Its memory footprint can be approximated as:

    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    2. """2 accounts for the key and value copies; bytes_per_elem=2 corresponds to half precision"""
    3. return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem

    The critical detail is that in MHA the number of key/value heads equals the number of query heads, typically several dozen. As batch size and sequence length grow, the KV Cache quickly approaches or even exceeds the memory footprint of the model weights themselves, and becomes the main bandwidth bottleneck of the decoding phase — every generated token requires reading the entire cache again. Two paths emerged to shrink the cache:

    • Multi-query attention (MQA): all query heads share a single set of keys and values, compressing the cache to 1/h of its original size. The cost is a significant drop in key/value expressive power and clearly degraded model quality.
    • Grouped-query attention (GQA): a compromise. The h query heads are divided into g groups, sharing one set of keys and values within each group, so the number of key/value heads is g. When g equals h it degenerates to MHA; when g equals 1 it degenerates to MQA. In practice, choosing g around 8 usually yields substantial cache compression with almost no quality loss.
    1. def group_kv_heads(num_query_heads, num_kv_heads):
    2. """Split query heads into groups that share the same set of keys/values"""
    3. assert num_query_heads % num_kv_heads == 0
    4. g = num_query_heads // num_kv_heads
    5. return [[i for i in range(h * g, (h + 1) * g)] for h in range(num_kv_heads)]
    6. # 32 query heads, 8 key/value groups -> every 4 heads share one set of K/V

    One common misconception needs clearing up: GQA only reduces the reading and caching of keys and values. The number of query heads is unchanged, and the floating-point work in the attention scoring stage does not decrease proportionally. Its benefit shows up mainly in memory footprint and bandwidth efficiency during decoding; actual throughput gains must be measured on the specific hardware and batching strategy and cannot be inferred from the structure alone. In addition, GQA has fewer key/value projection parameters than MHA, so converting a model from MHA to GQA introduces slight parameter and quality differences that need to be validated on the target dataset.

    6. Sliding-Window and Sparse Attention: Pushing the Quadratic Term Back to Linear

    If the requirement is very long context, GQA alone is not enough to solve the n² cost. The direction then shifts to limiting each token’s visible range:

    • Sliding-window attention: each token can only see a fixed-length window of w tokens to its left, reducing single-layer complexity from O(n²·d) to O(n·w·d). The cost is a limited receptive field, but with multiple layers stacked the receptive field expands roughly linearly with depth, so deep networks can still obtain distant information indirectly.
    • Mixing global attention with local layers: some layers retain global visibility, taking on the job of aggregating information over long ranges; the rest use local windows. A common arrangement alternates local and global layers at a fixed ratio, so the model keeps a global path without paying the full quadratic cost everywhere.
    • Hierarchical and dilated strategies: sampling keys and values with skips, or aggregating them hierarchically, trades controllable information loss for a larger nominal coverage length.

    These schemes are tightly coupled with positional encoding design. With relative positional encodings or scaled sinusoidal rotary schemes, behavior at window boundaries is smoother, and it is easier to extend to lengths not seen during training via positional interpolation.

    7. Cross-Attention: Aligning Information Across Sequences and Modalities

    Self-attention looks inside a sequence; cross-attention is used to align two sequences. Its typical form has the decoder side produce queries while the encoder side provides keys and values. It is a key component in two kinds of scenarios:

    • Sequence-to-sequence tasks: translation, summarization, and similar tasks that need soft alignment between a source and a target sequence.
    • Multimodal fusion: using the output sequence of a vision or audio encoder as keys and values, letting the text side query and extract cross-modal information on demand — one of the mainstream approaches for fusing features in image–text understanding models.

    Understanding the difference between these two kinds of attention helps localize problems quickly. If the generated output is clearly disconnected from the input, suspect feature alignment and projection layers on the cross-attention side first; if long-range internal dependencies are failing, the problem more likely lies in the self-attention window configuration or the positional encoding.

    8. How to Read a Model Architecture Card

    When facing a model card, reading it in the following order gives the highest information density:

    1. Scale parameters: layer count, hidden dimension, attention head count, key/value head count, vocabulary size, total and active parameter counts. The key/value head count is the critical field for judging inference cost.
    2. Attention variant: MHA / MQA / GQA, the number of key/value groups; whether sliding windows are used and their length; the ratio and layout of local versus global layers.
    3. Positional encoding: scheme type, maximum length during training, whether length extrapolation is supported.
    4. Normalization and residuals: pre-norm or post-norm, which has a significant impact on training stability.
    5. Feed-forward structure: intermediate dimension and gating structure, which usually accounts for the majority of the parameter count and directly affects the capacity-to-compute ratio.
    6. Context and data signals: training context length, multilingual coverage, tokenization granularity.

    When comparing models side by side, do not compare only the total parameter count. Two models with similar parameter counts can support per-GPU concurrency that differs by several times if their key/value head counts and attention variants differ — and that is often what actually determines the unit cost of serving.

    9. Engineering Recommendations and Common Pitfalls

    Capacity planning. First estimate the input and output length distribution of the target scenario, compute the peak KV Cache from it, then add weight, activation, and framework overhead to back out how many concurrent requests a single GPU can carry. Long-document summarization, code completion, and multi-turn conversation have drastically different cache growth curves; applying one configuration across all of them leads directly to wasted resources or constant queuing.

    Serving shape. Inference services are typically deployed as containers, paired with elastic autoscaling and batch scheduling. Using general-purpose capabilities from a cloud platform such as Baidu AI Cloud, you can archive model weights, architecture cards, and evaluation reports in object storage with version management, run inference services on a container platform, and track key metrics with logging plus a monitoring and alerting system. The metrics that matter most are time to first token, inter-token output interval, output tokens per second, GPU memory watermark, request queue time, and tail-latency percentiles — looking only at average latency hides the extreme experience inside batch queues.

    Optimization techniques. Quantization (of weights and of the KV Cache, separately), continuous batching, and IO-aware kernel-level attention implementations are the main levers for reducing the cost of long-context serving. Each carries precision and compatibility trade-offs and must be accompanied by regression evaluation rather than enabled by default.

    Common pitfalls. First, equating parameter count with capability and ignoring differences in data quality and training recipe. Second, equating “supported length” with “effectively used length” — recall of information in the middle of very long inputs often drops noticeably, which must be verified separately with retrieval-style probe tasks rather than measured only with language-model perplexity. Third, treating structural reasoning as a performance conclusion: any replacement of an attention variant should be confirmed with benchmark load tests. Fourth, ignoring the effect of tokenization granularity on both cost and quality.

    Finally, stay clear-eyed: attention is not the only path. Linear attention, state space models, and hybrid stacked architectures are all still evolving, and their appeal comes equally from linear complexity on long sequences and more cache-friendly properties. The right approach to engineering selection is to return to the business workload itself — run small-scale comparative experiments on data with the real distribution, turn structural differences into a quantifiable triple of cost, latency, and quality, and decide from there.