Latent Sparse Attention (LSA): Cutting Sparse Attention's Indexer Overhead at 10M-Token Contexts
As million-token context becomes a routine default for leading open-source models, the bottleneck of sparse attention has moved from the attention operator to the indexer that decides which tokens to read. At 10M tokens, roughly 98% of the prefill time in a global-attention layer can be spent on indexing rather than attention compute. This article explains why indexing cost still scales quadratically even when sparse aggregation stays linear, and how Latent Sparse Attention (LSA) compresses the scoring step into a low-dimensional latent space—preserving ranking rather than exact precision—to cut that overhead while holding accuracy. It also explains why reported speedups vary so widely (22.7x vs. 72.4x) depending on baseline maturity, and lists the practical factors to watch when adopting the technique.
1. After 1M Context, the Center of Attention Cost Is Shifting
In the first half of 2026, leading open-source models have progressively made million-token (1M) context the default spec for their official services, and inference frameworks and long-context benchmarks have largely matured around this scale. Million-token context is no longer a “can it be done” question but a routine engineering configuration.
An often-overlooked fact is that validating sparse attention at this scale really only validates reading. Letting each query stop scanning all historical tokens and fully interact with only a few of the most relevant ones is indeed a prerequisite for long-context inference to work at all. But beyond that conclusion lies another cost: to pick “that small part,” the system must first evaluate every candidate token.
In this setup, a layer that uses global attention, during prefill at 10M tokens, spends roughly 98% of its time not on the attention operator itself, but on the indexer responsible for “which tokens to look at.” Attention compute has stepped down from the bottleneck position, and the indexer has stepped up.
The nature of the problem therefore changes. Sparse attention proved that “reading only a small part is enough to preserve accuracy,” but it never solved “finding that small part still grows quadratically with context length.” Compute complexity was brought down; selection complexity stayed where it was—and the latter is exactly the dominant term in the 10M-token regime.
2. Where the Bottleneck Comes From
Attention can be split into two stages:
- Scoring/selection stage: evaluate each query’s relevance to candidate keys and produce the index set that will take part in full computation;
- Aggregation stage: take a weighted sum over the selected key-value pairs and output the context representation.
Sparsification acts only on the second stage. If the indexer uses the most common approach—compute a relevance score between each query and all candidate keys, then take Top-K—then the first stage is itself on the order of O(L²), where L is the sequence length. Sequence length up 10x, indexing cost up 100x, while the sparse aggregation part only grows 10x. The two curves meet near the million-token mark, and beyond that point the indexer pulls ahead on its own.
Worse, the indexer usually lacks the mature fused kernels, temporal sharing, and memory-hierarchy optimizations that the attention operator enjoys. Matrix multiply over long sequences is bandwidth-bound and can be fully pipelined; logic like “compute all scores, then sort and take the top K” inherently contains global reductions and sorting, with poor parallelism and irregular memory access, so its measured effective compute often falls far below the theoretical peak. For the same theoretical complexity, the indexer’s wall-clock time looks worse than attention’s.
So the real fix is not to keep shrinking K. It is to change the dimensionality along which the scoring process operates: let relevance evaluation happen in a space far lower-dimensional than the original head dimension.
3. The Core of LSA: Moving Scoring into Latent Space
The idea behind LSA (Latent Sparse Attention) can be summarized as follows: the indexer does not score the raw head-dimension key vectors directly, but scores a compressed representation (latent representation) of the keys. Keys are first projected into a low-dimensional latent space through a learnable down-projection, queries are projected the same way, and both relevance scoring and Top-K selection happen in this low-dimensional space. Only the selected tokens have their full key-value vectors retrieved to enter the second-stage aggregation.
Expressed in two stages:
# Stage 1: low-dim scoring + selection (cheap, approximate)K_lat = W_k @ K # [L, d] -> [L, d_lat], d_lat << dQ_lat = W_q @ Q # query is projected the same low-dim wayscores = Q_lat @ K_lat^T # scoring happens in latent spaceidx = topk(scores, k) # keep only the top-k indices# Stage 2: full-precision aggregation (expensive, but only on k tokens)out = softmax(Q @ K[idx]^T) @ V[idx]
Why this step is worth it: latent-space scoring does not need to recover the original precision; it only needs to keep the ranking approximately correct. This is a highly fault-tolerant approximation—the indexer’s sole job is to rank “the few most relevant tokens” into the top K, and slight ordering deviations do not affect final quality. That means the representation used for scoring can be far smaller than the head dimension, and the matrix-multiply scale of the scoring stage drops accordingly. In the original design the bottleneck was index scoring; once that piece is compressed, the dominant 98% overhead is pulled down as a whole.
4. Why the Speedup Ratios Differ So Much: 22x and 70x Are Not the Same Thing
LSA achieves very different speedups on different baseline models: 22.7x on a widely used open-source sparse-attention implementation, and 72.4x on another open-source weight model (both measured at 10M tokens, single-layer prefill).
This is not one model architecture favoring the method over another; it is that the denominators differ. The denominator of the speedup is the original implementation’s indexer time, and the maturity of each open-source indexer varies enormously: some heavily fuse scoring and sorting and reuse the attention kernel’s blocking and streaming; others are still a naive “full scoring + full sort,” which at 10M tokens is almost pure waste. The more naive the baseline, the larger the denominator, and the higher the multiple naturally.
Two points therefore need to be clear to readers:
- These numbers cannot be taken directly as end-to-end inference speedups. They measure the prefill stage of a single attention layer. End-to-end also includes embedding, feed-forward networks, communication, sampling, and scheduling, and sparse attention’s own share changes with the layer structure.
- A 72x magnitude is more of a signal about baseline maturity. If a long-context solution spends 98% of its time on indexing at 10M tokens, the first thing to do may simply be to raise the indexer’s own implementation efficiency.
This comparison also yields an independent engineering lesson: if your sparse-attention indexer dominates time on long sequences, then it deserves separate optimization, whether or not you adopt latent-space scoring.
5. Why Precision Doesn’t Collapse: The Approximation Sits in the Right Place
Sparse attention can preserve accuracy in long context on one premise: the “only keep a part” assumption holds—relevance is highly concentrated, and most tokens contribute something close to noise to a given query. LSA’s additional risk is that compressed representations may cause genuinely relevant tokens to be missed. It has several layers of protection against this:
- Binarization rather than fine-grained ranking. Selection only needs to decide “does this enter the top K”; latent-space scores need not restore the true weights of the original attention, and ranking approximation error is absorbed by tolerance.
- A conservative candidate pool. Top-K usually leaves headroom and selects on block boundaries, avoiding squeezing out edge tokens just to save compute.
- Local windows and attention sinks retained. Recent tokens and a few tokens at the very start of the sequence are often shared across many queries; these “must-select” items are kept out of the scoring competition.
- A hybrid of sparse and linear attention. Letting some layers retain a dense or linear-form global receptive field can significantly ease the sensitivity of retrieval-style tasks to exact recall.
In terms of task performance, LSA stays close to the original model on evaluations that require long-range information integration, rather than merely matching on mild metrics like perplexity. This matters more for real-world selection: what is genuinely hard in long-context inference is multi-hop integration and cross-segment coreference beyond “needle in a haystack,” and perplexity parity alone is unconvincing.
6. What to Watch When Putting It into Practice
- Measure your baseline first. Break down prefill time by operator on your own model and framework, and confirm the indexer’s real share. If it is not the dominant term, the gains from latent-space scoring get diluted and are not worth the complexity of changing the kernel.
- View the gains by stage. Prefill gains the most; in the decode stage each step has only one query, so index scoring is already cheap and the gains are limited—though you still benefit from a smaller indexer state and lower branch overhead.
- Separate lossless from lossy. Sparse selection itself sits on the “approximately lossless” side; the error introduced by compressed scoring is a different matter. The harder you compress, the more visible the recall loss, so you need long-context regression at both the 1M and 10M ends rather than looking only at short-sequence metrics.
- Watch hybrid-architecture consistency. When some layers keep a dense/linear form, training and inference must align strictly; if the difference in receptive fields between layers is too large, some layers will be “unable to see” information other layers have already retrieved.
- Don’t look only at single-layer speedups. Evaluation should rest on end-to-end first-token latency, output-token throughput, and the shape of the speedup curve from 1K to 10M; single-layer numbers are only explanatory tools.
- Account for framework-side support. Long-context inference also needs chunked and streaming prefill scheduling, hierarchical management and on-demand loading of the key-value cache, and tiered placement across device memory and host memory. These system capabilities stack with attention-kernel optimization; they are not substitutes for it. When deploying in the cloud, placing such workloads on a container platform that supports elastic scaling—together with monitoring and alerting on first-token latency and throughput—is usually more effective than tuning a single kernel parameter.
7. Conclusion: Sparsification Enters the Era of “Selection”
This round of million-context competition validated sparse attention’s ability to “read less yet read accurately.” The 10M-token round exposes something else: the indexer is becoming the new dominant cost, and its growth curve is steeper than attention’s. LSA compresses the scoring process into a low-dimensional latent space, directly attacks this dominant term, brings single-layer prefill at 10M tokens back into an acceptable range, and preserves accuracy at the same time.
The methodology it offers is more universal than the method itself: in a long-context system, question one step at a time “which step’s compute scale exceeds what its actual responsibility requires.” Scoring only needs ranking, and ranking does not need full precision; selection only needs coarse granularity, while aggregation is what needs detail. Putting complexity back where the responsibility lies is why this class of optimization keeps paying off.
What deserves a clear head is this: numbers like 22x to 72x come from controlled single-layer benchmarks, and each baseline’s implementation maturity determines the size of the denominator. To truly evaluate the gains, draw that speedup curve running from 1K out to 10M on your own model, your own kernel, and your own hardware.
