Long-Context LLMs: Solving the Million-Token Cost Problem
Long-context large language models are expensive, and this article explains why and how to make them affordable. It starts with the two forces pushing context lengths up—test-time scaling and AI agents—then breaks down the core bottlenecks: quadratic attention complexity and a KV cache that grows linearly with sequence length. It walks through the main technical strategies for keeping cost down while preserving capacity: Mixture-of-Experts for parameters, sparse and hierarchically compressed attention to cut attention cost, enhanced residual connections to keep deep networks trainable, and matrix-orthogonalization optimizers for convergence efficiency. It then covers the training, inference, and post-training infrastructure needed to ship these designs, how to evaluate effective context length rather than nominal limits, and four practical recommendations for engineering teams.
Over the past few years, the competition in large models centered on “bigger”: more parameters, more data, higher pass@1. After 2025, the focus shifted visibly toward “longer and better at reasoning.” On one hand, the test-time scaling paradigm emerged, encouraging models to generate longer chains of thought before answering. On the other hand, AI agent applications require models to continuously process massive trajectories. Together, these two forces pushed long context from a “nice-to-have” capability to a key threshold that determines whether a model can enter core business workloads. But long context is not free. As the context expands from a few thousand tokens to hundreds of thousands or even a million tokens, the quadratic complexity of attention computation, the GPU memory footprint of the KV cache, and the system overhead of training and inference all scale up rapidly.
This article centers on that contradiction: why long context is hard, how current mainstream technical approaches mitigate it, and how engineering teams should put these designs into practice.
1. Where the problem comes from: the shared pressure of test-time scaling and agents
The classic scaling law describes the rule that “investing more parameters and data and training longer makes the model stronger,” and scaling happens during training. Test-time scaling moves the scaling point to the inference stage: a model can invest more computation in a single answer, deliberating repeatedly and gradually approaching the correct answer. For tasks such as math competition problems, complex code repair, and multi-step logic planning, longer reasoning traces often mean lower error rates.
The agent paradigm pushes this trend to the extreme. A truly usable agent does not just reply with a sentence in a chat box; it has to read an entire code repository, call tools, execute commands, observe the returned results, and then revise its plan based on the feedback. Each round of “observe–think–act” leaves a record in the context: long tool-returned text, error stack traces, intermediate artifacts, and rejected approaches. After dozens of interaction rounds, the history trajectory can easily exceed hundreds of thousands of tokens.
As a result, almost all frontier capabilities are pushing in the same direction: the context must be long enough and complete enough. And the classic self-attention structure hits a physical-level obstacle precisely on this point.
2. The nature of the bottleneck: quadratic complexity and linear memory
The attention mechanism of a standard Transformer requires each token in the sequence to interact pairwise with all historical tokens. This brings two unavoidable costs.
First, the amount of computation grows with the square of the sequence length. Let the sequence length be L, the attention dimension be d, and the number of attention heads be h. Then the computation of a single attention layer is roughly 2·h·L²·d multiply-accumulate operations. Taking the product of d and h to be roughly equal to the model’s hidden dimension, when L grows from 8K to 1M, L² is amplified by about ten thousand times, while the overhead of linear layers (FFN, projection matrices) grows only linearly with L. This means that in long-context scenarios, attention quickly goes from being a “small contributor” to a “big contributor,” and can even dominate the compute time of the entire forward and backward passes.
Second, the KV cache balloons linearly with sequence length. During autoregressive decoding, the Key/Value of all past tokens must be kept in GPU memory. Roughly estimating with a common GQA configuration: each token per KV head needs to store 2 (K and V) × number of layers × head_dim elements, and each element takes 2 bytes under BF16. For a model with around 60 layers, 128 head_dim, and 8 KV heads, the KV cache per token is about 240KB, and 1 million tokens is around 240GB—equivalent to the entire memory of a high-end data-center GPU, and that is for a single request. More critically, every decoding step must move the entire KV cache from memory to the compute units, so bandwidth consumption rises linearly with length, while each step actually uses only one new token’s query for computation. Long-context decoding thus falls into a typical “memory wall”: compute sits largely idle, and the bottleneck lands entirely on memory capacity and bandwidth.
Third, there is a second quadratic factor in the real world. Long context is usually not passed in all at once, but fed in segments and inferred over multiple rounds. If every round recomputes attention over the full history, a 50-round long session accumulates compute multiplied by an additional factor related to the number of rounds. This also explains why long context is not just a “model capability” problem but a “service cost” problem—the compute consumed by a single long-context request can be tens or even hundreds of times that of a short request.
For exactly this reason, the core idea in the long-context field is highly consistent: keep a large enough model capacity while pushing down the long-sequence overhead of attention. A typical long-context technical solution usually revolves around four main lines, corresponding to capacity, attention, signal propagation, and training efficiency:
- Parameter capacity: Mixture-of-Experts (MoE). Keep parameter scale via sparse activation so the model has enough “knowledge reserves” without making every token pay the compute cost of all parameters.
- Attention cost: hierarchical compression and sparsification. Separate “seeing the whole” from “seeing the detail,” refining the historical part on demand, so that not every token has to interact pairwise with all history.
- Depth stability: enhanced residual propagation. A deep network of dozens or even hundreds of layers needs a more stable signal channel; otherwise, no matter how clever the attention design, it cannot be trained.
- Training efficiency: matrix-orthogonalization optimizers. Pursue better parameter-update quality and convergence speed under the same compute budget.
Let us break these down one by one.
3. Using MoE to preserve capacity: decoupling parameters from computation
The demand that long context places on model capability is actually “dual”: it needs both solid knowledge and strong instruction-following and long-range reasoning abilities. The former mainly depends on parameter scale, and the latter mainly on training adequacy. If you shrink the model purely to control cost, long-context capability itself is dragged down; but if all parameters are activated, the compute bill is also out of control. The Mixture-of-Experts architecture is aimed precisely at this contradiction.
Basic mechanism. MoE replaces the feed-forward network in the Transformer with multiple parallel “experts.” Each token passes through a gating network and is routed only to the Top-K experts for computation (commonly 1 or 2). The model’s total parameters can be 5 to 10 times or more the activated parameters, but the FLOPs per token grow only with the activated parameters.
Synergy with long context. This is worth emphasizing: the benefit of MoE is not orthogonal to long context but multiplicative. Under long sequences, attention overhead rises in order of magnitude, and the “constant per-token compute” property that MoE brings means that when pushing the context from 128K to 1M, the incremental overhead mainly falls on the attention part rather than being further amplified by parameter computation. In other words, MoE effectively “frees up budget space” for long context.
Four problems that must be handled in engineering:
- Load balancing. The gating network naturally tends to send tokens to a few popular experts, causing some devices to be overloaded while others idle. Common approaches include introducing auxiliary losses (such as load-balancing loss and expert capacity factor), or using lossless routing strategies (such as sequence-level auxiliary loss and auxiliary-loss-free bias adjustment).
- Communication from expert parallelism. Experts are usually spread across different devices, and each layer requires one All-to-All dispatch and one All-to-All combine. The longer the sequence, the larger the number of tokens per batch, and the greater the communication volume, which needs to be overlapped with computation.
- Memory allocation during inference. All expert weights must be resident or quickly swappable. In long-context scenarios, the KV cache already consumes a large amount of memory, so the residency strategy for expert weights (all resident / layered swap-in / quantization) directly affects throughput.
- Shared experts and fine-grained splitting. A common practical compromise is “several shared experts + multiple fine-grained routed experts,” where shared experts absorb general patterns and routed experts handle differentiated modeling. Fine-grained splitting makes routing more flexible while reducing the parameter count per expert and cutting the bubbles caused by mismatched communication granularity.
4. Pushing down attention cost: from “seeing everything” to “seeing hierarchically”
This is the most core and most active direction in long-context technology. An observation repeatedly validated is this: the attention matrix is extremely sparse. Over long sequences, the tokens a query actually needs to attend to often make up only a very small proportion, and a large amount of attention weight can be ignored without noticeably degrading quality. Around this observation, there are mainly three technical paths.
4.1 Sparse attention: local window + global anchors
The most straightforward approach is to let each token attend only to neighbors within a fixed window, plus a few “global anchors” (such as summary tokens for each block and fixed special tokens), replacing O(L²) with O(L·W). The window size W is usually on the order of hundreds to thousands. Its advantage is that it is simple, easy to implement, and nearly lossless for short tasks; its drawback is that it needs to precisely hit “the right positions to look at,” and once critical information falls outside the window, information loss can occur. Common practical improvements include: dynamically selecting blocks to attend to (based on the similarity between the query and block summaries), reusing selection results across layers, and assigning different sparse patterns to different layers (shallow layers lean local, deep layers lean global).
4.2 Hierarchical compression: distilling history into a summary
This is one of the most promising directions right now. The core idea is: for earlier historical tokens, do not directly retain the raw KV, but first compress it into a set of low-dimensional representations (like a “chapter summary”). The query then attends simultaneously to “the raw KV of the local window” and “the global summary KV,” and the two paths are fused. This way, the actual attention objects of each query are W + S (W is the window length, S is the number of summary blocks, usually independent of or weakly correlated with L), and the computation drops from quadratic to approximately linear.
A schematic key path:
def compressed_context_attention(q, k_local, v_local, c_ctx, block_summaries, W):# q: [B, H, S, D] query for the current batch# k_local: [B, H, S, W, D] raw local-window KV for each position# c_ctx: [B, H, C, D] summary KV compressed from history (C << S)# block_summaries: [B, H, S/stride, D] summary vector per history block, used for routing# 1) Local window: computed per position, complexity O(S·W)local_attn = softmax(q @ k_local.transpose(-1, -2) / sqrt(d), dim=-1)local_out = local_attn @ v_local# 2) Summary-block routing: pick only the most relevant blocks, avoiding attention over all history blocks# (similarity scoring -> top-k -> compute attention only on the selected blocks)scores = q @ block_summaries.transpose(-1, -2) / sqrt(d) # [B,H,S,S/stride]topk = scores.topk(k=R, dim=-1).indices # R = number of blocks selected per querysel = gather(c_ctx, topk) # take only the summary KV of the selected blocksctx_attn = softmax(q @ sel.transpose(-1, -2) / sqrt(d), dim=-1)ctx_out = ctx_attn @ sel_v# 3) Fusion: weight the two paths with a gate, which can be generated from q itself and block statisticsgate = sigmoid(W_g @ concat([q.mean(-1), local_out.mean(-1), ctx_out.mean(-1)]))return gate * local_out + (1 - gate) * ctx_out
The engineering value of this structure lies in predictable memory: the KV cache goes from “one copy per token” to “one small window per token + one summary per block,” so capacity scales approximately linearly with sequence length, and the constant factor is greatly reduced. The cost is that it requires an additional compression module and summary-routing computation, and the training itself must learn “how to compress.”
4.3 Working with the inference system: KV cache management
Sparsification and compression must land on memory management to actually save resources; otherwise the theoretical gains are eaten up by fragmentation and redundant computation. A few general practices:
- Paged KV cache. Manage the KV in blocks of fixed size, similar to virtual-memory paging in an operating system, avoiding the waste of reserving the maximum length for each request.
- Prefix reuse. Long-context requests usually share large prefixes (system prompts, code repository content, tool documentation). Prefill the KV for these prefixes only once, and let subsequent requests hit the cache directly.
- Prefill and decode separation. Prefilling for long context is compute-intensive, while decoding is bandwidth-intensive, and mixing the two in the same batch causes them to interfere with each other. Splitting the two phases into different resource pools and selecting hardware for each according to its bottleneck is a standard practice for large-scale serving.
- Quantized and sparsified caches. Quantizing the KV to INT8/FP8, or applying eviction strategies to unimportant tokens, can further reduce memory.
5. Signal propagation in deep networks: deciding whether it can be trained
Long-context solutions usually build the model to dozens of layers or more. Deepening the layers brings not only capacity but also optimization difficulty: gradients repeatedly pass through residual paths during backpropagation and are prone to gradient explosion or vanishing; drift in the activation scale across layers can cause loss spikes in late training or even divergence. The industry has several consensus practices for stable training of deep large models.
Modifying the residual stream. A standard Transformer uses a single residual stream of x + F(x), and information can only travel up and down along one channel. When there are many layers, this channel easily becomes an “information bottleneck” or an unstable link on the “gradient highway.” Enhanced approaches usually introduce multiple parallel residual streams, letting different layers read and write information “across streams,” similar to the idea of hyper-connections: each layer no longer takes values only from the previous layer, but aggregates weighted information from several historical streams and then distributes the output to different streams. This both shortens the effective gradient path and lets features of different depths travel through different channels, reducing mutual interference.
Normalization and initialization. Pre-Norm is already mainstream, but deep networks still need to pay attention to the symmetry of normalization placement, the scaling of initialization variance with depth, and constraints on the output magnitude of residual branches (such as learnable gates or scaling factors).
Engineering safeguards for training stability. These include loss-spike detection and rollback, checkpoint retention, gradient clipping, and numerically more stable attention implementations (such as chunked online softmax). These are not model design, but in large-scale long-context training they are often the deciding factor in “whether the run can finish.”
6. Optimizers: convergence efficiency from matrix orthogonalization
In recent years, the optimizer route based on matrix orthogonalization has drawn wide attention. The intuition comes from an observation: in neural network training, the “condition number” of the parameter-update matrix is often poor, the singular-value distribution is highly uneven, and effective updates concentrate in a few directions while the learning signal in most directions is wasted. Orthogonalization (approximately orthogonalizing the update matrix, e.g., via Newton–Schulz iteration) can make updates more balanced across directions, thus reaching the same loss with fewer steps.
A few points to view rationally:
- The benefit of such optimizers usually shows up as “convergence speed under equal compute” rather than faster per step. They require additional matrix operations (usually several matrix-multiply-level iterations), so the per-step overhead is higher than for naive optimizers.
- They benefit more for architectures with very large hidden dimensions and regular parameter-matrix shapes; for Embedding and LayerNorm parameters, regular optimizers are usually still used.
- In large-scale training, their interplay with learning-rate schedules, warmup, and weight decay needs re-tuning and cannot copy existing recipes directly.
The pragmatic engineering attitude is: treat it as a “tunable option,” run an A/B against existing schemes under the same evaluation system, and judge by “the total FLOPs needed to reach a target loss” or “the evaluation score per unit cost,” rather than by per-step time.
7. At the system level: making the design actually work
Model-structure optimizations solve only half the problem. The other half lies in how to do sequence parallelism well during training, how to schedule during inference, and how to actually teach the model to use long context during post-training. This part often decides the final quality and cost.
Training infrastructure. Long-context training needs to split along the sequence dimension (ring attention / sequence parallelism) to distribute the attention computation of a single sequence across multiple devices; it needs activation recomputation to trade compute for memory; it needs efficient sparse-attention kernels, otherwise the theoretical complexity advantage cannot translate into actual throughput; and it needs reliable fault tolerance and checkpoint resume, because a single step of long-context training takes longer and one failure costs more.
Inference infrastructure. Beyond the KV management mentioned in the previous section, this also includes batch-scheduling strategies (fairness when mixing long and short requests), speculative decoding (whose gains are especially pronounced under long context because the decode bottleneck is bandwidth), and resource pools tiered by context length.
Post-training. This is the most easily underestimated link. A model architecture that supports a million-token context does not mean the model knows how to use a million-token context correctly. Effective practices include:
- Curriculum-style training for long-context capability: gradually expanding from shorter contexts to the target length, avoiding the instability of starting with ultra-long sequences right away;
- Constructing task data that genuinely requires long-range dependencies: cross-file code modification, multi-hop QA over long documents, multi-turn tool-call trajectories—rather than simply concatenating short samples;
- Strengthening long-range retrieval and localization: needle-in-a-haystack probes can only verify “whether it can find it,” and must also verify “whether it can reason over and use it correctly after finding it.”
Observability. Both the cost and quality of long-context services depend heavily on observability. Metrics to collect include: the distribution of context lengths per request, prefix hit rate, KV memory usage and eviction rate, and the segmented timing of prefill and decode. These data are the basis both for cost optimization and for analyzing model behavior. When deploying in the cloud, these capabilities are typically realized by combining general cloud product categories—such as object storage (for checkpoints and datasets), message queues (to decouple inference tasks from state transitions), container platforms (to host training and inference clusters), and logging services with monitoring and alerting (to observe the metrics above)—rather than by any single vendor’s proprietary components.
8. Evaluating quality: don’t just look at “nominal context length”
The most common marketing trap with long context is to treat “supports 1M context” as a capability label. What you should really ask are three questions:
- How long is the effective context length? Run the same set of tasks at the 32K, 128K, and 1M tiers, and observe the score-decay curve. The gap between nominal length and effective length is the real cost.
- Do the gains from long context cover the cost? Compare two routes: “full long context” versus “retrieval augmentation + short context.” On many knowledge-intensive tasks, the latter may cost only a fraction of the former. The truly irreplaceable scenarios for long context are those that require global information interaction and are hard to pre-split for retrieval: whole-repository code understanding, long-document consistency checks, and long-session state maintenance.
- Is the reasoning-chain length effectively controlled? Longer thinking is not always better. The evaluation needs to report both quality and token consumption, and consider introducing thinking-budget control mechanisms.
9. Four practical recommendations for engineering teams
- Quantify first, then choose. Before optimizing, first use real business requests to measure the context-length distribution. If the P99 is only 20K, then the complexity and memory cost paid for 1M context is most likely wasted.
- Design the attention scheme and KV management together. Changing only the model without changing the inference system means the gains get eaten up by system overhead; changing only the system without changing the model means you are limited by quadratic complexity. The two must be coordinated.
- Prioritize investing in post-training and data. The ceiling raised by architecture improvements needs post-training to actually materialize. Most “capability gaps” in long context are essentially insufficient training data.
- Establish an “effect per unit cost” evaluation metric. Put GPU hours, memory usage, average token consumption, and evaluation scores in the same table to compare, rather than looking at a single benchmark score.
10. Summary
The competition in long context is essentially a competition over “how to keep intelligence cheap under long-range dependencies.” Mixture-of-Experts provides capacity; hierarchical compression and sparse attention push the quadratic complexity of attention down to approximately linear; enhanced residual propagation keeps deep networks trainable; matrix-orthogonalization optimizers improve convergence efficiency per unit of compute; and the three infrastructures of training, inference, and post-training decide whether these designs can turn into deliverable services.
One point worth remembering is this: long context is not an arms race where “longer is always better,” but a fine-grained trade-off around cost and benefit. Whoever can let the model truly “make use of” the full context at lower cost holds the core competitive advantage of the next stage.
