Million-Token Context in Production: Attention Design Trade-offs and System-Level Co-Design
This article explains why the real constraint on million-token context is KV cache size and bandwidth rather than the nominal context window, walks through the evolution from MHA to MQA, GQA and low-rank latent attention, and examines why some recent models have moved back toward shared (multi-query-like) attention: fused kernels, quantization, sparse/hybrid attention, hardware portability and per-token memory efficiency. It also breaks down why system-level co-design across structure, kernels, parallelism, quantization, scheduling and communication beats single-point optimization, and closes with a practical delivery checklist covering position-encoding extrapolation, long-context continued pretraining, sequence parallelism, layered evaluation suites, context budgeting and prefix-cache economics.
1. First, fix a common intuition: the bottleneck isn’t the window, it’s the KV cache
When people discuss long context, the most frequent misunderstanding is treating it as purely a positional-encoding problem or a training-data problem. What actually decides whether a million-token window can be turned on is the KV cache that is read and written at every single decoding step.
Under autoregressive decoding, the Key/Value of historical tokens must be cached and re-read repeatedly. Its memory footprint scales linearly with sequence length, and every generated token requires a full scan of it. As context grows from tens of thousands to a million tokens, the KV cache quickly overtakes the weights themselves and becomes a double bottleneck: both a memory wall and a bandwidth wall.
A formula that is handy for engineering estimates:
KV_bytes ≈ 2 × L × n_kv × d_head × bytes_per_element × seq_len
where L is the number of layers, n_kv is the number of K/V heads that participate in caching, d_head is the per-head dimension, and bytes_per_element is determined by precision (FP16/BF16/FP8/INT8). Estimated with an illustrative configuration (only to convey the order of magnitude, not representative of any specific model):
| Attention pattern | Cache per token per layer (FP8) | 1M-token context (61 layers) |
|---|---|---|
| Standard multi-head (independent K/V per head) | on the order of a few KB | hundreds of GB to TB scale |
| Multi-query (all heads share one K/V set) | on the order of ~0.25 KB | on the order of ~15 GB |
| Grouped-query (shared within groups) | rises linearly with the number of groups | tens to over a hundred GB |
| Low-rank latent compression | compressed latent vector + decoupled positional part | directly tied to the compression dimension |
The conclusion is unambiguous: the first lever that decides long-context feasibility is “how many bytes must be cached per token”, not the number of layers or the parameter count. This is also why the choice of attention mechanism has been elevated to the same level of importance as the overall model architecture.
The second fact that has to be acknowledged: a nominal window is not an effective window. It is widely observed in the industry that recall of information located in the “middle” of a long context is markedly lower than for the beginning or the end (the so-called lost-in-the-middle effect), and capability degradation on multi-hop reasoning, cross-segment aggregation and long-range constraint satisfaction typically shows up well before the token count is exhausted. In other words, “supports 1M tokens of input” and “completes tasks reliably within 1M tokens” are two different engineering metrics: the former is a memory problem, the latter is a data and evaluation problem.
2. The lineage of attention mechanisms: every step makes the same trade
From original multi-head attention to today’s large models, the evolution of attention structure is essentially the same trade made on the same scale: expressiveness versus per-token cache overhead.
1) Multi-head attention (MHA): every attention head maintains its own Key/Value projections. The strongest expressiveness, the largest cache overhead — memory traded for quality.
2) Multi-query attention (MQA): all query heads share a single set of Key/Value. Cache volume drops directly to 1/(number of heads) of the original, and memory traffic during decoding falls sharply, with substantial decoding-throughput gains for long context and for everything beyond the first token. The cost is limited expressiveness: training stability and quality on some tasks need additional compensation.
3) Grouped-query attention (GQA): a compromise between MQA and MHA, splitting query heads into groups that share K/V within each group. It has been the de facto industrial default for the past few years, because it turns the “cache compression ratio” into a tunable knob.
4) Low-rank latent attention (the MLA line of thinking): instead of caching full Key/Value directly, K/V are jointly compressed into a low-rank latent vector that is cached, and each head’s Key/Value is reconstructed via up-projection at decode time, with the position-dependent part decoupled and handled separately. It hits a very attractive point on the compression-versus-quality curve, and for a while many people believed the direction of open-source architecture evolution had converged and only minor refinements remained.
3. “Dropping low-rank compression and returning to shared attention” is not a technical regression
A recent batch of new models shows a noteworthy pattern: some models have abandoned the low-rank latent compression route in favor of shared (multi-query or approximately multi-query) attention, while other mainstream models in the same period still hold to low-rank compression. This is precisely evidence that the architecture has not converged, and that behind the divergence in routes lie different evaluation criteria.
Low-rank compression looks excellent on paper in terms of compression ratio, but it leaves several costs at the system level:
1) Coupling with fused kernels becomes more complex. Modern inference engines depend heavily on attention fused kernels (folding QKV reads, scoring, softmax and weighted summation into one efficient kernel, together with tiling and online softmax). Low-rank compression requires “decompressing” the up-projection at decode time, which introduces extra matrix multiplications and intermediate tensors. The kernel’s memory access pattern is no longer a simple “linear scan over the KV cache”, the room left for optimization shrinks, and matching the operator implementation to hardware characteristics becomes harder to get fully right.
2) Combining with sparse/linear attention becomes harder. The mainstream direction for reducing the cost of compressed long context is hybrid attention — a few layers keep global attention, most layers use local windows or sparse/linear attention. Such combinations require “the cache semantics of each layer to be clear and independently schedulable.” The cache semantics of shared attention are closer to the textbook definition, and align more easily with the stitching, switching and prefix reuse of hybrid schemes; when low-rank compression is stacked on hybrid attention, aligning the compressed domain with the sparse domain brings additional engineering complexity.
3) Compatibility with quantization differs. KV cache quantization is one of the key levers for cutting long-context cost. Compressed latent vectors and the decompressed Key/Value have very different numerical distributions, and the propagation path for quantization error is longer. The closer the cached content is to “raw K/V”, the more directly controllable the quantization scheme becomes (per-head or per-token scale design, group-wise quantization, outlier handling).
4) Hardware portability and the operator ecosystem. If a model architecture has to run across several generations of accelerators and multiple interconnect topologies, then the more “standard” the operators it relies on and the more regular its memory access patterns, the lower the cost of porting across hardware and of hitting performance targets. Low-rank compression usually comes with custom operators and specific kernel versions, which makes portability more expensive.
5) Trading parameter scale for per-unit memory efficiency. Once shared attention has lowered the cache volume per token, the memory and bandwidth budget it frees up can be invested in model capacity — which is why you can observe the combination “leaner attention, significantly larger parameter count.” This is especially reasonable under sparse mixture-of-experts architectures: total parameters determine capacity, activated parameters determine compute, and expert parallelism determines communication volume; the three can be tuned separately.
The correct framing of the route choice is therefore not “which one is more advanced”, but: once kernels, parallelism, quantization and scheduling are all inside the objective function, which attention structure wins on end-to-end metrics — throughput, time to first token, long-context usability, cost per token.
4. Why system-level co-design is harder than single-point innovation
The claim “system-level co-design is harder than single-point innovation” is worth unpacking, because its root cause is that end-to-end gains are diluted by Amdahl’s law:
Speedup_total = 1 / ((1 - p) + p / s)
Even if you speed up a stage that accounts for a fraction p of end-to-end time by a factor s, the total gain has a clear ceiling. In long-context scenarios, end-to-end time is split across several mutually coupled stages, and any single-point optimization has most of its benefit eaten by the remaining stages.
There are at least five typical coupling surfaces:
① Structure ↔ kernels. Attention shape, head count, head dimension and whether positional encoding is decoupled directly determine the kernel’s tiling strategy and its register/shared-memory footprint. Structure determines the kernel, and the kernel feeds back constraints on the structure.
② Structure ↔ parallelism strategy. Tensor-parallel split dimensions are constrained by head-count divisibility; sequence/context parallelism requires attention to be splittable and reducible across devices; expert parallelism introduces All-to-All communication that has to be overlapped with computation. The split strategy and the model shape constrain each other.
③ Structure ↔ scheduling. Continuous batching, chunked prefill, the resource ratio between prefill and decode (PD disaggregation), and the verification overhead of speculative decoding all have to be co-designed with the model shape. Under long context, prefill accounts for a very large share of cost, and chunked prefill plus priority scheduling directly determine whether long requests drag down overall throughput.
④ Structure ↔ precision. Combining weight/activation/KV-cache quantization requires joint optimization between “accuracy loss” and “bandwidth gain.” The benefit of KV cache quantization scales with sequence length, but the error accumulates over decoding steps.
⑤ Structure ↔ communication. Cross-device All-Reduce, expert-routing All-to-All and the ring reduction of context parallelism all need to be overlapped with computation and compressed to a volume that matches the topology.
A schematic expression of this joint-optimization idea:
# Joint search against an end-to-end objective: not tuning knobs one at a timedef optimize_serving(model_spec, hw_spec, workload_spec):candidates = []for attn_kv_heads in [1, 2, 4, 8]: # structure: cache compression ratiofor quant in ["bf16", "fp8", "fp8+int8_kv"]: # precisionfor parallel in ["tp", "tp+ep", "tp+pp+cp"]:# parallelismfor sched in ["chunked_prefill", "pd_split"]:cost = estimate(kv_cache_gb = estimate_kv(attn_kv_heads, quant, model_spec),comm_time_ms = estimate_comm(parallel, hw_spec),prefill_ms = estimate_prefill(model_spec, hw_spec, sched),decode_ms = estimate_decode(model_spec, hw_spec, quant),target_len = workload_spec.p99_context_len,)if cost.kv_cache_gb <= hw_spec.hbm_budget:candidates.append((cost.score(), cost))return min(candidates)[1]
This also explains why infrastructure engineering capability is becoming part of the model iteration speed itself: a structural change only translates into cost and latency improvements that users can perceive when all four layers — kernel, parallelism, quantization and scheduling — are working together. The concentration of talent toward inference infrastructure and open-source communities reflects essentially the same thing: system capability has become as important as model capability.
5. A checklist for shipping million-token context
If the goal is long context that is genuinely “usable” rather than a number on a spec sheet, each item below needs to be confirmed:
1) Position-encoding extrapolation. When training length and inference length differ, you must rely on scaling and interpolation strategies for positional encoding (position interpolation, NTK-aware scaling, YaRN-style frequency-wise scaling), and you must make the “capability degradation curve inside the extrapolation range” explicit instead of assuming it is lossless.
2) Long-context continued pretraining and the data recipe. Long-context capability comes mainly from “continued pretraining at a larger context length + high-quality long documents + synthetic long-range dependency tasks.” The data side has to cover at once: long-document comprehension, cross-document deduplication and comparison, long-codebase retrieval, and long-conversation state retention. If you only feed long documents without constructing supervision signals that “require answering across long distances”, the model degenerates into a local retriever.
3) Sequence parallelism and the memory ledger. Activations for million-scale sequences also need to be split; you must define the context-parallel degree, the recomputation strategy (activation checkpointing) and the allocation ceiling for the KV cache.
4) Evaluation: from “can it fit” to “does it answer correctly”. Build a layered long-context evaluation suite rather than a single “needle in a haystack” test:
LONG_CONTEXT_SUITE = [# name: (capability dimension, target position, required operation)("retrieval_single", "single-point retrieval", ["head", "middle", "tail"]),("retrieval_multi", "multi-needle retrieval", ["middle"]),("multi_hop", "multi-hop reasoning", ["middle"]),("aggregation", "global aggregation", ["whole document"]),("constraint_follow", "long-range constraints", ["whole document"]),("codebase_edit", "cross-file edit", ["middle"]),]def report(model, suite, lengths=(32_000, 128_000, 512_000, 1_000_000)):table = {}for name, dim, positions in suite:for L in lengths:acc = evaluate(model, name, context_len=L, positions=positions)table[(name, L)] = acc # watch the 2-D degradation surface of position × lengthreturn table
The key is to output a three-dimensional degradation surface of “capability × position × length.” Many schemes look good at 32K and show an obvious cliff at 512K — information that a single averaged metric will hide.
5) Governing useless context. The biggest hidden cost of long context is the “noise you feed in.” Context budget management has to happen at the application layer rather than concatenating without limit:
def build_prompt(messages, budget_tokens, retriever, summarizer):keep, recent = [], []used = 0for m in reversed(messages): # 1. keep recent turns verbatim first (recency)cost = count_tokens(m)if used + cost > budget_tokens * 0.5:breakrecent.append(m); used += costfor chunk in summarize_history(messages[:-len(recent)]): # 2. compress older history into summarieskeep.append(chunk)hits = retriever.search(query, top_k=8, max_tokens=budget_tokens * 0.3) # 3. fill gaps by retrievalreturn assemble(summarizer(keep), hits, list(reversed(recent))) # 4. put key constraints at head and tail
6. The cost equation: cache hit rate is the core of long-context economics
The commercial viability of long context is decided by input-side caching. A common industry practice is that when the same prefix is submitted repeatedly, tokens that hit the cache are billed at a price significantly below those that miss. For workloads with a “fixed prefix + incremental suffix” shape — agent loops, multi-turn long-document Q&A, codebase-level assistance — once the hit rate is stable, input cost differs by an order of magnitude.
Practical points:
- Prioritize prefix stability: put system prompts, tool definitions and knowledge snippets at the very front and avoid volatile content such as timestamps, otherwise the entire cache entry is invalidated.
- Cache granularity and lifecycle: define the minimum unit of caching, the expiration policy and the capacity ceiling; under long context, insufficient capacity lets the hit rate be flushed away by high-frequency requests.
- Coordinate routing with batching: route requests with the same prefix to the same inference replica and align with the continuous-batching scheduler, so that a “logical hit” becomes a “physical hit.”
- Don’t just look at unit price: end to end, measure “cost per successful task,” folding in retry rate, hallucination rate and rework on long outputs. A cheaper-per-token option that needs two retries is not necessarily cheaper.
7. An objective read on the capability gradient, and how to choose
From public developer feedback and slices of technical reports, the current capability gradient can be summarized as follows:
- Reasoning and code: the new generation of open models shows a fairly clear improvement over the previous generation in mathematical reasoning, code generation and debugging, and is close to the closed frontier on most routine tasks; on very long-horizon, multi-tool agentic tasks, however, the stability of instruction following, state consistency and error recovery over long chains still trails the most advanced closed options.
- Hallucination and factuality: improvements in factual hallucination are often more valuable from an engineering standpoint than gains on a single benchmark, because they directly reduce retry and human-verification cost.
- The trade-off between scale and price: rising parameter count usually comes with a rising unit price, but combinations such as cache-hit discounts, routing to distilled small models and difficulty-based routing can push the effective unit cost back into an acceptable range. When selecting a model, do “task-tiered routing”: send simple requests to small models or high-cache-hit paths, and complex requests to the large-model long-context path.
8. What the platform side needs to provide
To make the optimizations above actually land, the cloud side typically needs a full set of general-purpose capabilities rather than a single inference endpoint: an elastic container platform handles replica scaling and topology orchestration for inference; object storage carries long documents and raw multimodal data; message queues and stream processing handle asynchronous orchestration and backpressure for long-running tasks; logging plus monitoring and alerting carry observability for metrics such as token usage, cache hit rate, time to first token and cost per task; and data and model management services handle versioning, evaluation and rollback. Taking Baidu AI Cloud’s general-purpose product portfolio as an example, a closed loop can be built around “data ingestion → elastic inference → observability → cost accounting”, turning long context from a one-off demo into a service that can be operated sustainably. The specific products and their availability differ by region and should be confirmed in the official documentation.
9. Conclusions
- Long context is a systems-engineering problem, not a single parameter. Attention structure determines the cache cost per token, the cache cost determines where the memory wall sits, and the memory wall determines the feasible window.
- The architectural route has not converged. Low-rank compression and shared attention each carry costs: the former has the better nominal compression ratio, while the latter is easier to get right for kernel fusion, quantization, sparse/hybrid mixtures and hardware portability. The selection criterion should be end-to-end metrics, not the novelty of the structure itself.
- Gains from system-level co-design must be accounted for end to end. The ceiling of a single-point optimization is set by its share of total time; joint search across kernel, parallelism, quantization, scheduling and communication is where the main gains come from.
- Replace the nominal window with an effective window for acceptance testing. Use a three-dimensional “capability × position × length” degradation surface instead of single-point retrieval tests.
- The cost core is cache hit rate and task-tiered routing. Prefix stabilization, context budget management and splitting traffic across small and large models bring the economics of long context back into a sustainable range.
