Million-Token Long-Context Inference: A Practical Optimization Guide
This guide explains how to make million-token long-context inference practical for long-horizon AI tasks. It covers the limitations of chunk-and-summarize pipelines, sparse MoE and dynamic sparse attention, cross-layer index reuse, long-context training strategies, speculative decoding, KV cache paging, prefill-decoding disaggregation, agent-side context compaction, process review, anti-reward-hacking, and a checklist for moving from evaluation to production.
1. Introduction: Why Long-Horizon Tasks Need Million-Level Context
1.1 From single-turn Q&A to long-horizon tasks
How models are deployed is shifting from “ask one question, get one answer” to “give the model a goal and let it work continuously.” These tasks can be grouped under long-horizon tasks. They typically have the following characteristics:
- Many steps: A single task may involve dozens or even hundreds of tool calls.
- Complex state: Intermediate artifacts, error logs, temporary variables, and prior decisions often need to be revisited.
- Long duration: Development, operations, and data-analysis tasks can last tens of minutes or even hours.
- High failure cost: Losing critical information midway can lead to duplicated work, wrong direction, or even destructive actions.
Typical scenarios include:
- Bug fixing that requires understanding an entire codebase;
- Research and analysis over very long documents and meeting transcripts;
- Continuous operations troubleshooting in an AI-driven operations workflow;
- Multi-tool data-analysis pipelines that run across several stages.
1.2 Bottlenecks of the traditional chunked-processing paradigm
When facing very long inputs, the traditional approach is to split the content into chunks, summarize each chunk, process chunks separately, and then merge the results. This works well for single-turn Q&A, but in long-horizon tasks it exposes three problems:
- Irreversible information loss: Compression drops details, but troubleshooting and bug fixing often depend on exact error messages, variable names, and call chains.
- Broken global relationships: Cross-section causality, references, and conflicts are hard to reconstruct after merging.
- Sharp increase in management cost: Chunking strategies, merge logic, and state tracking add complexity and increase the chance of errors.
As a result, a context window that can hold hundreds of thousands or even millions of tokens has become an important foundation for long-horizon tasks.
1.3 The real challenge: from “fitting” to “affordable” and “stable”
Increasing the window size is not difficult. The hard part is making a very long context usable in real engineering environments. There are three main obstacles:
- Compute explosion: Full attention cost grows quadratically with sequence length. If sequence length increases from 32K to 1M, roughly 31 times longer, attention compute can theoretically become about 1,000 times larger.
- Memory expansion: KV cache grows linearly with context length and number of concurrent requests, directly limiting how many long requests a service can handle.
- Stability decay: In very long inputs, critical information can appear anywhere. Models may struggle to recall information from the middle of the context.
Therefore, when evaluating a long-context solution for long-horizon tasks, “the advertised window size” is not enough. The evaluation should consider four dimensions:
| Evaluation dimension | Core question | Key observation points |
|---|---|---|
| Architecture design | Is long-context cost controllable? | Sparse expert activation, attention sparsification, cross-layer index reuse |
| Training strategy | Is long-range capability learned or forced? | Starting stage of long-sequence training, long/short sample ratio, position-encoding extension |
| Inference system | Can it serve long requests stably and cheaply? | Speculative decoding, KV paging and reuse, prefill-decoding separation |
| Agent closed loop | Is long-horizon execution reliable? | Context compaction, process review, prevention of reward hacking |
It is worth noting that long-context capability usually originates from general research. Vendors then build engineering integration and system-level optimization on top of that, which is where differentiation often emerges.
2. Architecture Design: Reshaping the Compute Ledger with Sparsity
2.1 Cost structure of long context: quadratic and linear terms
Transformer forward computation mainly consists of two parts:
- Feed-forward networks: Each token is computed independently, so cost grows linearly with the number of tokens.
- Self-attention: Each token interacts with all historical tokens. For sequence length
L, total attention cost is roughlyO(L²).
When context is short, the feed-forward network dominates. When context becomes long enough, attention quickly overtakes it and becomes the bottleneck.
Therefore, the core goal of long-context optimization is to reduce the quadratic part tied to sequence length, control the linear part related to KV cache, and preserve model expressiveness as much as possible.
2.2 Sparse MoE: total parameters define capacity, activated parameters define compute
A Mixture-of-Experts (MoE) architecture splits the feed-forward network into multiple parallel “expert” subnetworks. A router activates only a small number of experts for each token. This introduces an important property: total parameter count and per-token activated parameter count are decoupled.
A simplified formula for per-token computation is:
FLOPs ≈ 2 × N_active × T + C_attn(L)# N_active: activated parameters per token# T: number of tokens# C_attn(L): attention compute, which grows rapidly with sequence length L
The implication is that model capacity, determined by total parameters, can be large, while per-token compute depends mainly on activated parameters. In other words, under the same compute budget, sparse MoE can hold more knowledge. This is especially valuable for long-horizon tasks that need long-range memory and complex reasoning.
However, MoE also has costs:
- Memory usage does not decrease: All expert weights must remain in memory, or be swapped in and out quickly. Deployment difficulty does not disappear just because activation is sparse.
- Communication overhead is significant: When experts are distributed across multiple devices, routing for each token may trigger cross-device data transfer.
- Load imbalance may occur: Routing distributions often have a long tail, and a few “hot experts” can become bottlenecks.
- Debugging becomes harder: When something fails, it is more difficult to tell whether the problem lies in routing, experts, or attention.
2.3 Dynamic sparse attention: from “attend to everything” to “attend on demand”
A general approach to reducing quadratic attention cost is sparsification:
- Static sparse attention: Use a fixed local window, or group tokens into blocks and attend only to adjacent blocks. This is easy to implement, but the receptive field is limited.
- Dynamic sparse attention: For each query, select the most relevant historical key/value blocks in real time based on similarity with the query.
Dynamic sparse attention better fits long-horizon tasks. For example, when troubleshooting an online incident, the model may need to focus on both:
- The latest error logs, covered by local attention;
- A configuration change introduced long ago, which requires dynamic retrieval.
But dynamic sparsity also has overhead. Selecting relevant content usually requires computing query-key similarity and performing Top-K selection. In a naive implementation, this is repeated independently at every layer:
- Compute grows approximately quadratically with sequence length;
- In a standard Transformer with dozens of layers, the same indexing work is repeated many times.
The result is that although the context can “fit,” the actual compute cost may become unacceptable.
2.4 Cross-layer index reuse: from “query every layer” to “query once per group”
An effective optimization is to let several adjacent layers share the same attention index. The logic can be illustrated with the following pseudocode:
GROUP_SIZE = 4 # Example: every 4 layers form one group; actual value depends on layer count, head count, and attention patterndef layer_attention(x, layer_idx, index_cache, block_size=128, top_k_blocks=1024):if layer_idx % GROUP_SIZE == 0:# First layer in the group: use a lightweight indexer to compute query-key block similarity,# select Top-K blocks, and cache themindex_cache[layer_idx // GROUP_SIZE] = select_topk_blocks(query(x), key(x), block_size=block_size, top_k=top_k_blocks)else:# Other layers in the group: reuse the index from the first layer,# skipping similarity computation and Top-K selectionpassblocks = index_cache[layer_idx // GROUP_SIZE]# Each layer still uses its own Q/K/V, but only computes attention within selected blocksreturn sparse_attention(query(x), key(x), value(x), blocks)
This approach provides three main benefits:
- Indexing changes from “once per layer” to “once per group.” This part of the cost becomes flatter as sequence length grows.
- Repeated filtering is eliminated. The memory-access and sorting overhead caused by Top-K selection is amortized.
- Implementation cost is relatively low. The index can be cached naturally. Inference frameworks only need to pass index handles between layers without changing the core model architecture.
However, reuse does not mean that all layers attend to exactly the same content. Each layer still uses its own query, key, and value parameters. Only the candidate blocks are selected together by the first layer in the group.
Therefore, cross-layer index reuse should not be treated as a temporary inference-side patch. The model should learn cross-layer information flow during training.
Three risks to watch during deployment:
- Group size trade-off: A larger group amortizes indexing cost more effectively, but reduces the attention flexibility available to each layer. A smaller group does the opposite. Different tasks depend on global information to different degrees, so the best group size should be determined through ablation experiments.
- First-layer index quality: If the first layer in a group selects the wrong blocks, the whole group may attend to the wrong content. A possible mitigation is to use lighter and cheaper features for the indexer, and to add fallback mechanisms for the first-layer index, such as always keeping a local window and the initial tokens.
- Consistency between sparse pattern and training: The attention pattern used during inference should match the one used during training. Otherwise, output quality may degrade due to train-inference mismatch.
3. Long-Context Modeling Training: Teaching Models to Use Long Contexts
3.1 Limits of extrapolation: why “train short, infer long” is unreliable
An early common practice was to train models on shorter sequences and then use position-encoding extrapolation, interpolation, or similar techniques to extend inference to longer contexts. This can work reasonably well at medium lengths, but as length increases significantly, several issues may appear:
- Attention distribution drift: Position relationships fall outside the training distribution, making long-distance dependencies harder to model accurately.
- Mid-context recall degradation: When critical information is located in the middle of the context, it can be “diluted” by content at the beginning and end.
- Output degradation: Repetition, truncation, and irrelevant answers become more common.
For long-horizon tasks, any one of these problems can cause a multi-step execution trajectory to fail midway.
3.2 Staged extension: long-range capability should be introduced mid-training
A more reliable approach is to train long-context capability as a core model ability. A typical path includes:
- Main pretraining: Use conventional sequence lengths to build a solid foundation in language and knowledge.
- Length extension during mid-training: Introduce longer sequences before the model has fully converged and while there is still enough learning capacity. This gives the model enough data to learn long-distance dependencies instead of merely “adapting temporarily” during late fine-tuning.
- Mixing long and short samples: Combine long documents, large codebases, and multi-turn conversations in controlled proportions to prevent the model from becoming good only at reading long text while degrading on short inputs.
- Position-encoding extension: For approaches such as Rotary Position Embedding, adjust the base frequency or use interpolation strategies so that position representations cover longer ranges.
- Alignment for structural patterns: If using cross-layer index reuse, dynamic sparsity, or similar structures, the model should be trained so that the attention pattern matches the sparse pattern used during inference.
A common misconception is to treat context-window size as a tunable parameter and assume that increasing the window after training is enough. Window size is only the upper bound. Whether the model can use it reliably depends on training.
3.3 Evaluation focus for long-context tasks
When evaluating long-context capability, it is better to go beyond simple “single-needle retrieval,” where one piece of information is embedded in a long text and the model is asked about it. More realistic evaluations include:
| Evaluation type | Design focus | Capability tested |
|---|---|---|
| Multi-needle retrieval | Embed multiple pieces of information at different positions in a long text, especially in the middle | Stability of recall across the full length |
| Multi-hop reasoning | The answer requires combining multiple distant fragments | Integration of information across sections |
| Distraction robustness | Insert similar but irrelevant content or contradictory statements | Resistance to interference and ability to distinguish evidence |
| Long-horizon task completion | Give a multi-step goal and measure end-to-end success rate | Reliability of long-horizon execution |
It is also useful to track three auxiliary metrics:
- Accuracy decay curve as context length increases;
- Accuracy differences across information positions;
- Output consistency when long-context and short-context requests are mixed.
4. Inference System: Making Million-Token Context Runnable and Affordable
4.1 Speculative decoding: trading drafts for throughput
During decoding, each generated token requires reading all model weights and the KV cache. This is a typical memory-bound scenario. Speculative decoding works by letting a small and fast “draft” source propose several tokens in a row, then letting the main model verify them in parallel. Accepted tokens are kept directly, and generation resumes from the first mismatch.
Two common approaches are:
- Independent draft model: A smaller model from the same family generates candidates, and the main model verifies them. This is flexible, but requires additional memory. If the draft and main model distributions differ too much, the acceptance rate may be low.
- Multi-Token Prediction (MTP): Add several prediction heads to the main model so that one forward pass can predict multiple future tokens. Because the draft and main model share contextual representations, the distribution is more consistent, and acceptance rates are often higher.
Speedup can be approximated as:
Speedup ≈ E[N] / (1 + k × r)# E[N]: average number of tokens accepted per round# k: number of draft steps# r: cost of one draft step relative to one main-model forward pass
From this, several engineering conclusions follow:
- Acceptance length is a core metric. Accepting 2 tokens per round versus 3.5 tokens per round can make a noticeable latency difference.
- Longer drafts are not always better. Every extra draft step adds compute cost. If acceptance rate decays with more steps, the benefit may be canceled out.
- Long context can weaken or change the benefit. A longer context makes the main model slower, and draft overhead becomes relatively smaller, which may seem favorable. However, the attention part does not necessarily scale proportionally with draft steps, so actual gains need load testing.
- Gains may narrow under high concurrency. When batch size is large, GPU compute may already be saturated, reducing the parallel advantage of speculative decoding. Draft strategies should be adjusted dynamically according to workload patterns.
In practice, teams can first measure acceptance rates across token positions, then tune draft steps and acceptance thresholds accordingly.
4.2 KV cache paging: turning memory fragmentation into shareable resources
In long-context inference, KV cache often becomes a bottleneck before model weights do. Its size can be estimated as:
KV bytes = 2 × layers × KV heads × head dimension × sequence length × concurrency × bytes per element
For example, assume:
- 80 layers;
- 8 KV heads;
- head dimension 128;
- FP16 precision, 2 bytes per element.
Ignoring batch size and other factors:
- Each token takes about 0.3 MB;
- A single 32K request takes about 10 GB;
- A single 1M-token request takes about 300 GB.
This means that even one million-token request may require multiple devices to share the KV cache and hierarchical storage to carry it.
Possible optimization directions include:
- Paging management: Split KV cache into fixed-size blocks and allocate them on demand to reduce fragmentation and support prefix sharing.
- Prefix reuse: When multiple requests share the same system prompt, codebase, or document, reuse the computed KV cache and skip repeated prefill.
- Tiered storage: Move cold data to host memory or local storage and load it back when needed.
- Low-precision storage: Store KV cache in FP8 or INT8 formats to reduce memory and bandwidth pressure, while controlling quality loss through precision evaluation.
4.3 Prefill-decoding separation: long requests should not block short requests
The prefill stage of a long-context request must process a large number of tokens at once, which can take seconds or even tens of seconds. If prefill and decoding are mixed on the same instance, short requests that are already generating answers may experience noticeable stalls.
A common approach is prefill-decoding separation (PD separation):
- Prefill nodes focus on long inputs and optimize for compute throughput;
- Decode nodes focus on token-by-token generation and optimize for memory-bandwidth efficiency;
- KV cache is transferred between the two through high-speed links or message queues.
Additional options include:
- Chunked prefill: Split long inputs into smaller chunks and schedule them together with decode requests to prevent long requests from monopolizing resources.
- Timeout and priority scheduling: Give long tasks separate queues and timeout policies to prevent a single extremely long request from degrading the whole service.
4.4 Systems engineering: observability and capacity planning
Operating long-context services is more complex than running normal inference services. At minimum, teams should have:
- Layered monitoring: Track prefill latency, decode latency, KV-cache hit rate, swap-in/swap-out rate, and speculative-decoding acceptance rate separately.
- Capacity planning: Based on peak concurrency and average context length, estimate required memory, device count, and cache hierarchy.
- Degradation policies: When memory is tight, limit concurrency or context length first to protect stable responses for existing requests.
- Traceability: Record the full call chain for each long-horizon task so teams can identify where a problem occurred and which part of the context caused it.
5. Agent Closed Loop: Keeping Models from Drifting in Long-Horizon Tasks
5.1 Context compaction: giving long tasks a rollback-capable memory
Even when the window reaches million-token scale, history in long-horizon tasks can still grow without bound:
- Repeated trial and error;
- Large logs returned by tools;
- Multiple intermediate artifacts.
Stuffing all of this directly into the context leads to high cost and diluted attention. A common approach is context compaction:
- Structured summary: Organize history into fixed fields such as “goal,” “completed,” “current assumptions,” “pending items,” and “key conclusions.”
- Layered retention: Keep the most recent turns in original form and compress older history into summaries.
- On-demand backtracking: Record traceable pointers in the summary, such as file paths or log fragment IDs, and retrieve the original content when details are needed.
Example pseudocode:
def compact_context(messages, budget):# budget: context budget in tokensrecent = messages[-6:] # Keep the most recent turns in original formolder = messages[:-6]summary = {"goal": extract_goal(messages),"done": extract_done(older),"assumptions": extract_assumptions(older),"todos": extract_todos(messages),"pointers": extract_pointers(older), # Traceable anchors: file paths, log IDs, API names, etc.}# If the budget is insufficient, prioritize trimming old raw content while keeping summaries and recent turnsreturn build_prompt(summary, recent)[:budget]
The key to compaction is to preserve the minimum information set required for decision-making:
- The goal cannot be lost;
- Conclusions cannot be lost;
- Critical error lines cannot be lost;
- Conversational filler can be discarded.
5.2 Critic mechanism: process correction in long trajectories
If long-horizon tasks are evaluated only by final results, feedback is sparse and delayed. Introducing a critic mechanism can provide signals for intermediate steps:
- Rule-based review: For coding tasks, check whether compilation and tests pass. For operations tasks, check whether high-risk commands were executed incorrectly.
- Model-based review: Use a model from the same family or a smaller model to score step rationality and evidence sufficiency.
- Self-reflection: At key points, have the model output a self-check list of “current assumptions, evidence, and risks.”
Critic signals can be used in both training and inference:
- In training, they can serve as process rewards for policy optimization.
- In inference, they can trigger retries or rollbacks.
The goal is to detect deviations early in a long trajectory.
5.3 Anti-reward-hacking: preventing “looks done” behavior
When using reinforcement learning to train long-horizon agents, a typical risk is reward hacking: the model learns to obtain “surface-level scores” instead of actually completing the task.
Common patterns include:
- Directly saying “the task is complete” without calling necessary tools;
- Modifying test cases or assertions just to pass tests;
- Repeating meaningless operations to extend the trajectory and gain process rewards;
- Exploiting parsing flaws in evaluation scripts to fabricate results.
Common countermeasures include:
- Outcome rewards first: Use real changes in the environment, such as whether tests truly pass or whether service metrics recover, as the main reward.
- Behavior constraints: Limit high-risk operations and apply allowlists and audits for tool calls.
- Anti-cheating detection: Identify abnormal repetition or abnormal file changes in trajectories.
- Human spot checks: Periodically sample high-scoring trajectories for review.
6. Implementation Practice: A Checklist from Evaluation to Launch
6.1 Requirement assessment: do you really need million-token context?
Not all scenarios need a million-token window. The following criteria can help:
- Must-have scenario: The task depends on global relationships across files or documents and cannot be effectively chunked.
- Optional scenario: Relevant fragments can be retrieved, but the retrieval pipeline is complex and expensive to maintain.
- Unnecessary scenario: The task has little information, or context growth mainly comes from redundant conversation history.
6.2 Capacity and cost estimation
For rough planning, the following formulas can be used. They are for capacity estimation only; actual results should come from load testing.
Per-request memory ≈ model weights (adjusted by precision) + KV cache (estimated by context length)Daily call cost ≈ average input tokens × input price × daily requests+ average output tokens × output price × daily requests
In long-context scenarios, input tokens usually dominate cost. If multiple requests share the same prefix, such as a fixed system prompt or the same codebase, prefix reuse can reduce cost significantly.
6.3 Pre-launch validation checklist
- Effectiveness: Do multi-needle retrieval, multi-hop Q&A, and distraction-robustness tests meet the requirements?
- Stability: Under mixed long-context and short-context workloads, are first-token latency and output quality stable?
- Speculative-decoding benefit: Is the actual acceptance length in line with expectations, and is draft overhead acceptable?
- KV cache: Are hit rate, swap-in/swap-out frequency, and memory usage within safe ranges?
- Agent reliability: What is the end-to-end success rate of long-horizon tasks, and can the agent recover from mid-task errors?
- Security and compliance: Are high-risk operations blocked, and are logs auditable?
- Cost: Is cost per task within budget, and is there a fallback plan for long requests?
6.4 Deployment-form suggestions
- Public cloud hosting: Suitable for rapid validation and elastic scaling. Long-context inference has high requirements for memory and bandwidth. Cloud platforms can provide compute instances with large memory capacity, object storage for long-document corpora, message queues to decouple prefill and decode tasks, and logging/monitoring services to track long-task chains.
- Private deployment: Suitable for data-sensitive scenarios. Key questions include whether a single card can hold the model weights and peak KV cache, and whether multi-card interconnect bandwidth is sufficient for paging and swapping.
- Hybrid form: Use the cloud for elastic long-running tasks and local infrastructure for sensitive data, connected through standardized APIs.
7. Conclusion: Million-Token Context Is Systems Engineering, Not a Single Parameter
Million-token context capability for long-horizon tasks is ultimately the result of architecture, training, inference systems, and agent closed-loop design working together:
- Architecture uses sparse experts and sparse attention to reduce the quadratic cost tied to sequence length.
- Training introduces long sequences during mid-training so the model genuinely learns long-range modeling instead of relying on inference-time extrapolation.
- Inference systems use speculative decoding, KV paging and reuse, and prefill-decoding separation to turn “it can run” into “it runs stably and cheaply.”
- Agent layer uses context compaction, process review, and anti-reward-hacking design to keep long trajectories from drifting or gaming the system.
For developers and enterprises, technology selection should return to three questions:
- Does the task truly need million-token context?
- Is the cost curve of the current architecture controllable?
- Is long-horizon execution reliable and auditable?
Only when these three questions are answered clearly does million-token context move from “a number in a specification table” to “deliverable production capability.”
