Million-Token Long Context: Engineering Practices for Architecture, Training, and Deployment
This article breaks down the core technology stack that powers million-token (1M) long-context capabilities for large-model training and inference teams. It walks through architecture choices, sparse MoE and attention optimization, cross-layer index reuse, training-inference co-design, inference serving optimizations, and long-horizon agent training, and provides a reusable evaluation checklist and a pre-launch checklist. Rather than tying to any specific product, it answers how to choose, how to train, how to deploy, and where the common mistakes happen.
Why Increasing the Context Window Alone Is Not Enough
Many people assume that long context is just a matter of “enlarging the window.” In reality, the compute and memory cost of standard dense attention grow quadratically with sequence length. When the context expands from 200K tokens to 1M tokens, the size of the naive attention matrix grows by roughly 25x, making it impractical to support through brute-force compute alone.
The mainstream industry approach is a combination of “sparsification + hierarchical reuse”: use sparse MoE to control per-token compute, use dynamic sparse attention to avoid full-sequence dense computation, use cross-layer index reuse to amortize repeated selection overhead, and use inference-serving system optimizations to guarantee throughput and latency. These four elements are tightly coupled and together determine whether long context is “demo-usable” or “engineering-usable.”
Controlling Cost at Ultra-Large Parameter Scales with Sparse MoE
Sparse MoE resolves the tension between capability density and inference cost through “large total parameters, small per-token activated parameters.” As an illustrative example (data for reference only, and should be verified empirically): total parameters can reach hundreds of billions, while the parameters actually activated per token are roughly one-tenth to one-twentieth of the total.
The benefit is that knowledge capacity is carried by a large pool of experts, while per-token inference FLOPs are determined only by the few experts that are routed and activated. Two engineering points deserve attention: first, routing load balancing, to avoid a few experts becoming overloaded while most sit idle; second, the All-to-All communication in expert parallelism, which easily becomes a bandwidth bottleneck in cross-node deployments.
Optimizing Attention Under Million-Token Windows
To keep compute usable within a million-token window, you must avoid dense attention over the full sequence. The idea behind dynamic sparse attention is: use a lightweight scoring/indexing mechanism to select the most relevant subset of Key/Value for each Query, compute attention only over that subset, and layer in a local window and global tokens to balance near-neighbor detail with global summary.
It is worth emphasizing that the code below only expresses the selection idea and is not an implementation ready for million-token use. Real systems typically use block sparsity, LSH/hash-based nearest neighbors, or a dedicated indexer to avoid materializing the full Q×K matrix.
# Illustrative only: candidate index selection for dynamic sparse attention# A real implementation uses block sparsity / LSH / hash-based nearest# neighbors / an indexer to avoid materializing the full Q x K matrixdef sparse_attention(q, k, v, index_fn, top_k, block_size, mask):# 1. A lightweight indexer generates candidates with ~O(N*k) complexity,# rather than O(N^2) for dense attentioncand_idx = index_fn(q, k, top_k=top_k, block_size=block_size) # [B,H,N,top_k]# 2. Gather the candidate subset along the key sequence dim (dim=-2)k_sel = gather(k, cand_idx, dim=-2) # [B,H,N,top_k,d_k]v_sel = gather(v, cand_idx, dim=-2) # [B,H,N,top_k,d_k]# 3. Compute attention only over the candidate subset; the mask filters# out-of-range / invalid indices (set to -inf)scores = einsum('bhnd,bhnmd->bhnm', q, k_sel) / sqrt(d_k) # well-defined scoresscores = scores + maskreturn softmax(scores, dim=-1) @ v_sel # [B,H,N,d_k]
The actual compute is mainly determined by top_k and the local window size: a larger top_k yields higher accuracy but more compute, and the complexity scales directly with top_k and window length. Watch out for the cost of sparse selection itself: if every token redoes an expensive global selection, the compute saved by sparsification is offset by the selection cost — which is exactly the problem that cross-layer index reuse addresses in the next section.
How Cross-Layer Index Reuse Works and What Risks It Carries
In deep networks, many layers share similar attention patterns. Computing the index independently at every layer stacks the overhead multiplicatively. The core idea of cross-layer index reuse is: after completing index selection in some layers, reuse it in subsequent layers to amortize the repeated selection compute.
The key architectural decision is to bake this mechanism into the model during training rather than patching it on at inference time. Example practice typically introduces index reuse at a medium-long sequence training stage such as 128K, letting the model adapt to long-text distributions at the base-model level. This brings two benefits: training and inference attention behavior stay consistent, avoiding the distribution shift of “trained short, inferred long”; and in million-context scenarios, per-token FLOPs can potentially drop.
It should be noted that the reuse benefit is not a universally fixed value. The reduction depends on model scale, training stage, hardware, and index strategy; some similar implementations report reductions on the order of ~3x, but this is a conditional result and should be verified empirically. In practice, pay special attention to:
- Reuse layer scope: which layers can skip indexing and which must compute independently;
- Skip strategy: adjacent layers, interval layers, or sharing across attention-head dimensions;
- Accuracy regression: after introducing reuse, run accuracy regression evaluation on long-text tasks to prevent missing key information.
Coordinating Training and Inference for Speedup
Another bottleneck in long context is slow per-token generation. Two mature techniques can be reused.
Multi-Token Prediction (MTP): beyond standard “next-token prediction,” additionally predict several future tokens, giving the model self-speculation ability. At inference time, combined with speculative decoding, a draft head proposes multiple candidates at once, which the main model then verifies in parallel. The key metric is “average acceptance length”: the closer it is to the draft length, the higher the speedup. Illustrative data (needs conditions noted and verified empirically): under a certain configuration, 7-step MTP raised the average acceptance length from ~4.56 to ~5.47, giving ~20% end-to-end speedup — but the specific numbers depend on the model, data, hardware, context length, and measurement methodology, and cannot be directly extrapolated.
KV cache sharing: share Key/Value caches across layers to reduce memory usage and repeated computation. Under million-level context, the KV cache is the main memory consumer, and the sharing strategy directly determines whether a single GPU can host the long window.
The shared value of both lies in the co-design of training and inference: the multi-step prediction objective optimized during training is precisely amplified by speculative decoding and cache sharing at inference time, rather than each handling its own segment.
What Optimizations Does the Inference Serving System Need?
No matter how strong the model is, if the serving system cannot handle long windows, it still cannot be deployed in practice. Common optimization directions are as follows:
| Optimization Direction | Effect | Applicable Scenario |
|---|---|---|
| Paged KV management | Manage KV in blocks like an OS manages memory, reducing fragmentation and improving batching efficiency | Long windows, high-concurrency batch processing |
| Prefix cache reuse | Cache the KV of common prefixes such as system prompts and long documents, avoiding repeated prefill | Fixed system prompts, multi-turn shared long documents |
| Continuous batching | Dynamically compose batches to improve GPU utilization | Requests with large length variance, online serving |
| Prefill/decode disaggregation | Route compute-intensive prefill and memory-intensive decode to different resource pools | Large-scale online serving, cost-sensitive scenarios |
In enterprise on-premise deployments, such inference serving typically runs on mainstream container platforms, paired with object storage to hold model weights and long documents, and uses logging and monitoring alerts to observe time-to-first-token (TTFT) and throughput — the infrastructure foundation that ensures “usable, stable, and low-cost.” When choosing public cloud or enterprise-owned platforms, you can reuse their general container, storage, and monitoring capabilities to reduce the burden of self-built operations.
How Do Long-Horizon Agents Avoid Going Off Track?
Long context ultimately serves multi-step task execution for agents. To keep an agent “on track, not lazy, and not gaming the system” over a long horizon, you need a reinforcement learning closed loop:
- Context compaction: when history exceeds the window, summarize and compress early conversations and tool calls, preserving key information and discarding redundancy to maintain task continuity;
- Critic-based PPO: use a value network (critic) to estimate state value, combined with proximal policy optimization (PPO) to stabilize policy updates and improve multi-step task success rate;
- Anti-reward-hacking: prevent the agent from executing opportunistic but wrong actions just to inflate its score — a key part of ensuring behavioral authenticity in long-horizon training.
What to Check Before Going Live
Architecture selection, training, inference, and agents form a four-in-one system. It is recommended to verify item by item against the table below:
| Dimension | Check Item |
|---|---|
| Sequence length | Target window (e.g., 200K/1M), the length at which reuse/sparsification is introduced during training |
| Memory budget | KV cache memory estimation, expert parallelism and All-to-All bandwidth |
| Serving metrics | TTFT, throughput, batching strategy, prefix cache hit rate |
| Index budget | top_k, local window size, indexer compute share |
| Agent evaluation | Long-horizon consistency, tool-call accuracy, reward-hacking rate, human spot-check ratio |
| Stability | Accuracy regression evaluation, monitoring alerts, rollback plan |
FAQ
Q: How should I choose a long-context model?
A: First look at the task window and cost budget, prioritize sparse MoE plus dynamic sparse attention, and require that the reuse/sparsification mechanism be introduced early in training to avoid distribution shift.
Q: Where does deployment cost mainly come from?
A: Mainly from KV cache memory and long prefill compute; this can be alleviated with paged KV, prefix caching, and prefill/decode disaggregation.
Q: How do I evaluate long-context capability?
A: Beyond task completion rate, focus on long-horizon consistency, tool-call accuracy, and anti-reward-hacking ability.
Q: What are the common failure causes?
A: Distribution shift from training short but inferring long, excessive sparse-selection overhead, insufficient KV memory, and agent reward hacking.
Long context is not a competition on a single metric, but a four-in-one systems engineering effort spanning architecture, training, inference, and agents. Only by understanding the trade-offs and coupling relationships at each layer can you reliably deliver ultra-long-context capability in real, complex tasks. The performance figures in this article are mostly illustrative; specific numbers should be verified empirically against your own model, hardware, and measurement methodology.
