Million-Token Context LLMs: Architecture Innovations and Engineering Practices

    This article examines million-token context large language models from both architectural and engineering perspectives. It explains why long-horizon tasks require effective context capability rather than context length alone, and analyzes sparse MoE, dynamic sparse attention, cross-layer index reuse, multi-token prediction, KV cache sharing, serving optimizations, long-horizon agent training, evaluation methods, and practical deployment considerations.

    1. Introduction: Why Context Windows Matter for Long-Horizon Tasks

    Over the past two years, competition among large language models has shifted from improving isolated benchmark capabilities to completing longer, more complex tasks reliably. Long-horizon tasks require progress across many interaction turns, multiple tool calls, or large volumes of source material. Examples include analyzing long documents and comparing chapters, modifying and refactoring large multi-file codebases, planning and executing long-running AI agents, and accumulating information and state across sessions.

    These tasks share a common property: key evidence is scattered across both the timeline of interactions and the space of the corpus. The model must continue reasoning in later steps without forgetting decisions made earlier.

    This leads to an important correction: context length is not the same as effective context capability. Benchmarks repeatedly show the “lost in the middle” effect: when critical information appears in the middle of a long input, recall and reasoning accuracy drop significantly. Other issues include positional bias, instruction drift, and degraded structured outputs under long inputs.

    Therefore, a truly usable long-context system must address three layers at once:

    1. Capability layer: accurate attention allocation and information fidelity over long sequences.
    2. Training layer: long-context adaptation must be built during pretraining or mid-training, not patched at inference time with positional interpolation.
    3. System layer: compute, memory, bandwidth, and scheduling costs must be controlled to a commercially viable level.

    Any solution that only addresses the first item while ignoring the latter two may work in a demo with millions of tokens, but it will not enter production.

    2. Architecture Foundation: Sparse MoE and Dynamic Sparse Attention

    Models based on “large-scale sparse MoE + dynamic sparse attention (DSA)” represent a mainstream engineering paradigm for ultra-long context. These two components solve different problems and should be understood separately.

    2.1 Sparse MoE: Decoupling Capacity from Decode Cost

    The core value of sparse mixture-of-experts is decoupling model capacity from per-token computation. A typical configuration discussed in public technical materials has 744B total parameters and activates about 40B parameters per token, often described as a 744B-A40B model. This ratio implies:

    • Parameter capacity is more than ten times that of a comparable dense model, providing storage for fine-grained knowledge, code, and domain patterns in long text.
    • A single forward pass activates only about 5% of the parameters, so decoding arithmetic intensity and memory access resemble a medium-sized dense model.

    The engineering costs of MoE are often underestimated compared with its architectural benefits:

    • Routing instability: Under long context, token semantic distributions shift with position, which can cause uneven expert loads.
    • Load-balancing loss: Auxiliary losses introduced to prevent a few experts from being overloaded can compete with long-sequence training objectives.
    • Communication bottlenecks: Expert parallelism introduces all-to-all communication, which is highly sensitive to interconnect bandwidth in long-sequence, large-batch settings. This directly determines deployment topology requirements.

    2.2 Dynamic Sparse Attention: From Quadratic Cost to Controlled Sparsity

    Standard full attention has O(n²) time complexity. At 1M tokens, both computation and KV memory access during prefill can become uncontrollable. The general idea of dynamic sparse attention is to compute relevance scores between each query and candidate keys, perform exact attention only on the top-K results, and approximate the rest with global summaries, local windows, or compressed representations.

    Prefill and decode have completely different bottlenecks:

    • Prefill: Compute-intensive. O(n²) computation spikes here and directly determines time to first token (TTFT).
    • Decode: Memory-intensive. The bottleneck is reading the entire KV cache at each step, not floating-point computation.

    As a result, sparsification is most directly beneficial during prefill, while decode optimization must focus on organizing and reusing the KV cache.

    3. Core Innovation 1: Cross-Layer Index Reuse (IndexShare)

    3.1 Identifying the Bottleneck

    The hidden cost of dynamic sparse attention comes from the indexer itself. Every Transformer layer needs its own query-key scoring and top-K selection. The longer the context, the more repeated meta-computation accumulates. At 1M tokens, per-layer independent indexing causes two problems: compute cost grows linearly across layers, and intermediate index tensors increase memory pressure. As a result, a model may support long context on paper but cannot serve it at reasonable cost.

    3.2 Mechanism: Reuse One Index Every Four Layers

    Cross-layer index reuse, corresponding to ideas such as IndexShare or IndexCache in public discussions, uses a “reuse every four layers” strategy. In every four consecutive Transformer layers, only the first layer runs a lightweight indexer and produces top-K attention indices. The next three layers reuse that index set, skipping repeated scoring and selection while keeping exact sparse attention.

    1. # Pseudocode for cross-layer index reuse: every GROUP layers share one Top-K index
    2. GROUP = 4 # Index reuse interval
    3. shared_topk = None # Shape: [batch, heads, n_q, K]
    4. for idx, layer in enumerate(model.layers):
    5. q, k, v = layer.project(hidden)
    6. if idx % GROUP == 0:
    7. # Only the first layer runs the lightweight indexer and Top-K selection
    8. index_score = layer.indexer(q, k) # Low-rank/low-cost scoring, much cheaper than full attention
    9. shared_topk = index_score.topk(dim=-1, k=layer.sparse_k)
    10. attn = sparse_attention(q, k, v, index=shared_topk, exact=True)
    11. else:
    12. # Later layers reuse the index and only perform sparse attention
    13. attn = sparse_attention(q, k, v, index=shared_topk, exact=True)
    14. hidden = layer.ffn(layer.norm(hidden + attn))
    15. # KV sharing across layers (KVShare) can further reduce memory access during decode

    3.3 Benefits and Costs

    The benefit is that indexing and selection costs are amortized across roughly one quarter of the layers. Public technical discussions cite a reference figure: at 1M context, per-token FLOPs drop by about 2.9x compared with per-layer independent indexing. This level of reduction is the dividing line between moving ultra-long context from unusable to commercially viable.

    However, index reuse introduces new sources of approximation error:

    • Index staleness: Sharing one sparse pattern across four adjacent layers reduces the model’s freedom to adjust attention targets layer by layer. Tasks that require progressive retrieval across layers, such as multi-hop reasoning, may lose accuracy.
    • Error accumulation: If the first layer makes a wrong top-K selection, the next three layers inherit the bias.
    • Hyperparameter sensitivity: The reuse interval, sparsity K, and indexer dimension must be tuned separately for each model scale. There is no universal optimum.

    Common engineering mitigations include keeping some layers in full attention or fine-grained indexing, using different K values across attention heads, and dynamically increasing sparsity in regions with dense long-range dependencies.

    3.4 Training Timing: A Training Problem, Not an Inference Trick

    The most important engineering judgment is that IndexShare-like mechanisms must be introduced during mid-training. Public discussions indicate that the mechanism participates from the 128K sequence-length training stage, rather than being enabled ad hoc at inference time.

    The reason is straightforward. If sparse attention patterns are applied only during inference, the model has never seen an input distribution where only part of the context is visible. Attention heads may degenerate into over-reliance on local windows, systematically weakening long-range dependency ability. In contrast, if index-sharing structures are fixed during training, the model learns to place critical evidence where indexed positions can capture it and internalizes cross-layer retrieval division into its weights. This is the dividing line for whether effective context utilization is realized.

    4. Core Innovation 2: Multi-Token Prediction and KV Sharing Across Training and Inference

    4.1 The Dual Role of MTP

    Multi-token prediction (MTP) provides an auxiliary objective during training, forcing hidden states to encode longer-horizon lookahead information. During inference, its prediction heads can naturally serve as draft models for speculative decoding. They propose multiple candidate tokens at low cost, and the main model verifies them in one parallel pass.

    The key property of speculative decoding is that the output distribution remains unchanged. The acceptance or rejection criteria during verification ensure that the final result is equivalent to token-by-token sampling from the original model. It affects speed, not quality. The prerequisite is a sufficiently high draft acceptance rate; otherwise, wasted computation can slow inference down.

    Public discussions provide one reference dataset: after configuring 7-step MTP, average acceptance length improved from 4.56 to 5.47, and end-to-end performance improved by about 20%. In ultra-long-context scenarios, MTP benefits are further amplified because the decode bottleneck is already KV cache memory access. Serving multiple tokens in one forward pass significantly amortizes that memory cost.

    4.2 KVShare and the Memory Wall

    KV sharing addresses a different problem. As layers increase, KV cache size grows linearly, but decode must read the full cache at every step. Sharing strategies—such as cross-layer shared projections and grouped-query attention—can compress KV size several-fold while keeping precision loss controllable.

    A rough estimate shows how severe the problem is:

    1. # KV cache size estimate: 2(K+V) * layers * kv_heads * head_dim * dtype_bytes
    2. per_token = 2 * 80 * 8 * 128 * 2 # 80 layers, 8 KV heads, head_dim=128, FP16
    3. print(f"Per token: {per_token/1024:.0f} KiB") # -> 320 KiB
    4. print(f"1M-token request: {per_token * 1024**2 / 1024**3:.0f} GiB") # -> 320 GiB

    A single 1M-token request requires hundreds of GiB of memory for KV cache alone, far beyond the capacity of any single GPU. This proves mathematically that million-token context is not a modeling problem but a systems engineering problem. It must be solved jointly through KV sharing, paged management, hierarchical storage, and compression.

    5. Core Innovation 3: Serving a 1M-Token Context Model

    Deploying a long-context model at a level that can handle real traffic usually requires several layers of optimization:

    1. Paged KV management: Manage KV cache in fixed-size blocks to eliminate reservation waste and fragmentation, making memory utilization predictable.
    2. Prefix sharing: KV for repeated prefixes—system prompts, long documents, codebase snapshots—can be computed once and reused across requests. This is especially valuable in long-document scenarios.
    3. Chunked prefill: Split long prefills into smaller chunks and interleave them with decode requests, preventing one long request from monopolizing devices and causing TTFT jitter for other requests.
    4. Prefill/decode disaggregation: The two phases have very different resource profiles. Separating them allows independent scaling and independent hardware selection, and is usually mandatory for long-context serving.
    5. Hierarchical KV storage and offloading: Move cold KV to main memory or shared storage such as object storage, and coordinate queries through message queues and schedulers.
    6. Context compaction: Compress history that exceeds budget through summarization, preserving task state while discarding redundant raw text.

    Observability should go beyond throughput. It should at least include: TTFT and TPOT percentiles, cost per million tokens, KV hit rate, preemption and queueing time, and context effectiveness rate (the proportion of evidence in a long input that is actually referenced).

    6. Core Innovation 4: Reinforcement Learning Loops for Long-Horizon Agents

    Ultra-long context creates business value only when combined with agent capabilities. Training long-horizon agents is difficult because of sparse rewards, hard credit assignment, and metrics that can be gamed.

    Compaction as a training target. Historical compression strategies—sliding windows, hierarchical summaries, and structured memory—often lose critical state if implemented only with heuristic rules. Including the compression strategy itself in training allows the model to learn what to remember and what to discard, which is key to long-horizon capability.

    Critic PPO and process rewards. Sparse terminal rewards provide almost no useful gradient in multi-step tasks. Adding a critic for value estimation can decompose terminal returns into intermediate steps. Process rewards—such as whether the agent called the right tool or verified a hypothesis—can significantly improve long-range credit assignment.

    The need for anti-hacking mechanisms. Once a reward signal can be optimized, reward hacking is inevitable. Models may learn to inflate token counts with verbose outputs, avoid verification with vague conclusions, or reuse template answers to pass automatic scoring. Common defenses include prioritizing executable verification—compilation, unit tests, format checks—over model scoring, maintaining adversarial evaluation sets that differ from the training distribution, and regularly auditing the reward model itself.

    7. Evaluation: How to Objectively Assess These Models

    A “three-dimensional plus sub-item” framework is more useful than a single leaderboard score.

    Quality dimension (by sub-item, not total score)

    • Retrieval precision: single-fact recall, cross-paragraph stitching, and negative questions where the information does not exist in the material.
    • Aggregative reasoning: cross-section summaries, multi-hop reasoning, and global statistics.
    • Long-range consistency: whether constraints are forgotten across multiple turns and how much instruction drift occurs.
    • Structured output: parseability of JSON or function calls under long inputs.

    Cost dimension: Bucket requests by input length and report p50/p95 latency and per-token cost for each bucket. Cost curves for long-context models are usually not linear, so bucketed data is more decision-relevant than averages.

    Effectiveness dimension: Apply perturbation tests—insert irrelevant distractor passages, move key information to different positions, and paraphrase without changing meaning. If metrics fluctuate sharply, the model is relying on surface patterns rather than semantic understanding.

    Evaluation pitfalls include data contamination, prompt sensitivity, and fixed-position bias. The rank difference of the same model across evaluation frameworks can exceed its real capability difference.

    Choosing between long context and RAG: Long context and retrieval-augmented generation are not substitutes. Retrieval is cheaper, easier to update, and traceable, but it is limited by chunking granularity and recall quality and struggles with global aggregation. Long context offers higher fidelity and global reasoning, but cost and latency rise with length. A reasonable production system is usually layered and hybrid: first compress candidates through retrieval, then apply long context where global judgment is required.

    8. Practical Deployment Recommendations

    • Open-source licensing and private deployment: Permissive licenses, such as commonly used one-time permissive open-source licenses, support commercial use and secondary fine-tuning. However, teams should verify compliance obligations themselves and stress-test hardware interconnect topology, quantization schemes, and KV quantization precision.
    • Tiered routing: Send short requests to lightweight models and long requests to long-context models, selecting dynamically based on task complexity. This is the most direct way to control cost.
    • Context budget management: Explicitly set token budgets for each task and prefer compression over truncation. The policy should clearly preserve task state and final goals.
    • Capability validation over metric worship: Run perturbation tests on a self-built business dataset before deciding whether to invest in large-scale deployment.

    9. Conclusion

    The jump from 200K to 1M context is not addition in parameter scale. It is multiplication of four engineering systems: cross-layer index reuse reduces the repeated cost of index meta-computation; MTP and KV sharing connect training objectives with inference acceleration; a dedicated serving system pushes compute cost down to a serviceable level; and a long-horizon reinforcement learning loop turns “can read long text” into “can complete long tasks.”

    The implication for developers is clear. When evaluating a million-token context model, ask three questions: Was sparse structure internalized during training? Are system-level costs predictable? What is the effective completion rate for long-horizon tasks? Million-token context has real production value only when all three hold.