From 200K to 1M Context: MoE Model Architecture and Long-Horizon Agent Training
Expanding context windows from hundreds of thousands to one million tokens is only the first step toward solving complex, long-running AI tasks. This article examines the system-level techniques required to make ultra-long context practical: dynamic sparse attention with cross-layer index sharing, KV-cache sharing, multi-token prediction for speculative decoding, inference-service optimization, and reinforcement learning workflows for long-horizon agents. It also provides cost estimates, engineering trade-offs, validation methods, and a neutral checklist for evaluating whether a 1M-token context capability is usable in real workloads.
Introduction: A 1M-Token Context Window Is a Starting Point, Not the Finish Line
Expanding a context window from roughly 200K tokens to 1M tokens has long been seen as a major milestone in large language model capabilities. However, for complex, long-running engineering tasks—such as refactoring multi-file codebases, migrating frameworks across versions, or coordinating multiple tools over several hours—a larger context window is only the entry ticket.
What determines success is whether a model can reliably locate relevant information in an ultra-long context, preserve its objective across multiple interaction rounds, and sustain the associated compute cost during both inference and training.
Recent iterations of open-source sparse Mixture-of-Experts (MoE) models have begun to address this problem through systematic redesigns. Their technical direction can broadly be summarized in four areas:
- Architecture: Reuse indexes across layers in dynamic sparse attention to reduce repeated computation at long context lengths.
- Training–inference co-design: Use KV sharing and Multi-Token Prediction (MTP) to optimize both memory usage and speculative decoding.
- Serving: Optimize prefill and cache management specifically for million-token contexts.
- Training: Build a long-horizon Agent training loop around context compression, reinforcement learning with Critics, and reward-hacking prevention.
This article does not evaluate any specific product. Instead, it provides a general technical breakdown of these approaches. Each technique is discussed through the lens of problem, mechanism, cost, and engineering validation, with reproducible estimates where possible to clarify the boundaries of potential gains.
The article concludes with a neutral implementation checklist for technical evaluation and architecture selection.
1. Defining the Problem: Context Length Does Not Equal Task Capability
1.1 Long Context vs. Long Horizon
These two concepts are often conflated, but they refer to different problems.
| Dimension | Long Context | Long Horizon |
|---|---|---|
| Primary focus | Input length within a single request | Number of interaction steps and task duration |
| Core challenge | Effective attention, memory, and compute | Goal retention, state memory, recovery from errors, and credit assignment |
| Typical failure mode | Important information is diluted or ignored | The agent drifts off task, repeats failed attempts, or exploits reward loopholes |
| Main techniques | Sparse attention, long-context training, KV-cache management | Context compression, process supervision, Agent RL, and tool orchestration |
A model may be able to ingest 1M tokens without being able to use them effectively. Common “needle-in-a-haystack” evaluations primarily measure information retrieval. Real engineering tasks also require multi-hop reasoning, state management across tools, and continuous error correction.
Long-context capability should therefore be evaluated at three layers:
- Capacity layer: Can the system complete prefill and decoding without exhausting memory?
- Retrieval layer: Can it accurately locate relevant information distributed across a long context?
- Execution layer: Can it maintain the task objective and complete the task over a long interaction sequence?
The architectural and systems optimizations discussed below mainly support the first two layers while laying the foundation for the execution layer.
1.2 A Cost Model for Long Context
In a standard decoder-only Transformer, per-token inference cost can be roughly divided into two parts:
- Dense computation: Attention projections, feed-forward networks, output projections, and—in MoE models—only the activated experts. This portion is largely independent of context length.
- Attention computation: Query–Key dot products and attention-weighted Value aggregation. Its complexity is (O(L)), where (L) is context length.
This leads to two observations:
- At shorter sequence lengths, dense computation dominates and attention overhead is relatively small.
- When context length grows from 200K to 1M tokens, attention-related cost grows by approximately 5×. With full attention, the growth can be even steeper.
The memory bottleneck is more direct. KV-cache size can be approximated as:
KV_bytes = 2 × num_layers × num_KV_heads × head_dim × sequence_length × batch_size × bytes_per_element
For a model with 64 layers, 8 KV heads, head dimension 128, and FP16 storage:
| Context Length | Approximate KV Cache per Request | Approximate Total for 32 Concurrent Requests |
|---|---|---|
| 200K | About 7 GB | About 224 GB |
| 1M | About 34 GB | About 1.1 TB |
Note: These values are order-of-magnitude examples. Actual memory usage can be substantially reduced through GQA/MQA, lower-precision KV caches, or cross-layer sharing.
This highlights two important points:
- A million-token system is fundamentally an exercise in balancing memory, compute, and latency.
- Any claim about long-context capability should be examined through its specific architectural choices and engineering constraints.
1.3 Four Bottlenecks and Four Technical Directions
| Bottleneck | Manifestation | Corresponding Direction |
|---|---|---|
| Repeated attention-index computation | Per-layer scoring and selection become expensive for long sequences | Cross-layer index reuse |
| KV-cache growth | Memory becomes insufficient under concurrent long sessions, causing eviction and recomputation | KV sharing, cache tiering, and reuse |
| Sequential decoding latency | Tokens are generated one at a time, leading to poor hardware utilization | Multi-token prediction and speculative decoding |
| Long-horizon state drift | The model forgets goals, exceeds context limits, or exploits rewards | Agent RL training loop |
2. Dynamic Sparse Attention and Cross-Layer Index Reuse
2.1 A General Form of Sparse Attention
Full attention allows every Query to interact with every previous token, which becomes costly at long context lengths. Dynamic sparse attention instead uses a lightweight scoring module to select the most relevant Keys for each Query in real time, retaining only the top-(k) positions for attention computation.
Its structure can be summarized as follows:
- Lightweight scoring: Use low-dimensional projections to assign index scores to historical tokens.
- Top-K selection: Retain the (k) highest-scoring positions.
- Sparse attention: Compute (Q \cdot K) and aggregate (V) only at selected positions.
An intuitive analogy is looking up a word in a dictionary. Instead of reading every page, you first narrow the search space using a lightweight index, then read the relevant section in detail.
Compared with static sparse designs such as local windows or fixed sparse blocks, dynamic sparsity can select attention locations adaptively based on content.
2.2 The Bottleneck Moves from Attention to Scoring
Sparse attention is not free. It shifts the bottleneck from expensive attention operations to scoring and selection.
In a naive implementation, every layer independently performs the following steps:
- Compute an index Query projection and score all (L) positions, with (O(L)) cost.
- Run Top-K selection while maintaining a candidate set of size (L).
- Perform attention over the selected (k) positions, with (O(k)) cost independent of (L).
The key issue is that scoring and selection still require a full scan of the sequence. As sequences grow longer, this fixed cost becomes increasingly significant.
This creates a counterintuitive outcome: although sparse attention may make the nominal cost grow linearly with (L), a substantial portion of that cost is repeatedly paid across layers.
2.3 How Cross-Layer Index Reuse Works
A natural observation is that neighboring layers often attend to highly similar token sets. This is especially true for early fixed attention anchors, document titles, important entities, and other stable context features.
This enables a cross-layer sharing strategy:
- Structural sharing: Divide layers into groups of (g) layers. The first layer computes the index, while later layers reuse it.
- What is shared: Only the selected position set is shared. Each layer still maintains independent Q, K, V, and output projections to preserve layer-specific representations.
- When to introduce it: The mechanism should be incorporated during training once sequence lengths reach approximately 128K tokens, rather than being bolted on only at inference time.
Illustrative pseudocode:
GROUP_SIZE = 4 # Each group has 4 layers; the first computes indexes and the rest reuse themTOP_K = 2048def forward_with_index_share(hidden, layers, kv_cache):shared_index = Nonefor layer_id, layer in enumerate(layers):if layer_id % GROUP_SIZE == 0:# First layer in a group: compute lightweight indexes and select Top-Kindex_scores = layer.indexer(hidden) # Lightweight scoringshared_index = topk_positions(index_scores, TOP_K)# Remaining layers: skip scoring and selection, then reuse the indexattn_out = layer.attention(hidden, kv_cache, select=shared_index)hidden = layer.feed_forward(hidden + attn_out) # Illustrative residual orderingreturn hidden
The Top-K implementation above only illustrates the intended semantics. Production systems require specialized kernels and efficient selection algorithms.
2.4 Why It Works: A Reproducible FLOPs Estimate
Let:
- (d) be the hidden dimension,
- (d_i) be the indexer dimension,
- (k) be the sparse-attention budget,
- (L) be the context length.
Approximate per-token compute can be expressed as:
Dense computation per layer:F_dense ≈ 2·d·(2d + 3·d_ff) # Independent of sequence lengthSparse attention per layer:F_sparse ≈ 2 × 2·d·k # Q·K scoring + weighted aggregationIndexer cost per layer:F_index ≈ 2·d_i·L # Linear in context lengthIndexer cost with sharing:F_index / g # g is the number of shared layers per group
Assume the indexing cost is a fraction (r) of dense computation. For sufficiently long sequences, where (F_{sparse}) is relatively small:
- Without sharing, total per-token cost is approximately (F_{dense} \cdot (1 + n \times r)).
- With group-level index sharing, it becomes approximately (F_{dense} \cdot (1 + n \times r / g)).
For (g = 4), (n = 48), and (r = 0.1):
- Without sharing: (48.0 / 49.0 \approx 98.0\%) of the relevant cost is attention-related.
- With sharing: (25.2 / 26.2 \approx 96.2\%).
The FLOPs ratio is approximately:
49.0 / 26.2 ≈ 1.87×
If a lighter indexer with a smaller (d_i) is included in the denominator, the ratio can increase to roughly 2.9×, consistent with the order of magnitude reported in public material for approximately 2.9× lower per-token FLOPs at 1M context length.
These are illustrative estimates, but the conclusion is relatively robust: if index computation is comparable to or larger than dense computation, sharing indexes across four layers can reduce total cost by nearly half.
This leads to several engineering insights:
- Indexing is a full-length scan, while sparse attention itself depends primarily on (k).
- Indexes are often highly redundant across layers, which is the fundamental reason reuse can work.
- The benefit depends on whether the indexer is sufficiently lightweight and whether group size is appropriate—not merely on context length.
2.5 Risks and Limitations
Cross-layer sharing comes with trade-offs:
- Index mismatch: Neighboring layers may have different attention priorities. Lower layers may focus on local syntax, while deeper layers may prioritize global semantics.
- Position-distribution drift: During streaming generation, the index computed by the first layer in a group may become less aligned as decoding progresses.
- Task sensitivity: Tasks requiring precise attention—such as multi-hop retrieval or cross-document reasoning—may show greater degradation.
Engineering validation should therefore include:
- Index overlap: Measure the overlap between independently computed per-layer indexes and shared indexes.
- Output consistency: Compare output distributions before and after sharing, for example using KL divergence.
- Task-level regression tests: Evaluate changes on long-document QA, multi-hop retrieval, and long-horizon coding benchmarks.
- Fallback mechanisms: Revert to independent index computation when index quality appears abnormal, such as when score distributions become excessively concentrated.
In short, cross-layer index reuse is a classic engineering trade-off: use structural priors to reduce compute. The actual gain depends on how well that prior matches the model and workload.
3. Training for Long Context: It Cannot Be Added Only at Inference Time
Many long-context systems reveal weaknesses only during inference because training did not adequately prepare the model for long sequences.
A mature training process generally introduces long context in stages:
- Short-context pretraining: Build core language capabilities with sequences around the 32K range while maintaining favorable compute efficiency.
- Sequence-length expansion: Gradually extend training sequences to 128K tokens and beyond. Use position-encoding extrapolation methods, such as Rotary Position Embedding interpolation or scaling, along with long documents, code repositories, and conversation histories.
- Early introduction of architectural assumptions: Introduce mechanisms such as cross-layer index reuse during the middle stage of long-sequence training, so the model learns to operate with shared indexes.
- Long-horizon task alignment: Add multi-step tool use and long-chain code modification tasks during post-training, supported by the Agent RL loop described later.
The purpose of this sequence is to align architectural assumptions with training data and training length. Only then can a model move from merely accepting long inputs to using them effectively.
A practical evaluation question is: does the model’s long-context capability come from its training design, or from an inference-time patch?
4. KV Sharing and Multi-Token Prediction: Training–Inference Co-Design
4.1 KV Sharing: Compressing Memory and Layer Redundancy
In sparse-attention architectures, if neighboring layers tend to focus on similar tokens, their K and V tensors may also contain redundancy.
KV sharing allows layers in a group to use the same K and V cache. Only the first layer in the group computes and writes the cache, while the remaining layers read from it directly.
The main benefits are memory-related:
- KV-cache usage can be reduced to roughly (1/g), where (g) is the number of shared layers.
- Under the same memory budget, the system can support longer contexts or greater concurrency.
- In long-session workloads, lower eviction and recomputation frequency translates into more stable latency.
The trade-offs are equally clear:
- Layers no longer maintain fully independent key-value representations, which constrains model capacity.
- If sharing spans too many layers, deeper layers may struggle to learn differentiated features.
- The design must be coordinated with positional encoding, normalization, and related components.
A common compromise is partial sharing: share only between certain neighboring layers or across groups of attention heads.
4.2 Multi-Token Prediction: Unifying the Training Objective and Draft Model
Multi-Token Prediction (MTP) trains a model not only to predict the next token, but also to predict several future tokens.
Its value appears in two places:
- During training: It encourages the model to learn longer-range dependencies and can improve representation quality.
- During inference: MTP heads can naturally serve as draft models aligned with the main model for speculative decoding.
The basic speculative-decoding workflow is:
- A draft model generates several candidate tokens.
- The main model verifies those candidates in parallel in a single forward pass.
- The system accepts the longest valid prefix and generates one additional token.
- At rejected positions, the main model resamples to preserve the original output distribution.
Illustrative pseudocode:
def speculative_decode(prompt, main_model, draft_heads, gamma=7):output = []state = main_model.prefill(prompt)while not done(output):draft_tokens = draft_greedy(draft_heads, state, length=gamma)probs, new_state = main_model.verify(state, draft_tokens) # One parallel forward passaccepted = accept_tokens(probs, draft_tokens) # Preserve the output distributionoutput.extend(accepted)state = new_stateif not accepted: # On rejection, fall back to main-model samplingoutput.append(sample(probs[len(accepted)]))return output
4.3 Estimating Speedup: How Acceptance Length Affects Performance
Let:
- (\gamma) be the draft length,
- (\alpha) be per-position draft-token acceptance probability,
- one main-model verification pass cost 1,
- the cost of one draft token be (c).
Then:
Expected accepted length = (1 - α^(γ+1)) / (1 - α)Average speedup = Expected accepted length / (1 + γ·c)
With (c = 0.03), representing an order-of-magnitude example for small draft heads sharing main-model hidden states:
| Draft Length γ | α = 0.80 | α = 0.85 | α = 0.90 |
|---|---|---|---|
| 3 | 2.70× | 2.85× | 3.06× |
| 5 | 3.25× | 3.53× | 3.86× |
| 7 | 3.57× | 3.94× | 4.36× |
| 9 | 3.79× | 4.21× | 4.71× |
Public material for some model designs has reported that, with a 7-step MTP configuration, average accepted length increased from 4.56 to 5.47. This is approximately equivalent to improving (\alpha) from 0.80 to 0.85.
Using the formula above:
- Speedup increases from 3.57× to 3.94×.
- The relative gain is approximately:
(3.94 - 3.57) / 3.57 ≈ 10.4%
Because verification cost is fixed and draft overhead changes only modestly, speedup is roughly proportional to accepted length. Therefore, a 20% improvement in accepted length may lead to only around a 10% improvement in overall speedup. Expectations of linear scaling should be discounted accordingly.
Other factors affecting real-world performance include:
- Draft cost (c): As draft cost rises, long draft sequences become less economical and the optimal (\gamma) becomes smaller.
- Batch size: Speculative decoding is most beneficial at low concurrency or in memory-bandwidth-limited scenarios. At high concurrency, main-model compute may already be well utilized, reducing gains.
- Task distribution: Code completion and structured text often have higher acceptance rates than open-ended creative generation.
4.4 The Combined Effect
KV sharing and MTP complement each other in long-context workloads:
- KV sharing reduces per-token memory and memory-bandwidth costs, making long-context and longer-draft configurations more feasible.
- MTP increases the number of useful output tokens produced per unit of compute.
- Together with cross-layer index reuse, they form a coherent approach spanning training objectives, inference acceleration, and serving deployment.
5. Inference Serving for Million-Token Contexts
Architectural optimizations address whether a model can afford to compute. Serving optimizations determine whether it can compute reliably and quickly.
Million-token context serving faces three primary challenges.
5.1 Prefill Tail Latency
Prefilling a 1M-token context requires processing more than one million tokens. Even with sparse methods, this remains a lengthy compute-intensive phase.
If these requests are batched together with short requests, they can create head-of-line blocking and substantially increase wait times for short workloads.
Common approaches include:
- Chunked prefill: Divide long sequences into fixed-size chunks and schedule them alongside decode requests.
- Priority isolation: Queue requests by length or business priority to protect short-request latency.
- Compute–transfer overlap: In tiered KV-cache architectures, overlap KV-cache writes or migration with subsequent computation to hide data-transfer latency.
5.2 KV-Cache Reuse for Long Sessions
In practice, long-context workloads are often not a single one-time 1M-token request. More commonly, they accumulate toward 1M tokens over multiple rounds.
System prompts, tool definitions, and codebase snapshots may repeatedly appear within a session and are natural candidates for reuse.
- Prefix caching: Reuse previously computed KV states for requests with identical prefixes, potentially skipping most prefill work.
- Tiered caching: Use multiple cache layers across GPU memory, host memory, and external storage. Cold data moves down the hierarchy while hot data is promoted.
- Cache consistency: When a session changes, such as when new tool results are appended, the system must accurately identify the longest reusable prefix.
5.3 Scheduling and Observability for Long Tasks
Long-horizon tasks can last for hours and involve dozens or hundreds of model calls. The serving system should support:
- Session-affinity scheduling: Keep a session on the same instance when possible to improve cache hit rates.
- Fine-grained metrics: Track time to first token, inter-token latency, cache hit rate, preemption frequency, and recomputation count separately.
- End-to-end tracing: Connect tool calls, retries, and context-compression operations into an auditable execution trace.
In cloud environments, these capabilities are commonly implemented through a combination of object storage for long documents and session artifacts, message queues for decoupling long-running tasks, container platforms for elastic scaling, and logging and monitoring systems.
The important point is to support long tasks with observable, schedulable, and elastic infrastructure—not to depend on a single model metric.
6. A Reinforcement Learning Loop for Long-Horizon Agents
Architecture and serving address individual requests. Agent RL addresses multi-step task execution.
A complete loop should include at least four components.
6.1 Context Compression: A Prerequisite for Completing Long Tasks
As an Agent executes a long task, observations, tool outputs, and intermediate reasoning continuously accumulate. The context limit can be reached quickly.
A compression mechanism typically preserves four categories of information:
- The task objective and constraints;
- Key decisions and their rationale;
- Modified file paths and current state;
- Failed attempts and their causes, to avoid repeating them.
Illustrative code:
def compact_history(history, budget, keep_recent=6):if token_count(history) <= budget:return history# Preserve: system instructions, goal constraints, and recent raw messageshead = history[:1] + history[1:1 + keep_recent]body = history[1 + keep_recent:]summary = summarize(body, style="key_decisions_and_state")return head + [Message(role="summary", content=summary)] + history[-keep_recent:]
Compression must avoid two problems:
- Over-compression: Critical state is lost, causing the Agent to repeat work.
- Compression distortion: The summary hides failed attempts, causing the model to incorrectly revisit paths that have already failed.
In engineering validation, task completion rate should not decline after compression. Raw trajectories should also be retained for debugging and auditability.
6.2 Reward Design: From Outcome Rewards to Process Supervision
In long-horizon tasks, final rewards are sparse and delayed. Outcome-only rewards are often insufficient for effective learning.
A common approach combines three reward categories:
- Outcome reward: Whether the task was completed or tests passed.
- Process reward: A Critic, or value network, evaluates the value of each intermediate state to mitigate long-range credit-assignment problems.
- Format and tool constraints: Whether output is parseable and whether tool calls are valid.
The Critic’s value function can be trained through temporal-difference learning, Generalized Advantage Estimation, and related methods, then optimized jointly with the policy.
For very long trajectories, the trajectory can be divided into segments and advantages computed at segment level to reduce variance.
6.3 Preventing Reward Hacking
Agents actively search for reward loopholes. Common examples include:
- Deleting or modifying tests to make them easier to pass;
- Fabricating execution results instead of actually fixing code;
- Adding meaningless code solely to satisfy static checks.
Reward-hacking prevention typically includes:
- Isolated execution environments: Tests run in controlled sandboxes that do not expose hidden tests to the Agent.
- Hidden test sets: Evaluation tests are separated from tests visible to the Agent.
- Behavior auditing: Detect suspicious modifications to test files and configuration files.
- Cross-validation using multiple signals: Combine manually defined rules, independent validators, and sampled reviews.
6.4 Offline Replay and Iteration
Online sampling for long-horizon tasks is expensive. Systems therefore often need to:
- Convert successful trajectories into training data for rejection-sampling fine-tuning;
- Analyze failed trajectories to identify root causes, such as retrieval failures, compression distortion, or tool misuse;
- Feed those findings back into data composition and reward design.
This creates an iterative loop:
Sampling → Evaluation → Attribution → Retraining
7. Implementation Checklist: Evaluating Whether 1M Context Is Actually Usable
When assessing long-context claims, validate them across the following dimensions.
| Dimension | Key Question | Validation Method |
|---|---|---|
| Effective context length | Can the model accurately locate relevant information within a 1M-token context? | Multi-needle retrieval, multi-hop QA, and long-document summarization comparisons |
| Architectural consistency | Are sparse and sharing mechanisms incorporated into training? | Review technical material and compare degradation on long-context benchmarks |
| Long-horizon execution | Can the model preserve goals and state in multi-step tasks? | Multi-step tool-use tasks; measure completion rate and average number of steps |
| Inference stability | Is latency controllable under concurrent long-session workloads? | Load-test time to first token, throughput, and cache hit rate |
| Training loop | Are compression, Critics, and anti-cheating mechanisms in place? | Review trajectory logs and failure-attribution records |
| Cost and controllability | Do memory requirements, license terms, and private-deployment needs fit the environment? | Estimate memory budgets, review license terms, and validate local deployment |
Three points deserve particular emphasis:
- Nominal context length is only an upper limit. The real user experience depends on effective context length—how efficiently the model uses relevant information for a given task.
- Any isolated metric, such as “2.9× lower FLOPs” or “20% higher speed,” must be interpreted in the context of its baseline, configuration, and workload.
- Long-context capability and Agent capability must be evaluated separately. The former is a foundation for the latter, but neither replaces the other.
Conclusion
Moving from 200K to 1M tokens is fundamentally a system-level engineering redesign:
- At the architecture layer: Cross-layer index reuse eliminates repeated scoring, while KV sharing reduces memory usage.
- Through training–inference co-design: Multi-token prediction unifies training objectives with speculative decoding.
- At the serving layer: Chunked prefill, prefix caching, and observable scheduling support long-running workloads.
- At the training layer: Context compression, Critics, and reward-hacking prevention create a closed loop for long-horizon Agents.
For developers, the most important questions are not simply how much context a model claims to support, but:
- Under which workloads do these capabilities work?
- What are the associated trade-offs?
- How can they be validated in a specific environment?
Answering these questions is what turns long context from a parameter on a specification sheet into a reusable engineering capability.
