Million-Token Context in Open-Source LLMs: Reconstructing Sparse Attention Architecture and How to Evaluate It

    A recent open-source LLM preview release extended native context from 256K to 1M tokens, backed by a systematic architecture overhaul rather than simply larger parameters. Treating that release as a reusable architectural case study—independent of any specific product—this article explains why attention must be restructured as context grows, breaks down the key designs (low-rank KV compression, gated sparse attention with a lightweight indexer, cross-layer index reuse, four-way residual streams, a native draft layer, and routed/shared experts), quantifies the memory and compute gains with code and formulas, and lays out a reproducible evaluation plan spanning long-context retrieval, reasoning, agent tasks, and engineering performance. It is aimed at developers who need to evaluate, deploy, and tune million-token models.

    Introduction: The Architectural Signal Behind a “Preview” Release

    On August 28, 2026, an open-source LLM released a preview of its next-generation large language model, extending native context from 256K tokens to 1M tokens. For developers, what deserves attention is less “the parameters got bigger and the context got longer” and more the systematic architectural reconstruction behind it:

    • Backbone total parameters rose from 295B to 770B, about 2.61x the original;
    • Activated parameters per token rose from 21B to 49B, about 2.33x the original;
    • The attention mechanism switched from conventional grouped-query attention (GQA) to a combination of “low-rank compression + gated sparse selection + cross-layer index reuse”;
    • The residual path expanded from a single channel to four parallel information streams;
    • The native multi-token prediction head was expanded into a billion-parameter-class “draft layer”;
    • The context window expanded from 256K to 1M tokens.

    This combination of “sparser parameters + sparser attention + deeper context” points to a typical scenario: a million-token flagship model aimed at agents and intelligent coding assistants.

    Setting aside specific product attributes, this article treats the release as a reusable architectural sample and develops it in four parts:

    1. Why attention must be restructured once context grows;
    2. A breakdown of the key designs in the new architecture;
    3. Quantifying its memory and compute gains with code and formulas;
    4. A practical evaluation plan for developers.

    Part 1: Background — The Three Costs of an Expanding Context Window

    Parameter scale and context window have never been two independent knobs. As context keeps climbing from hundreds of thousands toward a million tokens, attention shifts from a secondary component into the key factor that dominates both cost and quality. This shows up in three ways.

    (1) KV-cache memory grows linearly with length. During autoregressive decoding, the Keys and Values of historical tokens must stay resident in memory—fixed bytes for every layer, every KV head, and every token. A single request at million-token scale can require tens or even hundreds of GB of cache, directly determining how many concurrent requests a server can handle.

    (2) Attention compute grows with the square of length. Standard attention has time complexity O(L²·d) for a sequence of length L. Full attention in the prefill stage must compute the complete L×L attention matrix, so in long-context settings this single term alone consumes a large amount of compute.

    (3) Length extrapolation degrades as length increases. Even when a model claims support for very long context, real performance is still constrained by positional-encoding extrapolation, the decay of long-range dependencies, and “lost in the middle” effects, so there is often a gap between training length and inference length.

    For this reason, the industry roughly divides context capability into three levels:

    • Can hold: the model accepts very long input without errors;
    • Can finish computing: prefill and decoding complete within acceptable latency at controllable cost;
    • Can use it well: information retrieval is accurate, multi-hop reasoning stays coherent, and instruction-following does not degrade.

    Getting from “can hold” to “can use it well” cannot be achieved by scaling parameters alone—you must push on both the attention structure and memory management at the same time. The reconstruction in this preview can be read as a systematic response built around these three levels of capability.

    Part 2: From GQA to Gated Sparse Attention — The Core Logic of the Reconstruction

    2.1 The Evolution of Three Generations of Attention

    Dimension Multi-Head Attention (MHA) Grouped-Query Attention (GQA) Low-Rank Compression + Sparse Attention (this design)
    Number of KV heads Same as query heads Multiple query heads share one KV head No independent KV heads; caches compressed latent vectors
    Attention computation Each query attends to all historical tokens Each query attends to all historical tokens Each query attends only to the k tokens selected by the indexer (2048 here)
    Main bottleneck as length grows Memory + compute + bandwidth Memory + bandwidth Index-scoring cost + risk of wrong selection

    The first two generations are “lossless but expensive” dense attention. Sparse attention takes a “cheap but lossy” approach: trade a small amount of computation for a large reduction in total compute, then use a gating mechanism to keep information loss under control.

    2.2 Key Mechanism One: Low-Rank Compression

    1. Compress: first project the hidden state into a lower-dimensional latent vector, which becomes the sole main body of the KV cache;
    2. Decode: when needed, project it back up to Keys and Values to restore the attention computation;
    3. Positional component: the components tied to positional encoding are stored separately and are not compressed.

    The direct benefit is a large drop in memory usage. At inference time, the up-projection matrices can be “absorbed” into the query and output projections, avoiding the need to actually reconstruct full Keys and Values for every token.

    2.3 Key Mechanism Two: Gated Sparsity and a Lightweight Indexer

    1. Lightweight indexer: it estimates the relevance between each historical position and the current query using low-dimensional scoring. For example, apply a low-dimensional projection to the query and to the keys, then perform element-wise ReLU and a weighted sum;
    2. Select candidates: from the resulting relevance scores, pick the Top-k positions—here k is 2048, independent of the total context length;
    3. Full attention: the main attention reads and weights full Keys/Values only over the selected positions;
    4. Gated control: a gating quantity adjusts the output strength of sparse attention, so the model can still adaptively retain necessary information even when “looking at only a few positions.”

    Overall, the computation pattern shifts from “treat every token equally” to “consult the index first, then compute what matters.”

    2.4 Key Mechanism Three: Index Caching and Cross-Layer Reuse

    Running the full indexer at every layer would make the indexing overhead itself non-trivial under long context. So this generation runs the full indexer only on a subset of layers: the first layer provides global perception, layers 1, 5, 9, 13 … 77 refresh periodically (21 layers in total), and the remaining 57 layers directly reuse the index computed by other layers.

    This design matters for three reasons:

    • Compute: the indexing cost is amortized across only about a quarter of the layers;
    • Structure: shallow and deep layers do not have the same information needs, and the “global reference” established by early layers can stay stable across several layers without being recomputed layer by layer;
    • Inference: the index can be cached alongside the KV cache, avoiding re-scoring on every decoding step.

    2.5 Quantifying the Gains with Code: Memory, Compute, and Index Overhead

    The following illustrative code shows how to estimate KV-cache memory and attention multiply-accumulate count for long-context inference from the architecture parameters.

    1. # Illustrative code: memory and compute estimation for long-context inference
    2. # Architecture values come from public sources; head_dim is an example estimate
    3. def kv_cache_gb(n_layers, seq_len, d_c, d_rope, batch=1, bytes_per_elem=2):
    4. """Cache (d_c + d_rope) elements per token per layer (2 bytes each in BF16)."""
    5. per_token = (d_c + d_rope) * bytes_per_elem
    6. return n_layers * seq_len * batch * per_token / 1024**3
    7. def attn_macs(n_layers, d_c, seq_len, d_head=128, top_k=2048, idx_layers=21, d_index=64):
    8. """
    9. Approximate multiply-accumulate count per token, per layer:
    10. 1) Main attention runs QK^T and a weighted sum over the top_k selected positions;
    11. 2) Index layers score every historical position once.
    12. """
    13. main = 4 * d_c * top_k # independent of sequence length
    14. index = 4 * d_index * seq_len # index layers only, grows linearly with length
    15. return n_layers * main + idx_layers * index
    16. def mfu_util(gflops, peak_tflops, ratio):
    17. """The share (ratio) of total compute taken by an operator, and its effective
    18. compute under the target MFU."""
    19. effective_tflops = peak_tflops * ratio
    20. return effective_tflops * 1e3 / gflops # unit: seconds
    21. N_LAYERS, D_C, D_ROPE = 78, 512, 64
    22. # 1) KV cache: a single request of 1M tokens
    23. print("KV cache (1M tokens): %.1f GB" % kv_cache_gb(N_LAYERS, 1_000_000, D_C, D_ROPE))
    24. # 2) Per-token attention-related MACs: dense (full) vs. sparse (gated sparse)
    25. for L in (32_000, 128_000, 1_000_000):
    26. dense = attn_macs(N_LAYERS, D_C, L, top_k=L, idx_layers=0)
    27. sparse = attn_macs(N_LAYERS, D_C, L, top_k=2048, idx_layers=21)
    28. print(f"L={L:>9}: dense={dense/1e6:9.1f} M, sparse={sparse/1e6:7.1f} M")
    29. # 3) Latency share of a single-token draft forward (peak 1000 TFLOPS, target MFU 20%)
    30. draft_t = mfu_util(2 * 0.7e9, 1000, 0.20) # draft layer activates 0.7B
    31. main_t = mfu_util(2 * 49e9, 1000, 0.20) * 1.3 # backbone activates 49B; 1.3 is an empirical scaling factor
    32. print(f"draft latency: {draft_t*1e3:.2f} ms")
    33. print(f"main latency: {main_t*1e3:.2f} ms")
    34. print(f"acceptance needed to break even (gamma=4): {1 - draft_t/main_t:.2f}")

    This code yields three quantitative conclusions:

    • Memory: a single 1M-token request needs about 87.6 GB of KV cache alone (78 layers × (512+64) elements per layer·token, in BF16), before counting weights and activations. The core tension of long-context serving lies precisely in per-request memory.
    • Compute: at 1M tokens, the per-token, per-layer “main attention + indexer” multiply-accumulate count is about 1/79 of full attention. The indexer accounts for roughly 97% of it. In other words, with k=2048, the sparse scheme is essentially equivalent to “compute only the index plus 2048 positions.”
    • Draft-layer latency: the draft layer activates only 0.7B, with a single-token latency of about 0.14 ms—roughly one-ninth of the backbone. With a draft step length of 4, speculative decoding only starts to “break even” once the average accepted length exceeds about 2.23 tokens. This explains, from an engineering standpoint, why the native draft layer is worth expanding.

    Part 3: Supporting Designs — Multi-Path Residual Streams, a Native Draft Layer, and Shared Experts

    3.1 Four-Way Residual Streams: Splitting a Single Information Backbone into Parallel Channels

    • Conventional residual: there is only one information channel between layers. As layers deepen, features learned by different layers tend to interfere with each other, and gradients must backpropagate along a single path.
    • Multi-path residual streams: the hidden state consists of multiple parallel components, each with its own input and output projections, and layers “communicate” through a set of learnable mixing coefficients.
    • Simplified formula: with N hidden-state streams at layer l, the j-th stream at layer l+1 can be written as

      h_{l+1,j} = f_j(h_{l,j}) + Σ_{i=1..N} α_{ij} · h_{l,i}

      where α is a learnable connection weight. With N=4, this forms four parallel residual streams.

    The significance of this kind of structure is that:

    1. it lets different information streams “do their own jobs,” easing representational interference in deep networks;
    2. it gives gradients a shorter cross-layer path, improving training stability;
    3. combined with sparse attention, it lets the selected key information pass more smoothly across multiple layers.

    It is worth noting that multi-path residual streams are not free: once the hidden state is copied into multiple streams, activation and KV-cache usage may grow several-fold, and this must be offset by gating and sparsification.

    3.2 The Native Draft Layer: Upgrading the Multi-Token Prediction Head into a “Self-Speculation” Component

    Multi-token prediction (MTP) first appeared as a training auxiliary objective: predict several subsequent tokens while predicting the next one, to strengthen representation learning. At inference time, these extra prediction heads can also act as a draft model, sharing the KV cache with the backbone to enable self-speculative decoding. The workflow is:

    1. the draft layer autoregressively generates several candidate tokens;
    2. the backbone verifies these candidates in a single pass;
    3. accept the longest valid prefix, and obtain a new draft along the way.

    In this preview, the draft layer scaled from about 3.8B to 10B, with 0.7B activated—already a “one-size-smaller MoE model” rather than a lightweight projection head. The more accurate the draft, the higher the average acceptance length; but the larger the draft, the higher its own overhead. The code above shows that a draft layer activating just 0.7B has a per-token latency about one-ninth that of the 49B-activated backbone. This gain is especially pronounced in long-context scenarios, because the backbone must read the massive KV cache on every step, while the draft layer incurs little extra cost by reusing the shared cache.

    3.3 Routed Experts and Shared Experts: Rebalancing Capacity and Compute

    The parameter-configuration changes can be read as follows:

    • Routed experts increased from 192 to 256, with each token still activating 8; the increase is mainly “capacity” rather than “compute”;
    • One additional always-on shared expert was added to carry general knowledge and reduce the learning burden on routed experts.

    Taken together, these changes show that this generation is not positioned as a “smarter general-purpose chat model,” but as a “long-context flagship model” aimed at agents and long-horizon tasks.

    Part 4: A Practical Evaluation Plan — How Developers Can Judge “Is It Good to Use”

    For developers, “how it performs in practice” cannot rest on a few demo cases alone; it should be a reproducible evaluation process.

    4.1 Capability Baseline: Regressing Existing Strengths

    1. General knowledge, math, code, multilingual: first run the existing benchmark suite to confirm the reconstruction caused no capability regression;
    2. Code generation: focus on function-level completion and “executability”—measure by unit-test pass rate rather than text similarity;
    3. Format stability: check carefully whether JSON structures in long outputs and tool-call arguments are complete and valid.

    4.2 Long-Context Track: Four Task Types from “Retrieval” to “Reasoning”

    Task type Example Focus
    Retrieval Locating a function definition, a config entry, or a log segment across 1M tokens Whether hit rate degrades with depth and length
    Aggregation Counting calls across the whole codebase, merging duplicate configs Where sparse attention is most likely to “miss” content
    Reasoning Tracing data flow across files, answering with multiple scattered conditions Whether multi-hop chains break
    Conflict Contradictory clauses within a document Whether it can identify conflicts and cite evidence

    Aggregation tasks deserve special attention: sparse attention is naturally inclined to “look at what matters,” while global statistics require “looking at everything.” For such tasks, it is better to supplement with retrieval augmentation or chunked aggregation than to keep lengthening the context.

    The following illustrative code gives a reusable “multi-needle retrieval” test skeleton: insert several distinct clues at different depths into a long text, then ask about each one and measure the hit rate.

    1. # Illustrative code: multi-needle retrieval (NIAH) test skeleton
    2. import random, string, time, requests
    3. BASE_URL = "http://<host>:<port>" # point to a deployed inference service
    4. def call_model(prompt, max_tokens=256):
    5. """Illustrative interface: actual fields follow the API documentation of the
    6. inference service in use."""
    7. resp = requests.post(BASE_URL + "/chat/completions", timeout=120,
    8. json={"model": "assistant",
    9. "messages": [{"role": "user", "content": prompt}],
    10. "max_tokens": max_tokens})
    11. resp.raise_for_status()
    12. return resp.json()["choices"][0]["message"]["content"]
    13. def build_context(total_chars, filler_unit, needles, depths):
    14. """Insert multiple needles at given depths to build a long context."""
    15. parts = [filler_unit] * (total_chars // len(filler_unit))
    16. for text, d in zip(needles, depths):
    17. pos = min(int(len(parts) * d), len(parts) - 1)
    18. parts[pos] = text
    19. return "".join(parts)
    20. def run_niah(lengths=(32_000, 128_000, 512_000, 1_000_000),
    21. n_needles=4, trials=10):
    22. filler = "This is irrelevant text used to fill the context. "
    23. results = {}
    24. for length in lengths:
    25. hits, total, lat = 0, 0, []
    26. for _ in range(trials):
    27. depths = [random.uniform(0, 1) for _ in range(n_needles)]
    28. needles = [f"Please remember: the code of needle {i} is "
    29. f"{''.join(random.choices(string.ascii_uppercase, k=6))}."
    30. for i in range(n_needles)]
    31. ctx = build_context(length, filler, needles, depths)
    32. for i, d in enumerate(depths):
    33. t0 = time.perf_counter()
    34. out = call_model(ctx + f"\nQuestion: What is the code of needle {i}? Answer with the code only.")
    35. lat.append(time.perf_counter() - t0)
    36. hits += needles[i].split(" is ")[-1].rstrip(".") in out
    37. total += 1
    38. results[length] = {"accuracy": hits / total,
    39. "avg_latency_s": sum(lat) / len(lat)}
    40. return results
    41. if __name__ == "__main__":
    42. for length, r in run_niah().items():
    43. print(f"context={length:>9}: accuracy={r['accuracy']:.0%}, "
    44. f"latency={r['avg_latency_s']:.1f}s")

    In practice, two further variants are worth adding:

    • Distractor resistance: place semantically similar “decoys” with different answers near the needle, to test whether the model is misled;
    • Multi-needle joint queries: ask for the codes of several needles at once, ordered by depth, to test whether the model can assemble the index-selected fragments into a complete answer.

    4.3 Agent Track: Long-Horizon Tasks and Instruction Retention

    1. Long-horizon tasks: simulate a multi-step flow of “read the whole repo → locate the problem → modify code → run tests,” testing cross-tool calls and context retention;
    2. Session drift: insert new instructions that conflict with the original request multiple times during a long conversation, and check whether the model still follows its initial setup;
    3. Structured output: keep generating valid JSON under heavy load, and measure the parse-failure rate.

    4.4 Engineering Performance: Memory, Throughput, and Decoding Speedup

    1. Prefill latency: record first-token latency at different input lengths and plot its growth curve. Full attention typically grows roughly quadratically, while sparse attention is closer to linear;
    2. Decoding speed: watch how KV cache scales with concurrency and length. The index cache can usually be quantified per “layer × token” and occupies far less than the raw KV cache;
    3. Speculative decoding: measure the draft layer’s average acceptance length and end-to-end speedup. If the acceptance length is clearly below the draft step length, reduce the draft step length or evaluate whether the draft layer matches the current task;
    4. Quantization precision: compare KV-cache memory and precision loss at different precisions to choose a balance point for long-context scenarios.

    4.5 Evaluation Matrix: Turning “Experience” into Reproducible Conclusions

    Evaluation dimension Test method Key metrics Warning signals
    Long-context retrieval Multi-needle retrieval + decoy variants Hit rate at different depths and lengths Noticeable drop in the middle
    Long-context reasoning Cross-file tracing, multi-condition QA Answer completeness, citation accuracy Missing some clues
    Agent long-horizon tasks Repo-level modification, tool-chain calls Task success rate, average steps Deviating from the initial instruction mid-way
    Engineering performance Load testing at different lengths First-token latency, throughput, KV memory Latency rising sharply with length
    Decoding speedup Speculative decoding statistics Average acceptance length, end-to-end speedup Acceptance length too low, negative net gain

    Conclusions should come from “averaging multiple runs + fixed random seeds + unified prompts,” not from a single demo.

    Part 5: Practical Recommendations and Risk Warnings

    1. Identify the task type first, then decide whether to use very long context. Retrieval tasks benefit clearly; aggregation tasks are better served by “chunk + summarize” or retrieval augmentation. Very long context should complement retrieval augmentation, not replace it.
    2. Watch KV cache and concurrency. As the code in Section 4.1 shows, a single 1M-token request can consume tens of GB of KV cache. Before going live, estimate the maximum concurrency and context-length combinations against your memory budget, and optimize with paging, quantization, and similar techniques if necessary.
    3. Enable the native draft layer. In inference frameworks that support self-speculative decoding, it can significantly reduce interaction latency in long-context scenarios; if your framework does not support it, consider attaching a small external draft model as a supplement.
    4. Build a regression baseline. Every time you upgrade the model version, re-test with the same long-context evaluation suite to confirm that “the reconstruction did not trade capability for regression.”

    An objective reminder is warranted: while sparse attention lowers compute cost, it also introduces the risk of “wrong index selection.” For tasks that need a global view, keep dense attention or retrieval augmentation as a fallback. The concrete performance of this preview still needs to be judged by reproducible evaluation results—and the methods above are prepared precisely for that.

    Conclusion

    The essence of this preview is not “a bigger model,” but a systematic reconstruction that pushes long context from “can hold” to “can finish computing” and “can use it well.” For developers, understanding the gated sparse attention, index-cache reuse, four-way residual streams, and native draft layer helps both with model selection and with making sounder trade-offs during deployment and tuning. In the end there is only one criterion: did your task, within acceptable latency and cost, get a correct and complete answer?