Understanding Million-Token Context Sparse LLM Architecture and Evaluation Methods

    This article breaks down a new class of ultra-large sparse models built for million-token context, using a reference model (770B total parameters, 49B active per token, 1M context) as an anchor. It explains why a larger parameter count does not necessarily mean higher inference cost, how MLA, gated sparse attention, and layered index caching control memory and latency at long context, and how multi-way residual streams, a native draft layer for speculative decoding, and BF16/MXFP8 precision choices fit together. It also covers how to avoid benchmark contamination with a private evaluation matrix, how to estimate real serving cost, and how to roll out long-context models safely. Developers get a measurement-first framework: build private eval sets, model your own length distribution and cache hit rate, and track acceptance rate, truncation rate, and latency before committing.

    1. How to Read 770B, 49B, and 1M Context

    According to public information, a recently released next-generation open-source ultra-large sparse model can be treated as a “reference model.” Its core specifications can be summarized as follows:

    Metric Previous Gen Reference Model Change
    Total backbone params 295B 770B ~2.6x
    Active params per token 21B 49B ~2.3x
    Context length 256K 1M ~4x, near an order of magnitude
    Layers 80 78 Slightly fewer
    Hidden dimension 4096 6144 ~1.5x
    Routed experts 192 256 +64
    Activated experts per token Top-8 Top-8 + 1 shared expert Adds a shared expert
    Attention GQA MLA + gated sparse attention + index cache Architecture overhaul
    Residual path Standard residual 4-way multi-path residual stream Multi-path information flow
    Multi-token prediction 3.8B, 0.7B active 10B, 0.7B active Larger native draft layer
    Open precision BF16 BF16, MXFP8 Adds block-level FP8

    If you only look at the parameter count, 770B can look like a step backward in cost terms. But the more important change is not the parameter count itself — it is that three things happen at the same time:

    1. Sparsification goes deeper. Active parameters are about 6.4% of total parameters, a sparsity of roughly 15.7x, keeping per-token compute in the tens-of-billions range.
    2. Context grows from 256K to 1M. Context length increases about 4x, approaching an order of magnitude.
    3. Attention is redesigned. With a longer context, sticking with standard attention would sharply increase both KV Cache pressure and memory-access (memory bandwidth) demand. This calls for low-rank compression, sparse selection, and index caching working together.

    You can understand this generation of models as a piece of combinatorial engineering: sparsity controls per-token compute, the attention redesign controls the cost of long context, and multi-path residual streams plus a native draft layer improve training and decoding efficiency.

    2. Why 770B Is Not Necessarily More Expensive Than 295B

    In a sparse mixture-of-experts (MoE) model, total parameters determine capacity while active parameters determine per-token compute. The two can be decoupled.

    The reference model has 770B total parameters and 49B active parameters, a sparsity of about 15.7x. This means capacity has grown, but the number of experts each token actually passes through has not scaled up proportionally. A rough estimate:

    • Each routed expert averages about 3B parameters;
    • The 8 experts activated per token total about 24B;
    • The remaining ~25B comes from attention, embeddings, shared paths, and so on.

    So “more parameters” does not necessarily mean “slower inference.” But “unchanged compute” does not mean “unchanged cost” either. When deploying, watch three types of cost:

    1. Per-token compute: primarily determined by active parameters.
    2. Deployment barrier: primarily determined by total parameters, precision, GPU memory (VRAM), and parallelism strategy.
    3. Serving cost: determined by API pricing, cache hit rate, input-length distribution, and output length.

    Take GPU memory as an example. Under BF16 weights, 770B parameters need about 1.54TB of weight memory; even estimating at FP8, it is still about 770GB. You cannot simply conclude whether “8 cards on a single machine are enough.” You need to account for per-card VRAM, parallelism strategy, activation footprint, and communication buffers together. In practice, deployment usually needs a combination of tensor parallelism, expert parallelism, and pipeline parallelism.

    In addition, increasing experts from 192 to 256 and introducing a shared expert also changes routing load. More routed experts means a larger selection space per token, but the risk of expert collapse can also rise. The load-balancing loss can be written as:

    1. L_bal = N_exp × Σ f_i × p_i

    where f_i is the actual selection ratio of expert i, and p_i is its routing probability. The value of a shared expert is to concentrate general-purpose capability in a set of always-active parameters, letting the routed experts focus more on differentiated knowledge.

    3. How Does a 1M Context Control Memory and Latency?

    3.1 Why Standard GQA Is Not Enough

    Grouped-query attention (GQA) reduces KV Cache by letting multiple query heads share key/value heads. But as context grows, the pressure is still significant: attention computation grows quadratically with sequence length, KV Cache grows linearly with sequence length, and per-step decode memory access grows with the number of already-generated tokens.

    At a 1M context, decoding each new token requires reading a large amount of historical positions. Here, the bottleneck is often not just compute, but memory capacity and memory bandwidth.

    The reference model responds on three levels:

    1. Multi-head latent attention (MLA): cache a low-dimensional latent vector instead of the full K and V for each head.
    2. Gated sparse attention: each query attends only to a small number of selected historical positions, with a gate value controlling the contribution.
    3. Layered index cache: only some layers run a lightweight indexer, while the rest reuse existing indices.

    3.2 How the Index Cache Is Reused

    According to public information or the technical report, only 21 of the reference model’s 78 layers run a full lightweight indexer; the remaining 57 layers reuse existing indices. Schematically:

    1. Full indexer: layers 0, 1, 5, 9, 13, ..., 77 (21 layers total)
    2. Shared index: the other 57 layers reuse existing indices

    This design relies on one assumption: which historical positions are relevant is usually determined at the shallow-to-middle layers, so adjacent layers — or layers close together — can share the selection results.

    The benefit is direct: the indexer must compute lightweight relevance scores for queries and historical positions. Running it on only 21 layers instead of 78 can cut this overhead down to about 27%.

    The risk also exists: the larger the reuse span, the more pronounced the drift in deeper representations. If index selection is inaccurate, deeper layers may miss critical information. This must be validated with multi-hop, aggregation, and long-document QA tasks.

    3.3 How Big Is the Memory and Access Gap?

    Let the sequence length be L, hidden dimension d, number of attention heads h, and per-head dimension d_h. The key differences among structures are:

    • Standard attention: the cache grows linearly with sequence length, but each token’s cache holds many elements.
    • GQA: reduces the cache by cutting the number of KV heads.
    • MLA: caches a low-dimensional latent vector c_KV with dimension d_c.
    • Sparse attention: the final attention stage reads only a few positions, so compute and memory access scale roughly with the selection count k.

    The following uses an easy-to-follow example for estimation. To avoid confusion in accounting, this adopts a “single-side cache accounting”: it counts only the equivalent cache of either K or V, normalized according to common engineering implementations. If K and V are both stored independently, the absolute numbers go up, but the relative relationship under the same accounting stays the same.

    Assume h=64, d_h=128, d_c=512, 78 layers, BF16.

    1. Standard MHA accounting
    Per layer, per token, cache elements ≈ h × d_h = 64 × 128 = 8192.
    Under BF16, about 16KB. Across 78 layers, about 1.25MB/token. For 1M tokens, about 1.25TB; accounting for implementation overhead, roughly 1.5TB.

    2. GQA compressed to 1/4
    Per layer, per token, cache elements ≈ 2048.
    Under BF16, about 4KB. Across 78 layers, about 0.312MB/token. For 1M tokens, about 312GB; accounting for overhead, roughly 0.375TB.

    3. MLA with d_c=512
    Per layer, per token, cache elements ≈ 512.
    Under BF16, about 1KB. Across 78 layers, about 78KB/token. For 1M tokens, about 78GB; accounting for overhead, roughly 96GB.
    Relative to GQA: 512 / 2048 = 1/4, i.e., about another 4x compression.
    Relative to single-side MHA: 512 / 8192 = 1/16, i.e., about 16x compression.

    4. Sparse attention
    If each query selects only k=2048 historical positions, the single-step read volume is about:

    1. 2048 × 512 × 78 × 2 bytes ≈ 164MB

    This is far smaller than a full scan of a million historical positions. However, the lightweight indexer may still need to scan all positions, so the overall benefit depends on whether the indexer implementation is lightweight enough.

    The conclusion is clear: the bottleneck in long context is first memory capacity and memory bandwidth, and only then raw compute.

    1. # Conceptual illustration: a real implementation skips unselected blocks
    2. # rather than computing over the full sequence and then masking.
    3. def sparse_attention_concept(q, kv_latent, block_ids, topk_blocks=16, block_size=128):
    4. # q: [batch, heads, d]
    5. # kv_latent: [seq_len, d_c]
    6. # block_ids: [seq_len]
    7. n_blocks = (len(kv_latent) + block_size - 1) // block_size
    8. # indexer_score must be provided by the model or the engineering layer
    9. block_scores = indexer_score(q, kv_latent, block_size)
    10. selected = block_scores.topk(topk_blocks).indices
    11. valid_block = block_ids.unsqueeze(-1).eq(selected.unsqueeze(-2)).any(dim=-1)
    12. scores = (q.unsqueeze(-2) @ kv_latent.transpose(-1, -2)) / (q.shape[-1] ** 0.5)
    13. scores = scores.masked_fill(~valid_block, float('-inf'))
    14. weights = scores.softmax(dim=-1)
    15. return weights @ kv_latent

    Note that sparse attention strictly zeros out the weights of some positions, so the output is no longer a convex combination over the full sequence. This affects gradient propagation and numerical stability, so long-context fine-tuning requires recalibrating the learning rate and numerical precision.

    4. What Problem Does the Multi-Path Residual Stream Solve?

    The reference model changes the standard residual into 4 parallel residual streams. Multi-path residual methods, represented by hyper-connections, let different layers read from and write to different residual channels.

    This can bring several benefits:

    1. More direct gradient paths. Deep gradients can bypass some nonlinear transforms, alleviating the difficulty of deep-network optimization.
    2. Information separation. Different channels can carry different information — for example, raw token representations, cross-layer aggregation results, and task-relevant intermediate states.
    3. Synergy with sparsification. The input to a routed expert depends on how the residual streams are mixed, so the channel structure affects expert load distribution, and routing entropy must be monitored.

    The cost is more explicit dependencies. Parallel splitting, operator fusion, activation checkpointing, and custom-operator complexity all increase, and the inference framework needs dedicated support.

    5. How Does Speculative Decoding Get Its Speedup?

    The reference model provides a native draft layer with 10B total parameters and 0.7B active parameters, used for multi-token prediction and speculative decoding.

    The basic flow is:

    1. A small model guesses k tokens in a row;
    2. The large model verifies them in parallel in one pass;
    3. The longest matching prefix is accepted.

    If the acceptance rate of drafting k tokens per step is α, and the cost of a draft step relative to a target-model decode step is c, the ideal speedup can be approximated as:

    1. S(k) = (1 - α^(k+1)) / ((1 - α) × (k × c + 1))

    Here c=0.1 means a draft step costs only one-tenth the time of the target model. When the draft cost c is negligible and the acceptance rate is high enough, the speedup can approach k+1.

    But the acceptance rate is very sensitive to the task:

    • Code completion, formatted text, and repetitive extraction tasks usually have higher acceptance rates;
    • Open-ended reasoning, multi-hop QA, and long-document generation can have lower acceptance rates;
    • If the draft error rate is high, the extra verification can actually slow things down overall.

    The reference model’s native draft layer is trained jointly with the main model in terms of weights, training-data mix, and vocabulary, so the draft-target distribution is more consistent. Even so, the actual benefit should still be measured against your own business distribution rather than judged only by the theoretical formula.

    6. How to Choose Between BF16 and MXFP8?

    The reference model opens both BF16 and MXFP8. MXFP8 generally refers to an 8-bit floating-point format that uses small blocks as the granularity, each with its own scaling factor.

    Potential benefits:

    1. Lower weight memory usage;
    2. If the hardware and framework natively support FP8 matrix operations, it may improve compute throughput.

    But note: throughput improvement is only realistic when the hardware and framework natively support FP8 matrix operations; otherwise, the main benefit may just be reduced memory usage.

    In long-context inference, attention logits accumulate, and outlier channels plus fine-grained scaling can amplify numerical error. A common practice is to set precision per layer:

    • Use FP8 for backbone weights;
    • Keep BF16 for numerically sensitive parts such as attention and normalization;
    • Validate whether precision is acceptable using downstream-task regression metrics.

    7. How to Avoid Benchmark Contamination?

    Public-benchmark score-chasing is not uncommon, for reasons including:

    1. Training data contamination: public questions may already be in the pretraining corpus;
    2. Prompt sensitivity: rephrasing the same question can change the score;
    3. Inconsistent evaluation protocols: few-shot settings, temperature, chain-of-thought length, and whether tools are used all affect results.

    Therefore, for a flagship long-context model, it is recommended to build your own evaluation matrix.

    The evaluation matrix should cover at least:

    • Capability dimensions: long-context retrieval, multi-hop reasoning, aggregation/statistics, long-document extraction, code-repository understanding, tool calling, structured output;
    • Length tiers: 8K, 32K, 128K, 512K, 1M — observe the performance-decay curve as length grows;
    • Task types: build separate sets for retrieval, aggregation, and reasoning tasks;
    • Latency metrics: time to first token (TTFT), time per output token (TPOT), end-to-end throughput;
    • Cost metrics: cost per request, cache hit rate, input-length distribution, output-length distribution.

    For managing a private question set, it is recommended to add the following fields:

    • Question hash;
    • Source;
    • Version;
    • Suspected-leakage reason;
    • Whether a similar question is public;
    • Last updated time;
    • Current active status.

    It is advisable to keep at least 200–500 internal private questions and update them regularly, to prevent answers from being “learned away” as model versions iterate.

    8. How to Estimate Cost?

    Billing for long-context models usually includes input-token price, output-token price, cache-hit discount, and possibly storage or retrieval overhead.

    You can use the following cost model for a rough estimate:

    1. Cost = (L_in / 1e6) × P_in × (1 - r) + (L_out / 1e6) × P_out + additional storage and retrieval overhead

    where:

    • L_in: number of input tokens;
    • L_out: number of output tokens;
    • P_in: input unit price;
    • P_out: output unit price;
    • r: prefix-cache hit rate.

    When estimating, you cannot just compare unit prices — you also need to factor in:

    1. Whether the cache is hit;
    2. The context-length distribution;
    3. Average output length;
    4. Retry rate and failure rate.

    Once a long-context request times out or is truncated, it must be resent, and the cost can multiply.

    9. How to Roll Out Gradually?

    A recommended rollout approach:

    1. Small-traffic canary. Start by putting 5%–10% of traffic on it, focusing on P99 latency and truncation rate.
    2. Context segmentation. Documents over 128K can first be structurally chunked and summarized, then handed to the long-context model for final inference — usually more economical than directly stuffing in 1M tokens.
    3. Monitoring first. Collect at least five metrics: input-length distribution, cache hit rate, TTFT, cost per request, and timeout rate. Mainstream cloud providers’ logging and monitoring/alerting services can handle the observability.
    4. Data and compliance. Long context means more raw data enters the inference pipeline, so you need sensitive-information masking, access control, and audit logging at the gateway layer.

    10. Selection Takeaways

    The key to this generation of ultra-large sparse models is not just the 770B total parameters or the 1M context, but three architectural changes working together:

    1. Attention shifts from standard GQA to MLA, gated sparse attention, and an index cache;
    2. The residual path expands to 4 ways, to improve deep training and information flow;
    3. A native draft layer trained in sync with the main model is provided to support speculative decoding.

    For developers, the more pragmatic principles are:

    1. Don’t decide on a public leaderboard — build a private evaluation set first;
    2. Don’t decide cost by unit price — first compute your own length distribution and cache hit rate;
    3. Don’t decide effectiveness by parameter count — first look at acceptance rate, truncation rate, and latency curves under your task distribution.

    Measure first, then decide. That is more reliable than any single leaderboard.