Evaluating and Deploying Trillion-Parameter Sparse Long-Context Models

    As model architectures evolve, three trends stand out: parameter sparsification, more efficient attention, and native multimodality combined with agentic training. This article uses a representative sparse multimodal model (about 2.8 trillion total parameters, roughly 104 billion activated parameters, and native support for a 1M-token context) as a case study. It explains why such models are designed this way, what engineering barriers training and deployment must clear, and which dimensions ordinary teams should use to evaluate them rationally. The core takeaway: ultra-large sparse models, hybrid attention, multi-stage long-context training, and agentic reinforcement learning form a systematic answer to the "scale, efficiency, long-range capability" trilemma. You cannot judge these models by parameter count alone, nor estimate concurrency by simply dividing total parameters by activated parameters.

    Introduction: Don’t Just Watch the “Trillion Parameters”

    Over the past few years, the main thread of large model architecture evolution can be summarized in three points: parameter sparsification, attention efficiency, and native multimodality with agentic training. A sparse multimodal model with about 2.8 trillion total parameters, roughly 104 billion activated parameters, and native support for a 1M-token context sits precisely at the intersection of these three routes.

    When developers see a parameter table like this, the most common questions are: does more parameters necessarily mean stronger capability? Does fewer activated parameters necessarily mean lower cost? Is long context actually usable?

    This article does not compare specific products. Instead, using this kind of model as a sample, it answers three questions: why is it designed this way? How high are the engineering barriers to training and deployment? From which dimensions should ordinary teams evaluate it rationally?

    The core conclusion: ultra-large-scale sparsity, hybrid attention, multi-stage long-context training, and agentic reinforcement learning form a systematic solution under the trilemma of “scale, efficiency, and long-range capability.” Evaluating such models cannot rely on parameter scale alone, nor can concurrency be estimated by simply dividing total parameters by activated parameters.

    1. Architecture Breakdown: How Sparsity, Attention, and MoE Work Together

    1.1 Why is a sparse architecture “cheap in compute, expensive in memory”?

    104 billion activated parameters make up about 3.7% of the total parameters. This means each token only invokes about one twenty-seventh of the parameters, so the per-token compute stays relatively controlled. But a trillion-scale total parameter count means the weights must reside in GPU memory (VRAM); otherwise every routing step would have to move weights across devices.

    Two kinds of parameters need to be distinguished here: activated parameters determine per-token compute, while total parameters determine knowledge capacity and the lower bound of memory footprint.

    The FLOPs accounting must also be separated. Typically, training can be estimated at about 6× the activated parameters, with forward at about 2× and backward at about 4×. Therefore:

    • Per-token training compute ≈ 6 × 104B ≈ 624B FLOPs, about 0.624 TFLOPs/token;
    • Per-token forward compute ≈ 2 × 104B ≈ 208B FLOPs, about 0.208 TFLOPs/token;
    • Forward compute for a 1M-token request ≈ 0.208 × 1,000,000 ≈ 2.08×10^17 FLOPs, about 208 PFLOPs.
    1. BYTES_PER_PARAM = {"BF16": 2, "FP8": 1}
    2. GIB = 1024 ** 3
    3. total_params = 2.8e12
    4. active_params = 104e9
    5. for dtype, bpp in BYTES_PER_PARAM.items():
    6. total_gib = total_params * bpp / GIB
    7. active_gib = active_params * bpp / GIB
    8. print(dtype, f"weights approx {total_gib:.0f} GiB", f"per-token path touches activated weights approx {active_gib:.0f} GiB")
    9. # forward 2N, training approx 6N
    10. forward_flops_1m = 2 * active_params * 1_000_000
    11. train_flops_1m = 6 * active_params * 1_000_000
    12. print(f"1M-token forward: {forward_flops_1m:.3e} FLOPs")
    13. print(f"1M-token training: {train_flops_1m:.3e} FLOPs")

    Key takeaway: the inference cost of a sparse model is not “smaller activation always means cheaper.” The weights must be resident as a whole, and long context adds pressure from the KV cache and activations.

    The concurrency ceiling cannot be estimated by “total parameters / activated parameters” either. The correct approach: take the available memory, subtract the weight memory, then divide by the per-request KV cache, activations, and framework overhead. It is also constrained by prefill compute, decode memory bandwidth, and scheduling strategy.

    As an illustration, if FP8 weights take up about 2604 GiB and the cluster has 2900 GiB total memory, about 296 GiB remains. If a single 1M-context request needs about 15 GiB of KV cache, and you further reserve memory for activations and framework overhead, the theoretical concurrency is not very high. Shorter 32K requests allow higher concurrency, but are still limited by compute and bandwidth.

    1.2 How do linear attention and full attention complement each other?

    The compute and memory cost of standard attention grow quadratically with sequence length. Linear attention maintains a fixed-size state, reducing complexity to linear, but its exact-retrieval ability is weaker. Therefore, long-context models typically adopt hybrid attention.

    In this kind of architecture, it can be roughly understood as three modules dividing the work:

    • KDA (gated delta attention) handles “remembering.” Through an update mechanism similar to the Delta rule and forgetting gates, its state gains the ability to actively update and selectively forget, making it well suited to carrying long-range memory.
    • Gated MLA (gated multi-head latent attention) handles “finding.” MLA compresses Key/Value into low-dimensional latent vectors to reduce the KV cache; the gating mechanism further filters information across attention heads, retaining a small amount of high-precision global retrieval ability.
    • AttnRes (attention residual) handles “transmitting.” It lets the residual stream collect and weighted-merge information across layers, mitigating the problem of deep models repeatedly accessing shallow-layer information.
    Aspect Standard MHA MLA Linear Attention Gated Delta Attention
    Sequence complexity O(n²) O(n²), but less KV O(n) O(n)
    Cache form Large KV Low-rank latent vectors Constant-size state Constant-size gated state
    Information retention Nearly lossless Compressed Compressed Selectively forgettable
    Exact retrieval Strong Fairly strong Weak Fairly weak

    Note: linear attention is not zero-cache, but a constant-size state that does not grow linearly with sequence length. The hybrid ratio, the placement of full-attention layers, and performance on tasks like multi-needle retrieval and exact matching are key to evaluating long-context capability.

    1.3 How do 896 routed experts stay stable?

    896 routed experts mean the activation probability of any single token is very low. The finer the experts, the more flexible the combinatorial expression, but the more prone training is to load imbalance, routing jitter, and numerical instability.

    Common designs include:

    • Shared experts carry general knowledge, while routed experts carry specialized knowledge;
    • Latent-space MoE performs expert transformations in a compressed space to reduce redundancy;
    • Quantile-based load-balancing constraints cap overload, tracking the long-tail distribution better than a single mean penalty;
    • Smooth gating, within-expert normalization, and z-loss improve numerical stability.

    One key principle: the routing bias is used only to select experts, not to compute the gating weights.

    1. def route_tokens(router_logits, router_bias, topk):
    2. # bias only participates in "who to pick", not "how much weight"
    3. selection_scores = torch.sigmoid(router_logits) + router_bias
    4. expert_ids = torch.topk(selection_scores, k=topk, dim=-1).indices
    5. # gating weights are read from the original unbiased scores
    6. base_scores = torch.sigmoid(router_logits)
    7. weights = base_scores.gather(-1, expert_ids)
    8. weights = weights / weights.sum(dim=-1, keepdim=True).clamp_min(1e-9)
    9. return weights, expert_ids

    This prevents the routing bias from directly changing the output weights of experts, keeping the gating estimate unbiased.

    1.4 How is native multimodality different from a bolted-on vision module?

    For multimodal approaches with a bolted-on vision encoder, the visual capability is often capped by the encoder. Native multimodality, by contrast, treats modalities such as images and video as first-class inputs from the start of pretraining, sharing the Transformer backbone with text, which makes cross-modal reasoning more natural.

    The main difficulties are three: vision tokens are numerous, and high-resolution images can produce a large number of tokens; imbalanced modality ratios can cause language ability to degrade; and different resolutions and aspect ratios need unified handling.

    Common solutions include dynamic-resolution tiling, adaptive visual token compression, and modality curriculum sampling. When evaluating, you cannot look only at visual question-answering scores; you must also observe whether visual input affects pure-text ability, and whether reference resolution and reasoning stay stable in interleaved image-text scenarios.

    2. Training Strategy: Why Expand Context in Stages?

    2.1 Why not scale to 1M in one step?

    In theory you can train on ultra-long context directly, but it is very hard in engineering. The reasons include: attention compute and memory pressure grow rapidly with length; activations and the KV cache increase significantly for long context; million-scale high-quality corpora are scarce and expensive; and position-encoding extrapolation ability is limited.

    Therefore, the common route is to expand stage by stage: 8K → 64K → 256K → 1M.

    • 8K stage: build a solid foundation of language and knowledge;
    • 64K stage: introduce book chapters and long reports to learn discourse structure;
    • 256K stage: introduce code repositories and multi-document collections to learn cross-document integration;
    • 1M stage: use ultra-long samples for position extrapolation calibration and stability finishing.

    Note that it is not simply correct to say “optimizer states are configured for the first stage’s length, so only about twice the length can be trained afterward.” A more accurate statement: long-context expansion is limited jointly by the position-encoding extrapolation ceiling, the scarcity of ultra-long corpora, memory footprint, activation growth, and communication cost.

    2.2 Document packing and position encoding

    Long-sequence training often packs multiple short documents into a single training sample to improve efficiency. At this point, you must prevent cross-document attention from creating spurious correlations.

    1. def build_packed_sequence(docs, max_len):
    2. input_ids, doc_ids, pos_ids = [], [], []
    3. for d_idx, doc in enumerate(docs):
    4. for p, tok in enumerate(doc.tokens[:max_len]):
    5. input_ids.append(tok)
    6. doc_ids.append(d_idx)
    7. pos_ids.append(p) # position counted independently within each document
    8. return (
    9. torch.tensor(input_ids),
    10. torch.tensor(doc_ids),
    11. torch.tensor(pos_ids),
    12. )

    The ideal attention mask makes tokens visible to each other only within the same document. But for million-scale sequences you should not build a dense [seq_len, seq_len] boolean matrix; in real engineering you should use varlen attention, block-diagonal FlashAttention, or an efficient implementation that is document-boundary aware.

    For position encoding, you can adjust the base frequency or scaling factor of the rotary position encoding (RoPE) so the model adapts to longer distances across stages. Combined with document-level masking and stage-wise corpus ratios, this can mitigate distribution shift in long-context training.

    3. Post-Training: Agentic RL Turns Long Context into Task Capability

    Long context itself does not equal long-horizon task capability. A typical agentic task chain is: read documents, call tools, inspect results, and decide again. After multiple rounds, the context easily accumulates to hundreds of thousands or even over a million tokens.

    Post-training usually adopts a pipeline of supervised fine-tuning for a cold start, reinforcement learning, and rejection-sampling iterations. A sketch:

    1. for step in range(num_steps):
    2. tasks = sample_tasks(batch_size)
    3. trajectories = async_rollout(policy, tasks)
    4. valid = [t for t in trajectories if t.is_valid and t.steps >= min_steps]
    5. rewards = [
    6. verify(t) + w_fmt * format_score(t) + w_eff * efficiency_score(t)
    7. for t in valid
    8. ]
    9. advantages = groupwise_normalize(rewards, group_key=lambda t: t.task_id)
    10. loss = ppo_clip_loss(policy, valid, advantages) + beta * kl_penalty(policy, ref_policy)
    11. optimizer.step()
    12. sync_weights(rollout_cluster, policy)

    Key designs include: verifiable rewards first, such as unit tests, answer verification, and tool execution results; format and efficiency rewards to constrain tool-call conventions; KL constraints and group-relative advantages to suppress policy drift; and asynchronous sampling to alleviate the problem of slow sampling over long trajectories and low GPU utilization.

    4. How Should Developers Evaluate Such Models?

    4.1 Effective utilization of long context

    A nominal 1M-token context does not mean the model can stably use 1M tokens. You should test in length steps of 32K, 128K, 256K, and 1M, covering tasks such as single-point retrieval, multi-needle retrieval, cross-segment reasoning, and long-document summarization.

    Pay special attention to how well information at the beginning and middle of the document is retained, because the state updates of linear attention and the retrieval ability of gated MLA both affect the quality of long-range signals.

    4.2 The real cost of a sparse architecture

    Evaluating cost cannot look only at per-token compute. You should measure separately:

    • First-token latency: affected by prefill compute and weight-loading bandwidth;
    • Per-output-token latency: affected by decode memory bandwidth;
    • Peak memory for long requests: determined by the KV cache, activations, and framework overhead;
    • Unit cost: measured by combining cost per million tokens with the maximum effective concurrency.

    Concurrency estimation should use the correct approach: take available memory, subtract weight usage, then divide by the per-request KV cache and activation/framework overhead, and account for compute and bandwidth constraints.

    4.3 Is native multimodality genuine?

    Use real business images to test scenarios such as charts, screenshots, scanned documents, and document layouts; use interleaved image-text to test cross-modal reference; observe whether visual input lowers performance on pure-text tasks.

    4.4 Indirect indicators of training stability

    Even without training logs, you can observe behavior: whether output variance stays stable across multiple samples of the same prompt; whether quality drops off a cliff as input length increases; and whether the expert activation distribution changes dynamically with requests without being overly concentrated.

    4.5 Agent long-horizon task capability

    Evaluate multi-step tool-chain tasks, such as “look up information → summarize → generate a report.” Metrics include task completion rate, average number of steps, failure recovery rate, tool-call format compliance, and whether the goal drifts after long-context accumulation.

    5. Deployment Recommendations

    Dimension Key question Recommended action
    Hardware Can the weights fit entirely in GPU memory? Plan multi-machine, multi-GPU deployment by FP8/BF16 weight size; if memory is insufficient, evaluate distillation or a smaller model
    Serving Throughput and latency under long context Use prefix caching, chunked prefill, and KV-cache quantization; monitor first-token latency and memory levels
    Data pipeline How do million-token documents flow through? Store long documents and multimodal assets in object storage; use a message queue to decouple data fetching, preprocessing, inference, and write-back
    Agent How to orchestrate and observe long trajectories? Host tool environments on a container platform; log every call and intermediate result for easy failure tracing
    Evaluation How to prove it is “good to use”? Build your own business evaluation set covering length steps and multimodal scenarios; do baseline comparisons before launch

    The core principle: validate with real business data first, then decide whether to adopt it. Advanced architecture has real value only when it translates into business metric improvements.

    For developers, the key is not how long the parameter table is, but answering three questions: can it work stably on my data? Does its cost structure match my traffic characteristics? Do its capability boundaries cover my task requirements?