Trillion-Parameter LLM Selection: Attention Mechanisms and Sparse Experts

    As models scale into the trillions of parameters, raw parameter count stops being a reliable proxy for capability. This article explains why, and how two structural techniques reshape the cost of large models: hierarchical hybrid attention for long-context efficiency, and sparse Mixture-of-Experts (MoE) for decoupling capacity from compute. It walks through KV cache memory math, prefill and decode cost tradeoffs, attention complexity, hierarchical attention with learnable summaries and local windows, MoE routing strategies (shared and routed experts, top-k gating, load balancing, capacity factors), activation ratios, and a practical selection and evaluation checklist for choosing a long-context model.

    1. The Trillion-Parameter Era: The Real Dividing Line Is Architecture

    When total model parameters reach the trillions, a common misconception is that more parameters automatically means a stronger model. In reality, parameter count determines a model’s “knowledge capacity,” but three other factors shape actual capability:

    • Effective parameters: Sparse models activate only a small fraction of the total parameters per forward pass, and this portion determines the actual compute of a single inference.
    • Architecture design: The attention mechanism determines how long a context the model can “see,” while expert routing determines which knowledge modules different tasks invoke.
    • Training and data quality: The gains from parameter scaling show diminishing returns as scale grows, while the importance of data quality and training strategy keeps rising.

    Under a sparse architecture, a model usually reports two parameter counts:

    • Total parameters: the full memory footprint of the model’s weights;
    • Active parameters: the parameters that actually participate in compute per token forward pass.

    The ratio of the two is called the activation ratio. Some public model configurations show that sparse models can have activation ratios below 10%, and some flagship configurations approach 3%. This means a model’s knowledge breadth can keep expanding with total parameters, while per-token compute can be held at a relatively low level.

    This leads to the first core judgment of this article: to evaluate whether a model suits ultra-long-context scenarios, you can’t look only at parameter scale — you must look at what structure it uses to separate “capacity” from “cost.”


    2. The First Threshold for Long Context: The KV Cache

    2.1 Where the KV Cache Comes From

    Transformer decoding is autoregressive. To generate the t-th token, the model needs the Key and Value vectors of all preceding tokens. To avoid recomputation, the inference engine caches the K and V of every layer — this is the KV cache.

    The longer the sequence, the larger the KV cache. The memory formula is:

    1. KV cache memory (bytes) = 2 × layers × sequence length × KV heads × per-head dim × bytes per element

    Here the coefficient 2 corresponds to the two copies of data, K and V, and each element takes 2 bytes under fp16. Mainstream attention variants differ mainly in KV head count:

    • MHA: KV head count equals the attention head count;
    • GQA: multiple queries share one group of KV, with KV head count much smaller than query head count;
    • MQA: all queries share one group of KV — the extreme case of GQA.

    Estimation function:

    1. def kv_cache_gb(seq_len: int, layers: int = 70, kv_heads: int = 8,
    2. head_dim: int = 128, bytes_per_elem: int = 2) -> float:
    3. """Estimate KV cache memory (GB). bytes_per_elem: 2 for fp16, 1 for fp8/int8."""
    4. return 2 * layers * seq_len * kv_heads * head_dim * bytes_per_elem / 1e9
    5. print(kv_cache_gb(1_000_000)) # ~287.0 GB (GQA 8 groups, fp16)
    6. print(kv_cache_gb(1_000_000, kv_heads=1)) # ~35.8 GB (MQA, fp16)

    2.2 Order-of-Magnitude Estimates

    Taking 70 layers, hidden dimension 16384, 128 attention heads, 128 dimensions per head, and fp16 precision as an example:

    Attention variant KV heads KV cache for 1M-token context
    MHA 128 ~4.6 TB
    GQA (8 groups) 8 ~287 GB
    MQA 1 ~36 GB

    Even ~287 GB of KV cache under GQA is comparable to the combined memory of several high-end accelerators.

    2.3 The Hidden Cost of Decoding

    The cost of long context is not just about “fitting it in,” but about “reading fast enough.” Each decoded token requires the attention computation to traverse the entire KV cache; on a 70-layer model, that means each token must fully read hundreds of GB of data.

    Taking GQA with 8 groups, 1M tokens, and fp16 as an example, the KV cache is about 287 GB. If memory bandwidth is 3 TB/s and decoding reaches only 50% effective bandwidth, then KV reads alone take roughly per token:

    1. 287 GB ÷ 1.5 TB/s ≈ 0.191 s ≈ 191 ms

    With MQA, the KV cache is about 36 GB, which needs roughly 24 ms under the same bandwidth. This gap shows that the bottleneck in long-context decoding is often not compute but memory bandwidth. Even if the inference engine uses techniques such as paged cache management to improve memory utilization, memory access can still be the primary bottleneck.

    2.4 The Compute Overhead of Prefilling

    The attention compute in the prefill stage grows with the square of the sequence length. For a 70-layer model with hidden dimension 16384, the full attention compute for a single 1M-token input can be estimated with:

    1. attention FLOPs ≈ 4 × layers × sequence length² × hidden dim

    Plugging in the numbers:

    1. 4 × 70 × (1,000,000)² × 16384 ≈ 4.6 EFLOPs

    Since causal attention only needs the lower-triangular part, the actual value can be roughly half, i.e. about 2.3 EFLOPs.

    If the model has about 1.6 trillion total parameters, the linear part costs about 1.6 trillion operations per token, so the linear part for 1M tokens is roughly:

    1. 1.6 × 10^12 × 1,000,000 = 1.6 EFLOPs

    So under ideal conditions, a single 1M-token prefill involves at least a few EFLOPs up to a dozen-plus EFLOPs of compute. If a single node offers 6.4 PFLOPs of theoretical compute, the theoretical time still reaches the minute level, and real-world engineering efficiency also depends on MFU, communication overhead, and batching strategy.

    2.5 Compute Complexity Comparison

    Mechanism Attention compute complexity Per-token KV cache
    Full attention (MHA/GQA/MQA) O(n²) Grows linearly with context length
    Windowed local attention O(n·w) Fixed at window size w
    Hierarchical attention (global summary + local window) O(n·(2k+w)) Fixed at w plus per-layer summaries
    Sparse attention (block selection) O(n²) but with a very small constant Kept on demand

    n is the sequence length, w is the window size, k is the number of summary slots, and m is the hidden dimension. The complexities and cache sizes in the table are order-of-magnitude estimates. Hierarchical attention’s summaries usually assume sharing across shallow layers or progressive passing between layers; if every layer maintains its own summary independently, both cache size and implementation complexity rise.


    3. Route One: Hierarchical Hybrid Attention

    3.1 Three Basic Techniques

    The basic techniques for long-context optimization fall into three categories:

    • Compression: merge the information of multiple tokens into fewer vectors;
    • Restriction: allow a token to attend only to nearby local regions;
    • Selection: let each token pick a few relevant segments on demand.

    Early approaches often used only one of these — for example, splitting attention into several strips while keeping a few global tokens. Hierarchical hybrid attention combines “compression” with “restriction.”

    3.2 Core Idea: Global in Shallow Layers, Local in Deep Layers

    A typical structure is:

    • Shallow layers use compressive shared attention: compress the long sequence into a fixed-length summary vector. All tokens share the same summary, and the summary length is decoupled from the original sequence length.
    • Deep layers use local window attention: keep only the fine-grained information inside the window, and leverage the hierarchical context formed by the shallow layers to focus on the most relevant segments.

    An intuitive way to think about it:

    • The shallow layers act like an editor, who reads the whole text and writes a fixed-length summary;
    • The deep layers act like a reviewer reading segment by segment, able to consult the summary while closely examining the passage at hand.

    Whether the source is 10,000 words or 1,000,000 words, the summary can keep the “global information that must be accessible at any time” at a fixed length.

    3.3 Illustrative Implementation

    The code below illustrates the data flow and focuses on structure rather than engineering details. For actual deployment, you still need engineering work on causal masking, KV cache management, sliding-window operators, and training objectives.

    1. import torch
    2. import torch.nn as nn
    3. import torch.nn.functional as F
    4. class HybridAttention(nn.Module):
    5. """Illustrative hierarchical hybrid attention: summary branch + local window branch."""
    6. def __init__(self, d_model: int, n_heads: int, window: int = 8,
    7. n_summary: int = 4):
    8. super().__init__()
    9. assert d_model % n_heads == 0
    10. self.d_model = d_model
    11. self.n_heads = n_heads
    12. self.window = window
    13. self.n_summary = n_summary
    14. self.q_proj = nn.Linear(d_model, d_model, bias=False)
    15. self.kv_proj = nn.Linear(d_model, 2 * d_model, bias=False)
    16. self.o_proj = nn.Linear(d_model, d_model, bias=False)
    17. # Summary slots: like learnable "global memory addresses"
    18. self.summary_slots = nn.Parameter(torch.randn(n_summary, d_model) * 0.02)
    19. self.summary_gate = nn.Parameter(torch.tensor(-4.0))
    20. def _split_heads(self, x: torch.Tensor) -> torch.Tensor:
    21. # [B, L, D] -> [B, H, L, D/H]
    22. B, L, D = x.shape
    23. H = self.n_heads
    24. Dh = D // H
    25. return x.view(B, L, H, Dh).transpose(1, 2)
    26. def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:
    27. # [B, H, L, D/H] -> [B, L, D]
    28. B, H, L, Dh = x.shape
    29. return x.transpose(1, 2).contiguous().view(B, L, H * Dh)
    30. def forward(self, x: torch.Tensor) -> torch.Tensor:
    31. B, L, D = x.shape
    32. q = self._split_heads(self.q_proj(x))
    33. k, v = self.kv_proj(x).chunk(2, dim=-1)
    34. k = self._split_heads(k)
    35. v = self._split_heads(v)
    36. # 1) Global summary: use learnable summary slots as queries to read the
    37. # context and produce a fixed-length summary vector. The summary
    38. # length n_summary is decoupled from L.
    39. summary_q = self.summary_slots.unsqueeze(0).repeat(B, 1, 1)
    40. summary_q = self._split_heads(summary_q)
    41. summary = F.scaled_dot_product_attention(summary_q, k, v)
    42. summary = self._merge_heads(summary) # [B, n_summary, D]
    43. # 2) Global branch: each token attends to the summary,
    44. # injecting global information into every position at low cost.
    45. summary_kv = self._split_heads(summary)
    46. out_global = F.scaled_dot_product_attention(q, summary_kv, summary_kv)
    47. out_global = self._merge_heads(out_global) # [B, L, D]
    48. # 3) Local branch: build a causal sliding-window mask so each position
    49. # can only see the most recent `window` tokens.
    50. pos_i = torch.arange(L, device=x.device).unsqueeze(1) # [L, 1]
    51. pos_j = torch.arange(L, device=x.device).unsqueeze(0) # [1, L]
    52. mask = (pos_j <= pos_i) & (pos_j >= pos_i - self.window + 1)
    53. out_local = F.scaled_dot_product_attention(
    54. q, k, v, attn_mask=mask.unsqueeze(0).unsqueeze(0)
    55. )
    56. out_local = self._merge_heads(out_local) # [B, L, D]
    57. gate = torch.sigmoid(self.summary_gate)
    58. return self.o_proj(out_local + gate * out_global)
    59. def sparse_moe_forward(x: torch.Tensor, experts, gate,
    60. top_k: int = 2) -> torch.Tensor:
    61. """Illustrative sparse expert layer.
    62. x: [B, L, D]
    63. experts: dict containing a shared module and a list of routed modules
    64. gate: routing module that returns logits, output shape should be [B*L, E]
    65. """
    66. B, L, D = x.shape
    67. flat_x = x.reshape(-1, D) # [N, D]
    68. logits = gate(flat_x) # [N, E]
    69. probs = logits.softmax(dim=-1)
    70. top_probs, top_ids = probs.topk(top_k, dim=-1)
    71. top_probs = top_probs / top_probs.sum(dim=-1, keepdim=True)
    72. out = experts["shared"](flat_x) # [N, D]
    73. routed_out = torch.zeros_like(out)
    74. # Process grouped by expert, avoiding direct tensor indexing with int(eid)
    75. for eid, expert in enumerate(experts["routed"]):
    76. token_idx, slot_idx = (top_ids == eid).nonzero(as_tuple=True)
    77. if token_idx.numel() == 0:
    78. continue
    79. weights = top_probs[token_idx, slot_idx].unsqueeze(-1) # [M, 1]
    80. routed_out.index_add_(
    81. 0, token_idx, weights * expert(flat_x[token_idx])
    82. )
    83. return (out + routed_out).reshape(B, L, D)

    In this code, both the global summary branch and the local window branch of HybridAttention output [B, L, D], so they can be fused along the same token dimension. In an engineering implementation, the local window usually relies on a dedicated sliding-window attention operator rather than building a full mask matrix.

    3.4 Training Objective: Teaching the Model “What Should Be Compressed”

    Changing only the inference structure without adjusting training usually has limited effect. If a model has never been trained to “understand context through a summary,” the summary slots may degenerate into meaningless extra tokens. So this kind of scheme typically introduces the following objectives during pretraining:

    • Summary reconstruction: use the summary vectors to reconstruct the key information of compressed segments, ensuring the summary carries information;
    • Masked global understanding: randomly mask part of the context and answer questions using only the summary and window;
    • Long-range dependency tasks: for example, cross-document linking and paragraph-level question answering, explicitly training the model to use the summary to fill in information outside the window.

    3.5 Dynamic Compression and Engineering Points

    Not every request needs heavy compression. Short documents and single-turn conversations carry limited key information, and forcing compression loses detail instead; long reports and large code repositories need much more aggressive compression. Adapting the compression ratio to input complexity trades less information loss for lower cache overhead.

    Engineering deployment also requires attention to:

    • Coordination with positional encoding: the sliding-window branch needs relative position information to distinguish order within the window; schemes like RoPE and ALiBi affect extrapolation;
    • Pairing with low-rank caches: gains from low-rank compressed key-value caches can stack on top of attention variants;
    • Variable-length handling: summary length, window size, and compression ratio should be configured per length bucket rather than “one-size-fits-all.”

    4. Route Two: Parameter Configuration of Sparse Mixture of Experts

    4.1 Shared Experts + Routed Experts

    A modern MoE layer is usually composed of two kinds of experts:

    • Shared experts: every token passes through them; they carry general knowledge and language ability;
    • Routed experts: dynamically selected per token by the router; they carry fine-grained, domain-specific specialized knowledge.

    In early sparse models, if some experts were undertrained for a long time, “expert collapse” would occur — a few experts being over-selected. Fixing shared experts in every sparse layer lets every token always get stable general information, while routed experts focus on differentiated modeling, improving training stability.

    4.2 Routing Strategies

    • Top-k gating: the router scores all experts for each token and selects the top k. Larger k brings the effect closer to a dense model, but compute and expert-parallel communication overhead rise accordingly; smaller k relies more on router accuracy.
    • Load balancing loss: without constraints, the router tends to repeatedly select a few experts. In practice, a load balancing loss is added to encourage experts to be chosen evenly.
    • Capacity factor: set an upper capacity limit for each expert; tokens beyond the limit are dropped or bypassed, preventing a single expert from becoming a hotspot.

    4.3 Task-Aware Routing

    If some experts are skewed toward a certain kind of task in their training data — for example, code understanding, code generation, or structured text — the router may spontaneously form “domain experts.” The benefit of such routing can be more pronounced in code-related evaluations: the model doesn’t have to spend all its capacity on learning general language, and fine-grained knowledge can be routed to specialized experts.

    It’s worth emphasizing that routing results are usually data-driven and emergent, not manually enforced. The actual effect must be validated with your own evaluation set and can’t be inferred from architecture design alone.

    4.4 Activation Ratio: The Tradeoff Between Efficiency and Capability

    Activation ratio Characteristics Suitable scenarios
    Low activation ratio (~7%) Small per-token compute, good throughput and latency; heavier routing pressure High-concurrency short requests, real-time conversation
    Medium activation ratio (~30%) More balanced between compute and quality, noticeably higher inference cost General Q&A, moderately complex tasks

    Low-activation-ratio schemes aim to “get good enough results with as few active parameters as possible.” But as active parameters shrink, more of the model’s capacity comes from the fine-grained division of labor among routed experts, and router quality becomes one of the deciding factors of the quality ceiling. So the choice of activation ratio should serve the throughput and latency requirements of the business scenario, rather than “lower is always better.”

    4.5 Routing Example

    1. import torch
    2. import torch.nn as nn
    3. class TopKGate(nn.Module):
    4. """Illustrative top-k routing with load balancing loss."""
    5. def __init__(self, d_model: int, n_experts: int, top_k: int = 2,
    6. alpha: float = 0.01):
    7. super().__init__()
    8. self.router = nn.Linear(d_model, n_experts, bias=False)
    9. self.top_k = top_k
    10. self.alpha = alpha
    11. def forward(self, x: torch.Tensor):
    12. # Supports [B, L, D] or [N, D]
    13. logits = self.router(x) # [..., E]
    14. probs = logits.softmax(dim=-1)
    15. top_probs, top_ids = probs.topk(self.top_k, dim=-1)
    16. top_probs = top_probs / top_probs.sum(dim=-1, keepdim=True)
    17. # Load balancing loss: encourage each expert to be selected at a similar frequency
    18. one_hot = torch.zeros_like(probs)
    19. one_hot.scatter_(
    20. -1, top_ids, torch.ones_like(top_probs, dtype=probs.dtype)
    21. )
    22. reduce_dims = tuple(range(one_hot.dim() - 1))
    23. f = one_hot.mean(dim=reduce_dims) # actual assignment ratio, shape [E]
    24. P = probs.mean(dim=reduce_dims) # average routing probability, shape [E]
    25. aux_loss = self.alpha * self.top_k * (f * P).sum()
    26. return top_probs, top_ids, aux_loss
    27. @torch.no_grad()
    28. def load_balance(self, x: torch.Tensor) -> float:
    29. """Return max expert share / mean share; closer to 1 is more balanced."""
    30. logits = self.router(x)
    31. probs = logits.softmax(dim=-1)
    32. top_ids = probs.topk(self.top_k, dim=-1).indices
    33. counts = torch.zeros(
    34. self.router.out_features, device=x.device, dtype=torch.long
    35. )
    36. counts.scatter_add_(
    37. 0,
    38. top_ids.reshape(-1),
    39. torch.ones(top_ids.numel(), device=x.device, dtype=torch.long)
    40. )
    41. return (counts.max().float() / counts.float().mean()).item()

    5. Configuration Comparison and Selection Decisions

    Long context and sparsification are two mutually supporting pieces of the puzzle: hierarchical attention lowers “the per-token context cost,” and sparse experts lower “the per-token model compute.”

    Dimension Dense model Sparse expert model Hierarchical hybrid attention
    Main bottleneck Compute and memory grow together Memory footprint, expert routing communication Training objectives and implementation complexity
    Advantages Stable structure, predictable per-token compute Capacity decoupled from compute, domain expert routing Long-context cache and compute drop substantially
    Risks High inference cost for large-parameter models Routing imbalance, uneven expert load Summary information loss, long-range accuracy drop
    Key parameters Parameter count, quantization bits Total params, active params, expert count, top-k, shared experts Compression ratio, window size, summary length, training objective

    In concrete forms, several common approaches can be summarized as:

    • Shared experts + routed experts: one shared expert guarantees general capability, while the remaining routed experts are dynamically selected per token, giving stable training and moderate inference activation;
    • Full-size coverage: offer different tiers with active parameters ranging from billions to tens of billions under the same training framework, letting developers choose based on hardware;
    • Huge total parameters + domain routing: total parameters reach the trillions while activation stays in the billions, gaining advantages on specific tasks through task-aware routing;
    • Extreme low-activation-ratio efficiency: push the active-parameter share down to single-digit percentages, trading minimal activation for high inference efficiency.

    For selection, you can refer to the following decision table:

    Business scenario Priority focus Recommended validation points
    Long documents over 1M tokens Hierarchical hybrid attention, GQA/MQA, low-rank caches Effective context testing, KV cache memory, prefill time
    High-concurrency short requests Low-activation-ratio MoE, quantization, cache reuse Time to first token, per-token decode latency, expert load balance
    On-premise deployment Dense or small-activation MoE, quantizable architectures Memory footprint, inference framework support, elastic scaling cost
    API calls Context billing, cache hits, throughput tiers Long-input cost, output speed, cache discounts
    Budget-sensitive Low activation ratio, low-precision inference, window compression Per-token cost, effective quality, peak concurrency

    6. Implementation Checklist: How to Evaluate a Long-Context Model

    1. Do the memory math: use the KV cache formula to estimate cache usage for the target context length and concurrency, then add model weights and inference framework overhead.
    2. Verify the effective context: use “needle in a haystack” style tests, placing key information at different positions in a long document and checking whether the model recalls it accurately, rather than only trusting the officially advertised maximum context length.
    3. Separate the stage costs: prefill is compute-bound and decode is memory-bandwidth-bound. First clarify whether the workload is mainly long input or long output, then choose the matching optimization.
    4. Check architecture details: confirm that the attention variant, expert structure, activation ratio, routing strategy, and other design choices match the business needs, rather than comparing total parameter counts alone.
    5. Evaluate with business data: public benchmarks only reflect general capability. Building a small evaluation set on real business data is what reveals “high leaderboard scores, low business efficiency” problems.
    6. Choose the right deployment form: for on-premise deployment, prioritize inference optimizations such as quantization, expert parallelism, and cache reuse; for API calls, pay attention to context billing and cache-hit discounts. Mainstream cloud platforms and inference frameworks typically offer capabilities such as containerized elastic scaling, object storage, logging services, and monitoring/alerting that can be combined.

    Common Mistakes

    • Only trusting the advertised maximum context: the official context length doesn’t equal the length a model can effectively use in real tasks.
    • Ignoring prefill compute: for long-input scenarios, prefill can become the bottleneck before decoding does.
    • Ignoring KV cache concurrency amplification: 36 GB for a single request may seem manageable, but it quickly exceeds the memory budget at high concurrency.
    • Only looking at total parameters: total parameters determine capacity, while active parameters are closer to the per-token compute cost.
    • Overlooking MoE routing risk: uneven expert load, routing jitter, and communication overhead all affect real throughput.
    • Ignoring the billing model: some platforms bill per token and offer discounts for cache hits; clarify the cost model before selecting.

    7. Conclusion

    A trillion-parameter model’s “strength” comes more from its structure than from its numbers. Hierarchical hybrid attention answers “can long context be read efficiently,” while sparse experts answer “can a large model be afforded and run.”

    For developers, rather than chasing parameter rankings, it’s better to break the architecture apart with the memory formula, the complexity model, and a business evaluation set to see clearly: which part carries general capability, which carries domain knowledge, and which carries long-range memory. Once these three points are clear, selection and deployment both rest on a firmer foundation.