Trillion-Parameter Model Architectures Explained: Hybrid Attention and Mixture-of-Experts

    As language models scale from hundreds of billions to trillions of parameters, the central challenge is no longer simply adding capacity. It is reducing the computation and memory required to use that capacity. This article explains why KV caching becomes a hard constraint for long-context inference, how hybrid attention combines global compression with local precision, how four common MoE routing designs trade off capacity and efficiency, and which training and serving techniques are needed to make trillion-parameter models practical.

    Introduction: Trillion-Parameter Models Are About Efficiency

    Over the past two years, large language models have expanded from hundreds of billions to trillions of parameters. However, larger parameter counts do not always produce proportional quality improvements, while training and inference costs continue to rise.

    The underlying issue is computational structure. In a standard Transformer, the work for each token can be roughly divided into two parts:

    • Attention: Related to sequence length. Its computational complexity is approximately O(L²·d), while inference-time KV-cache usage is approximately O(L·d).
    • Feed-forward network (FFN): Related to parameter count. In a dense model, every token passes through the entire FFN.

    At the trillion-parameter scale, sending every token through all parameters is prohibitively expensive. The industry’s primary solution is sparsity, applied at two levels:

    Layer Dense approach Sparse approach Main benefit
    Attention Each token attends to the entire history Global compressed summaries plus local fine-grained attention Reduces computation and KV-cache cost for long sequences
    Expert layer (FFN) Every token uses the full FFN Mixture-of-Experts (MoE), where each token activates only a few experts Increases total capacity while keeping per-token computation manageable

    These two approaches can be adopted independently. However, combining them is what makes it possible to balance quality and cost at the trillion-parameter scale. This has become a common design principle for leading model architectures.

    This article examines four questions:

    1. Why KV caching becomes a hard constraint in long-context workloads;
    2. How hybrid attention combines global compression and local precision, and what engineering problems it introduces;
    3. How four common MoE designs differ and when each is appropriate;
    4. Which engineering techniques are required during training and inference.

    1. KV Cache: The Real Bottleneck in Long-Context Inference

    1.1 What Is a KV Cache?

    During autoregressive decoding, each newly generated token attends to all previous tokens. Recomputing the historical Keys and Values at every step would create substantial redundant work. Inference systems therefore cache historical K and V tensors. This is known as the KV cache.

    Positional encoding and KV caching

    • Rotary positional embeddings are commonly used during pretraining. Their relative-position properties allow historical K and V tensors to be reused directly when generating new tokens.
    • Some positional-encoding schemes introduce transformations related to the query position during attention computation. Their cache organization must therefore be designed accordingly.

    When evaluating an attention modification, it is essential to confirm that its training and inference behavior is consistent.

    1.2 Estimating Memory Usage

    KV-cache size is determined directly by the model architecture:

    1. KV-cache memory =
    2. 2 (one K and one V)
    3. × number of layers
    4. × number of KV heads
    5. × head dimension
    6. × sequence length
    7. × batch size
    8. × bytes per element
    • 2: One copy each for K and V;
    • Number of KV heads: In grouped-query attention (GQA), multiple query heads share a smaller number of KV heads;
    • Bytes per element: BF16 and FP16 use 2 bytes, while FP8 and INT8 quantization use 1 byte.
    1. def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch=1, dtype_bytes=2):
    2. """Estimate KV-cache size in bytes."""
    3. return 2 * layers * kv_heads * head_dim * seq_len * batch * dtype_bytes
    4. # Scenario 1: a typical dense model with 60 layers,
    5. # 8 KV heads, head dimension 128, and BF16 KV cache
    6. per_token = kv_cache_bytes(60, 8, 128, seq_len=1)
    7. print(per_token / 1024, "KiB per token")
    8. print(kv_cache_bytes(60, 8, 128, 1_000_000) / 1024**3)
    9. # Approximately 223 GiB for one million tokens
    10. # Scenario 2: a large model with 64 layers and hidden size 8192,
    11. # using standard multi-head attention (MHA)
    12. # Head dimension = 8192 / 64 = 128
    13. print(kv_cache_bytes(64, 64, 128, 1_000_000) / 1024**3)
    14. # Approximately 1.86 TiB for one million tokens

    At a context length of one million tokens:

    • The KV cache can easily reach hundreds of gigabytes, exceeding the memory capacity of a single accelerator;
    • Model weights, communication buffers, and temporary activations create additional pressure.

    The problem becomes even more severe under batching because the cache requirement grows approximately linearly with concurrency. Long-context inference is therefore constrained by memory capacity and memory bandwidth, not just raw compute.

    1.3 Three Generations of Optimization

    Approach Representative design Historical information read per token Characteristics
    Full attention Every token attends to the entire history Grows linearly with sequence length Complete information, but computation and cache usage grow with length
    Sparse attention Sliding windows, block sparsity, global tokens Local windows plus a small number of global positions Lower cost; pure windows lose distant information quickly, so global anchors are useful
    Compressed shared attention Compress blocks into summaries shared globally A fixed number of summary vectors Computation and cache usage become approximately decoupled from sequence length; quality depends on compression

    The engineering implication of the third approach is straightforward: if each token reads only a fixed number of summaries and local fragments, KV-cache reads no longer grow linearly with context length. This is a key prerequisite for running long-context models within finite memory.

    2. Hybrid Attention: Combining Global Compression and Local Precision

    A widely discussed hybrid-attention design divides attention into two paths:

    • Compressed shared attention provides global coverage. It divides a long sequence into blocks and compresses them into summaries that all tokens can access.
    • Hierarchical contextual attention provides local detail. Guided by the compressed representation, it performs fine-grained attention over only a small number of relevant segments.

    The structure resembles retrieval-based inference: first build a coarse index with summaries, then perform detailed computation on the selected segments.

    Implementations can organize these paths differently. Some assign different roles to different layers, with shallow layers emphasizing global compression and deeper layers emphasizing local detail. Others place both branches in every layer and fuse their outputs. The underlying principle is the same:

    Use coarse filtering to control scale, and precise computation to preserve accuracy.

    2.1 Compressed Shared Attention: From Global Tokens to Learnable Summaries

    The idea of retaining a small number of global positions accessible to all tokens has existed for some time. Hybrid attention advances this approach by generating learnable summaries for individual blocks.

    A typical process is:

    1. Divide the sequence into blocks, such as several hundred tokens per block;
    2. Compress each block through pooling, a linear projection, or a small network;
    3. Let all tokens attend to the shared summary cache instead of the full original sequence.

    Two improvements are particularly important.

    Dynamic compression ratios

    Uniform compression—for example, converting every fixed number of tokens into one summary—does not account for differences in information density.

    • Code, tables, and logs are information-dense, so excessive compression can remove important details;
    • Repetitive text may tolerate much stronger compression.

    The compression granularity can be adjusted according to content complexity, such as the degree of variation among token representations within a block. Compression itself can also be made learnable and differentiable, allowing training to determine which information should be retained.

    Hierarchical pretraining objectives

    If compression is introduced only during inference, training and inference become inconsistent. The model has not learned how to use summaries. A more systematic approach incorporates compression and retention into pretraining objectives, for example:

    • Asking the model to reconstruct or complete important information from summaries;
    • Constructing long-context examples in which the answer is located in a distant segment, forcing the model to establish long-range connections through summaries.

    The number of context tokens and summaries can also be increased progressively through curriculum learning, allowing the model to adapt gradually to the workflow of “read summaries, then inspect local details.”

    2.2 Hierarchical Contextual Attention: Making Local Computation Selective

    Attention restricted to a fixed sliding window is simple and hardware-friendly, but it truncates information outside the window.

    A hierarchical design can use compressed summaries to score blocks—for example, by measuring similarity between a query and each block summary—and then select the highest-scoring blocks for fine-grained attention:

    • First stage: Match against block summaries at low cost;
    • Second stage: Perform token-level attention over the selected blocks.

    This is similar to an indexed database lookup followed by a table fetch: use an index to narrow the search range, then access the original records. Whether the system selects branches or blocks, the underlying structure is the same—an index plus a detailed lookup.

    2.3 How the Two Paths Work Together

    The two paths are complementary:

    • The compression path ensures that information from any position remains reachable in summarized form, solving the problem of distant information that cannot fit into a local window;
    • The local path preserves token-level detail in the regions that matter, solving the problem of information loss caused by summaries alone.

    This combination is particularly useful for long-document question answering, where the model must first locate the relevant chapter and then inspect the exact passage, and for code understanding, where it must first identify module structure and then verify function signatures.

    2.4 Four Engineering Challenges

    1. Information bottlenecks

    Compression is inherently an information bottleneck. The shorter the summary, the greater the potential loss. Compression ratios should reflect task-specific information density and should be supported by learnable mechanisms rather than fixed mean pooling alone.

    2. Positional information

    The way summaries encode position directly affects the model’s ability to reason about distance.

    • If a summary is treated as a virtual token, how should its position index be mapped?
    • How can the extrapolation behavior of rotary positional embeddings be preserved?

    This is one of the most failure-prone parts of hybrid-attention implementations.

    3. Training and inference consistency

    The attention mask, block partitioning, and selection strategy used during training must match the cache structure used during inference. Otherwise, the model may learn attention patterns that cannot be reproduced in production, causing offline evaluation and online quality to diverge.

    4. Hardware affinity

    Attention performance depends heavily on memory-access patterns.

    • Dynamic indexing and branching during block selection or summary assembly can break memory contiguity and make kernels more difficult to optimize;
    • Lower theoretical FLOPs do not necessarily translate into proportional real-world speedups;
    • In practice, block sizes and access patterns are often designed to be regular and aligned with hardware requirements.

    The value of hybrid attention lies not in one isolated mechanism, but in the layered cooperation of compression, retrieval, and precise computation. Its evaluation should consider both theoretical complexity and actual throughput and latency on target hardware.

    3. Comparing Four Typical MoE Routing Designs

    Attention determines how much context a token reads. MoE determines how much of the model’s parameter capacity a token uses.

    3.1 How MoE Works

    MoE replaces a Transformer’s FFN with multiple parallel expert networks. A router selects a small number of experts for each token.

    • Total parameters are the sum of all experts;
    • Active parameters are the parameters actually used by each token.

    Therefore, “trillion parameters” usually refers to total parameters. A single token may activate only a small fraction of them.

    MoE designs primarily differ in four dimensions:

    1. Whether shared experts exist;
    2. The number and granularity of experts;
    3. How Top-k selection is performed;
    4. How load balancing is enforced.

    The following four categories use illustrative configurations only and do not represent any specific model.

    3.2 Design One: Shared Experts Plus Routed Experts

    • Structure: Several routed experts plus one always-active shared expert. Routed experts are selected dynamically, such as with Top-2 routing.
    • Example: Eight routed experts with Top-2 routing and one shared expert. Each token activates two routed experts, producing an active parameter scale in the tens of billions.
    • Assumption: The shared expert handles general knowledge and language patterns, while routed experts specialize in domains.
    • Advantages:
      • General capabilities have a stable computational path;
      • Overlap among routed experts can be reduced;
      • The shared expert preserves baseline quality even when a routed expert is not selected.
    • Cost: The shared expert runs for every token, creating fixed overhead. As the number of routed experts grows, this fixed cost becomes a larger proportion of active computation.

    3.3 Design Two: Fine-Grained Experts with Shared-Expert Isolation

    • Structure: Split a conventional FFN expert into many smaller experts, potentially dozens or hundreds, and increase Top-k accordingly, such as Top-6 or Top-8. A small number of shared experts are retained.
    • Core idea: More small experts combined with a larger k can approximate the capacity of fewer large experts with a smaller k, while providing more flexible combinations.
    • Advantages:
      • A larger combination space enables finer specialization;
      • Smaller experts can be partitioned more evenly under expert parallelism;
      • Shared experts isolate general capabilities from specialized knowledge.
    • Costs: Routing decisions become more frequent, increasing gating and expert-switching overhead. More experts also increase the requirements for routing quality, load balancing, and communication efficiency.

    3.4 Design Three: Fully Routed Experts Without Shared Experts

    • Structure: Every expert is selected dynamically; no expert is always active.
    • Advantages: The structure is simple, and active computation contains no fixed shared-expert overhead. The activation ratio can be reduced significantly.
    • Costs: General knowledge must be distributed across multiple experts. Early in training, expert specialization may be unclear, increasing reliance on auxiliary losses for convergence.

    3.5 Design Four: Expert Choice

    The first three designs use token-choice routing, where each token selects experts. Expert Choice reverses the direction: each expert selects tokens up to its capacity.

    • Advantages:
      • Each expert’s token count is directly limited by capacity, providing natural load balancing;
      • Auxiliary balancing losses may no longer be necessary;
      • The design is suitable for large-scale distributed training with more regular communication.
    • Costs:
      • Different tokens may be selected a different number of times, and some may not be selected by any expert;
      • The mechanism scores all tokens for each expert. During autoregressive decoding, future tokens do not yet exist, so this design generally applies to training or non-autoregressive settings and cannot be directly used for token-by-token generation.

    3.6 Side-by-Side Comparison

    Dimension Shared + routed Fine-grained + shared Fully routed Expert Choice
    Always-active shared experts Yes, typically one Yes, a small number No No
    Example expert count Single digits to the low dozens Dozens to hundreds Dozens to several dozen Dozens to several dozen
    Active experts per token Small, such as two Larger, such as six to eight Small Determined by capacity
    General capability carrier Shared experts Shared experts Distributed Distributed
    Load balancing Auxiliary losses usually required Auxiliary losses usually required; fine granularity can help Auxiliary losses usually required Naturally balanced through capacity
    Training stability Relatively strong Strong Moderate Strong during training
    Inference suitability Good Good Good Limited for autoregressive generation

    Selection guidelines

    • For stable training and strong general capabilities, shared-plus-routed designs are a practical choice;
    • For stronger specialization and more even expert utilization, fine-grained designs are worth considering;
    • For the lowest possible activation ratio, fully routed designs may be appropriate;
    • For training-oriented load balancing and regular communication, Expert Choice can be evaluated.

    The final decision should consider total parameters, active-parameter ratio, training-data scale, and the chosen distributed-parallelism strategy.

    3.7 Routing Example

    The following pseudocode illustrates a shared-expert plus routed-expert forward pass. It is conceptual and is not tied to a specific framework API.

    1. class SparseMoELayer(nn.Module):
    2. def __init__(self, hidden_size, num_routed, top_k=2, num_shared=1):
    3. super().__init__()
    4. self.router = nn.Linear(hidden_size, num_routed, bias=False)
    5. self.routed_experts = [Expert(hidden_size) for _ in range(num_routed)]
    6. self.shared_experts = [Expert(hidden_size) for _ in range(num_shared)]
    7. self.top_k = top_k
    8. def forward(self, x): # x: [batch, seq_len, hidden]
    9. flat = x.reshape(-1, x.shape[-1])
    10. logits = self.router(flat)
    11. scores, idx = top_k(logits, k=self.top_k)
    12. weights = softmax(scores, dim=-1)
    13. # Every token passes through the shared experts
    14. out = sum(shared(flat) for shared in self.shared_experts)
    15. for e_id in unique(idx):
    16. mask = (idx == e_id)
    17. if not mask.any():
    18. continue
    19. token_ids = where(mask)
    20. e_out = self.routed_experts[e_id](flat[token_ids])
    21. out[token_ids] += weights[token_ids, mask] * e_out
    22. return out.reshape_as(x)

    4. Training and Inference: Four Supporting Engineering Requirements

    Even a carefully designed architecture requires supporting systems and optimization techniques.

    4.1 Training: Load Balancing and Communication Efficiency

    A typical MoE challenge is uneven expert utilization. A small number of popular experts may receive most tokens, while others are rarely activated.

    Common techniques include:

    • Auxiliary load-balancing loss
    1. L_balance = α · N · Σ_i (f_i · P_i)
    • f_i: The proportion of tokens assigned to expert i in a batch;
    • P_i: The average gating probability for expert i;
    • N: The total number of experts;
    • α: The loss weight.

    When routing is concentrated, an expert with both high f_i and high P_i increases the loss, encouraging more balanced routing.

    • Capacity factor: Set a capacity limit for each expert. Excess tokens can be dropped and sent through the residual connection, or redirected to a secondary expert. A capacity factor that is too small causes token dropping and can hurt training; one that is too large wastes memory.
    • Routing stability: Apply a z-loss constraint to gating logits to prevent numerical drift and unstable routing.

    MoE communication also differs significantly from dense-model communication. Dense models mainly rely on collective operations such as AllReduce. Expert parallelism requires tokens to be dispatched to the devices hosting the corresponding experts and then gathered afterward. This dispatch and aggregation process is sensitive to bandwidth and network topology, so communication and computation must be carefully overlapped.

    4.2 Inference: Active Parameters Determine Compute, Total Parameters Determine Memory

    A frequently overlooked fact is that MoE compute per token depends on active parameters, while model-weight memory depends on total parameters.

    This means:

    • A low activation ratio can significantly reduce computation, but it does not remove the memory pressure of loading all expert weights;
    • In long-context workloads, the KV cache adds another major memory requirement.

    Common optimizations include weight quantization, KV-cache quantization, prefix-cache reuse, chunked prefill, and speculative decoding. Compressed caching in hybrid attention is complementary to these methods, but its memory layout must remain compatible with quantization and paged-cache mechanisms.

    4.3 Evaluation: Parameter Count Is Not Enough

    A trillion-parameter model should be evaluated across at least four dimensions:

    1. Quality: Long-context retrieval, cross-document reasoning, code, mathematics, and other targeted capabilities;
    2. Efficiency: Time to first token (TTFT), time per output token (TPOT), throughput in tokens per second, and memory usage at long context lengths;
    3. Cost: Per-token inference cost, training stability, and support for fine-tuning and continued training;
    4. Engineering: Inference-framework support, distributed deployment complexity, and observability.

    Looking only at total parameter count or a single benchmark score can lead to misleading conclusions.

    4.4 Cloud Deployment: Turning Architectural Gains into Reliable Services

    Once the architecture is designed, stable training and inference operations become a separate challenge.

    • Container platforms can provide elastic scheduling for training jobs;
    • Object storage can hold datasets, model artifacts, and checkpoints;
    • Logging, monitoring, and alerting systems can track throughput, latency, and memory usage.

    These general cloud capabilities help turn architectural optimizations into maintainable production services.

    5. Conclusion

    The strength of a trillion-parameter model does not come from its parameter count alone. It depends on whether its two major sparsity mechanisms are designed effectively:

    • Hybrid attention combines global compressed summaries with local fine-grained attention. This can transform KV-cache behavior from linear growth with sequence length into an approximately fixed overhead, creating a path toward million-token contexts. Its success depends on learnable compression, consistency between training and inference, and hardware-friendly kernels.
    • MoE routing—whether shared plus routed experts, fine-grained experts, fully routed experts, or Expert Choice—balances general capabilities, specialization, load distribution, and active computation. There is no universally optimal design; the right choice depends on training scale and data distribution.

    For practitioners, three factors deserve particular attention:

    1. Actual throughput and memory consumption under long-context workloads;
    2. MoE routing balance and communication efficiency;
    3. Consistency between training and inference.

    Understanding these mechanisms is more valuable than memorizing a model’s total parameter count.